Connecting to Cluster not working with docker ENETUNREACH

[kirrg001]

Description

Hi!

My docker setup works great with ioredis, but not with node-redis.
(I can successfully connect to each single node with createClient, but createCluster does not work.)

REDIS_NODE_1: '127.0.0.1:6380',
REDIS_NODE_2: '127.0.0.1:6381',
REDIS_NODE_3: '127.0.0.1:6382',

// const redis = require('redis');  - same problem
const redis = require('@redis/client');
const cluster = redis.createCluster({
  rootNodes: [
    {
      url: `redis://${process.env.REDIS_NODE_1}`
    },
    {
      url: `redis://${process.env.REDIS_NODE_2}`
    },
    {
      url: `redis://${process.env.REDIS_NODE_3}`
    }
  ],
  defaults: {
    socket: {
      connectTimeout: 50000
    }
  }
});

[cluster].forEach(c => {
  c.on('error', err => log('Redis Cluster Error', err));
});

(async () => {
  log(`Connecting to cluster. (${process.env.REDIS_NODE_1}, ${process.env.REDIS_NODE_2}, ${process.env.REDIS_NODE_3})`);
  await cluster.connect();
  log('Connected to cluster');

  connectedToRedis = true;
})();

Redis Cluster App (7540):	Connecting to cluster. (127.0.0.1:6380, 127.0.0.1:6381, 127.0.0.1:6382)
...
Redis Cluster App (7232):	Redis Cluster Error Error: connect ENETUNREACH 172.30.0.2:6379
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1606:16)
    at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
  errno: -51,
  code: 'ENETUNREACH',
  syscall: 'connect',
  address: '172.30.0.2',
  port: 6379
}

$ nc -zv 127.0.0.1 6380
Connection to 127.0.0.1 port 6380 [tcp/*] succeeded!

$ docker exec -it 22917895915d redis-cli -p 6379 cluster nodes
d13eae594c1c7e6b483aacd7798ab613b8037397 172.30.0.4:6379@16379 master - 0 1723455985696 3 connected 10923-16383
44458d987c67d19dcdd673fe805533459bd1b046 172.30.0.3:6379@16379 myself,master - 0 1723455984000 2 connected 5461-10922
57e8d72cbed15e21248effe767b61ef133570a38 172.30.0.2:6379@16379 master - 0 1723455984643 1 connected 0-5460

stdout: redis-node-0-1  | Cluster correctly created
stdout: redis-node-1-1  | 1:M 12 Aug 2024 09:49:22.107 * Cluster state changed: ok
stdout: redis-node-0-1  | 89:M 12 Aug 2024 09:49:22.108 * Cluster state changed: ok
stdout: redis-node-2-1  | 1:M 12 Aug 2024 09:49:22.210 * Cluster state changed: ok

$ docker network inspect nodejs_redis-cluster-network
[
    {
        "Name": "nodejs_redis-cluster-network",
        "Id": "0f790833c4b8098627f1b6942831f264016e776722bfe7cad4d6c965de91e943",
        "Created": "2024-08-12T09:31:27.645823964Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.30.0.0/16"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "116560657e78e6267ed0ae50aebce2b175912da1602c3b2b3628a434f8692060": {
                "Name": "nodejs-redis-node-0-1",
                "EndpointID": "b102c8e309ec241301ec9f11bd400b630209d110059ed7f793d5693325c45882",
                "MacAddress": "02:42:ac:1e:00:02",
                "IPv4Address": "172.30.0.2/16",
                "IPv6Address": ""
            },
            "22917895915d80bfd88af52cc336c6d8d3fe14272f3898578730dae2eef7b10f": {
                "Name": "nodejs-redis-node-1-1",
                "EndpointID": "1e87aae09b68fd8d9521cdbd6033ac4069c8969413827b7cae0840e715c18855",
                "MacAddress": "02:42:ac:1e:00:03",
                "IPv4Address": "172.30.0.3/16",
                "IPv6Address": ""
            },
            "a69acfdde86e808fc1ba97afb47c7e08c0c731627d5180722bc162d81be1c9c3": {
                "Name": "nodejs-redis-node-2-1",
                "EndpointID": "9c1bda5eba16afdc1962c59774ab09f91ebf8fa1b4080120e8c44bcb2ffc66d5",
                "MacAddress": "02:42:ac:1e:00:04",
                "IPv4Address": "172.30.0.4/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "redis-cluster-network",
            "com.docker.compose.project": "nodejs",
            "com.docker.compose.version": "2.23.3"
        }
    }
]

version: '2'
services:
  redis-node-0:
    image: docker.io/bitnami/redis-cluster:7.2
    environment:
      - ALLOW_EMPTY_PASSWORD=yes
      - REDIS_NODES=172.30.0.2:6379 172.30.0.3:6379 172.30.0.4:6379
      - REDIS_CLUSTER_CREATOR=yes
      - REDIS_CLUSTER_REPLICAS=0
      - BITNAMI_DEBUG=1
      - REDIS_CLUSTER_SLEEP_BEFORE_DNS_LOOKUP=30
    ports:
      - '6380:6379'
    networks:
      redis-cluster-network:
        ipv4_address: 172.30.0.2
  redis-node-1:
    image: docker.io/bitnami/redis-cluster:7.2
    environment:
      - ALLOW_EMPTY_PASSWORD=yes
      - REDIS_NODES=172.30.0.2:6379 172.30.0.3:6379 172.30.0.4:6379
    ports:
      - '6381:6379'
    networks:
      redis-cluster-network:
        ipv4_address: 172.30.0.3
  redis-node-2:
    image: docker.io/bitnami/redis-cluster:7.2
    environment:
      - ALLOW_EMPTY_PASSWORD=yes
      - REDIS_NODES=172.30.0.2:6379 172.30.0.3:6379 172.30.0.4:6379
    ports:
      - '6382:6379'
    networks:
      redis-cluster-network:
        ipv4_address: 172.30.0.4
networks:
  redis-cluster-network:
    driver: bridge
    ipam:
      config:
        - subnet: 172.30.0.0/16

Node.js Version

20.13

Redis Server Version

docker.io/bitnami/redis-cluster:7.2

Node Redis Version

4.7.0

Platform

macOS

Logs

See above.

[kirrg001]

I have tried this comment and I can establish a connection. I just added a default url.

const cluster = redis.createCluster({
  rootNodes: [
    {
      url: `redis://${process.env.REDIS_NODE_1}`
    },
    {
      url: `redis://${process.env.REDIS_NODE_2}`
    },
    {
      url: `redis://${process.env.REDIS_NODE_3}`
    }
  ],
  useReplicas: false,
  defaults: {
    url: `redis://${process.env.REDIS_NODE_1}`
  }
});

[sjpotter]

so just as an observation, there's a difference between the IP you list in the env and the docker IPs. My guess is you are having the ports from the docker containers be visible on the host (and hence why can connect via localhost via a normal client).

However, the cluster itself is setup to resolve via the docker IPs (and hence what they return to the client when it interogates a single node for the cluster setup) and then it fails to connect to 172.30.0.2:6379 as per the error shown.

if you try to connect via redis-cli to 172.30.0.2:6379 does it work or does it fail as well?

[sjpotter]

if the above is the case - see redis/redis#7460

[kirrg001]

Thank you for taking a look!

if you try to connect via redis-cli to 172.30.0.2:6379 does it work or does it fail as well?

From my host: no
From within docker: yes

so just as an observation, there's a difference between the IP you list in the env and the docker IPs. My guess is you are having the ports from the docker containers be visible on the host (and hence why can connect via localhost via a normal client).

Yes docker compose exposes the ports to the host. The internal docker IP range for the bridge net work is 172.30.0.0/16.

if the above is the case - see redis/redis#7460

If that is the case, feel free to close my issue 👍

[sjpotter]

yea, you have to use the announce ip feature, otherwise redis cluster will enumerate the docker IPs, which is why you get the error.

[kirrg001]

you have to use the announce ip feature

I assume you are talking about the missing feature redis/redis#7460?

Could you please confirm that the workaround is acceptable?
Thanks!

[sjpotter]

ok, you're right, that doesn't help directly. The issue is that its not as much an issue in the client, but an issue in the network setup / redis cluster. I can't figure out a way for redis cluster to distinguish the network it uses for internal communication and the addresses it tells clients to use. Not an expert in that.

The easiest solution would be to somehow expose the docker network to be routable from your local machine so that the addresses returned by redis to the client are valid, but it seems the NAT issue above are related.

another possible solution would be to use iptables to rewrite all outgoing connection to <docker ip1:port 6370 to 127.0.0.1:6379 for each ip/port in use)

https://superuser.com/questions/661772/iptables-redirect-to-localhost

[kirrg001]

My docker setup works great with ioredis

Any idea why ioredis works great without doing any changes?

[sjpotter]

Honestly, dont know exactly @leibale any ideas?

In glancing at ioredis I see it has a "natMap" mode (unsure what it is, just making guesses here). It's possible the mode is dynamic as well (not just something a user has to setup), i.e. when it connects to 127.0.0.1:6379 it learns that said host is really 172.30.0.2:6379 and hence then is able to map that so it always then connects to 127.0.0.1:6379 for that.

It's a cute fix (and perhaps even more than cute, somewhat smart). But it will fail if their is any changes in the cluster (i.e. at runtime new replicas are added, that weren't known originally).

but this is just a guess on my part.

[kirrg001]

Hey!

Could we please reopen this ticket? This is not resolved.

The workaround only works to connect to the cluster, but I quickly experienced problems when operating on the docker cluster:

e.g. await cluster.hset throws

[ErrorReply: MOVED 6684 172.30.0.3:6379]

Increasing the maxCommandRedirections from 16 to 100 did not help.

Connecting to a remote Azure Redis cluster resolves the problem. Not related to my code.

[sjpotter]

this isn't a bug in node-redis, it could be a feature to add, but its not a bug. The issue is that redis-cluster isn't designed to operate behind a NAT. io-redis seems to have some support for mapping "cluster IP/ports" to a different ip/port.

i.e. the azure redis cluster returns to the client the proper ip/ports to connect to.

[kirrg001]

Can we please reopen the ticket as a feature request?

[leibale]

@sjpotter @kirrg001 you can use https://github.com/redis/node-redis/blob/master/docs/clustering.md#node-address-map to map the addresses

[sjpotter]

ah great, didn't know about that feature.

[kirrg001]

I'll try it asap. Thank you!