Problem with overlay network and DNS resolution between containers

[ggaugry]

Description

I noticed some problems with DNS resolution inside a overlay network and Swarm.
DNS entries are not always updated automatically.
I have 10 containers over 4 hosts on Ubuntu 16.04 connected by Swarm. I created an overlay network for those containers.
When I redeploy one of those containers (I stop the current one, rename it to OLD, and create a new one with the same name), the container will not always have the same IP as before (which is not a problem). But It looks like it does not always update the DNS entry for the others containers in the network. The new created container is then unreachable from other one.

My docker version is 1.13.0.

Steps to reproduce the issue:

1. Create a Swarm architecture with multiple hosts
2. Create a overlay network
3. Deploy few containers with specific names on each node and attach them to this network
4. Remove one of this container and create exactly the same one.

Describe the results you received:
If IP of this new container has changed, the DNS entry will not be updated automatically for others containers. If you try to ping this new container dns name from others containers, sometimes you will notice that the resolved IP is actually the IP of the previous removed container.

Describe the results you expected:
DNS entries should be updated for every containers when these last ones have their IP changed.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Docker version 1.13.0, build 49bf474

Output of docker info:

Containers: 14
 Running: 10
 Paused: 0
 Stopped: 4
Images: 449
Server Version: 1.13.0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 571
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: active
 NodeID: okiqm8slow52nm4rx8qt08rpc
 Is Manager: true
 ClusterID: 7b3cohqvxgp3q9qm19xq4dj97
 Managers: 2
 Nodes: 4
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 172.17.10.83
 Manager Addresses:
  172.17.1.224:2377
  172.17.10.83:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e
runc version: 2f7393a47307a16f8cee44a37b262e8b81021e3e
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-43-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 19.61 GiB
Name: piranha
ID: 3SY6:AAEL:NLUO:4BTD:U5ZK:AMWA:PNGQ:4ZVM:F7S4:7GFH:E2KG:V32H
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
 172.17.11.100:5000
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

[ggaugry]

Hi,

Any news about this issue?

[sanimej]

@ggaugry Are you using swarm mode with attachable networks or the classic swarm (which also needs an external KV store) ?

[ggaugry]

@sanimej : I use swarm mode with attachable option.

[ggaugry]

I still have the problem. Results of a dig for DNS "mysql":

root@24c350f685ef:/home/nightmare# dig mysql

; <<>> DiG 9.9.5-9+deb8u10-Debian <<>> mysql
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13379
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;mysql. IN A

;; ANSWER SECTION:
mysql. 600 IN A 192.168.14.17
mysql. 600 IN A 192.168.14.25
mysql. 600 IN A 192.168.14.8

;; Query time: 0 msec
;; SERVER: 127.0.0.11#53(127.0.0.11)
;; WHEN: Mon May 22 09:44:01 UTC 2017
;; MSG SIZE rcvd: 86

[sudo-bmitch]

I think I ran into this one myself. Overlay networks were managed by swarm mode and attachable. We migrated containers from running via docker-compose and classic swarm to deploying services in swarm mode. After this migration, from one node it was resolving what I believe was the old address along with the new VIP address. The other node only resolved the new VIP address.

Recreating the container that happened to have the invalid IP and the service with the DNS name we were trying to correct (using docker service update --force) did not remove the stale mapping. In the end we bounced the docker daemon with the bad DNS mapping and it came back up with the correct entries.

Our Docker version:

Client:
 Version:      17.03.1-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Fri Mar 24 00:36:45 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.1-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Fri Mar 24 00:36:45 2017
 OS/Arch:      linux/amd64
 Experimental: false

Our servers are running RHEL 7.2 with the 3.10.0-327 kernel.

Unfortunately I don't believe we can reliably recreate this, and it happened in production during a limited outage window so I didn't have time to gather logs.

As best I can tell, there are situations where stopping a container on an attachable overlay network doesn't propagate the change the other nodes in the swarm.

[danielmrosa]

Hi,
Does anyone have news about this issue ? I read in another thread that it´s a know issue and one guy from docker was working to develop a patch, but I don´t have additional information.
Our version is the same 17.03.1-ce

[cpuguy83]

@danielmrosa 17.06 has a lot of patches and should work much better all-around wrt service discovery and networking.
GA is imminent but RC's are available on the test channel.

[danielmrosa]

@cpuguy83
Hi Brian, thanks a lot for your feedback, let´s try 17.06

[ggaugry]

Hi all

Still the same issue with docker 17.06.01-ce:

Exemple inside one container:

root@b617061009ee:/# getent hosts bazoocaster-sfr0092vu
192.168.14.20 bazoocaster-sfr0092vu
192.168.14.10 bazoocaster-sfr0092vu

When does it happens?
We realised this happens when we redeploy a new container version. Exemple:

• Stop container with name "test"
• Disconnect it from the overlay network
• Rename it with name "test_OLD"
• Redeploy a new container "test" (attached on the overlay network)

This is the scenario where we see sometimes these DNS problems (most of the times it works well, but sometimes the DNS resolution goes crazy)

[ggaugry]

The only way to fix this DNS problem is to remove the problematic node (the one which appears to have 2 Ips) from the Swarm and rejoin it again.

[danielmrosa]

Hi All,
We continue facing this problem on 17.06.01-ce too.
It seems that the problem was not solved yet. Does someone has some news about ? I´m not sure but probably @fcrisciani are working on this issue.

[fcrisciani]

@ggaugry I still have to go through the rename part, that can create issues.

@danielmrosa can you share something more about it? Do you have a set of steps that are consistently reproducing the problem?

[ggaugry]

@fcrisciani : thanks for the answer. I was saying earlier that remove/re-join the Swarm was a workaround but it actually doesn't work. Do you have any idea on how I can clean up the wrong DNS entries?

[ggaugry]

FYI:
root@e429d14e3de9:/# getent hosts esus-sfr0092vu
192.168.14.22 esus-sfr0092vu
192.168.14.21 esus-sfr0092vu

The 2 IPs shown are the 2 IPs actually used by the containers running on the target machine:
"Containers": {
"d84bf6f6c871ee19a0fc946b927bff0e5566553d5370cb0d677dbc1edd55da17": {
"Name": "bazoocaster-sfr0092vu",
"EndpointID": "e3a6b2ea98fd9729989ad096cd9eeb6162193f4afdcfef4c6d51c0195df90352",
"MacAddress": "02:42:c0:a8:0e:16",
"IPv4Address": "192.168.14.22/24",
"IPv6Address": ""
},
"d9fc0a52feea016e104140132062db27527de97ce8d3d5272d39ebc60ee61139": {
"Name": "esus-sfr0092vu",
"EndpointID": "74c674d46fa980f5945ea9731259561c6c170792d8b698d3c12db20480297cc0",
"MacAddress": "02:42:c0:a8:0e:15",
"IPv4Address": "192.168.14.21/24",
"IPv6Address": ""
}
},