k3 does not support mDNS

[jdmarshall]

Environmental Info:
K3s Version:

k3s version v1.18.6+k3s1 (6f56fa1)

Node(s) CPU architecture, OS, and Version:

intended master:
Linux nanopct4 5.4.49-rockchip64 #20.05.7 SMP PREEMPT Sun Jun 28 18:17:52 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux
intended worker:
Linux nanopineo2 5.4.45-sunxi64 #20.05.4 SMP Sun Jun 14 23:12:39 CEST 2020 aarch64 aarch64 aarch64 GNU/Linux

Both are Armbian, fresh installs

Cluster Configuration:

under construction

Describe the bug:

The documentation https://rancher.com/docs/k3s/latest/en/quick-start/ says that I can create a worker by running the curl | sh with a couple of environment variables. The machines don't seem to see each other.

Steps To Reproduce:

• Installed K3s:

  curl -sfL https://get.k3s.io | KS3_URL=https://192.168.2.9:6443 K3S_TOKEN=**** sh -

• checked nodes

  k3s kubectl get nodes

Expected behavior:

Multiple nodes shown, or errors during initialization (neither happened)

Actual behavior:

[INFO]  Finding release for channel stable
[INFO]  Using v1.18.6+k3s1 as release
[INFO]  Downloading hash https://github.com/rancher/k3s/releases/download/v1.18.6+k3s1/sha256sum-arm64.txt
[INFO]  Downloading binary https://github.com/rancher/k3s/releases/download/v1.18.6+k3s1/k3s-arm64
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s
root@seven:~# k3s kubectl get nodes
NAME    STATUS   ROLES    AGE   VERSION
seven   Ready    master   65s   v1.18.6+k3s1

Additional context / logs:

is the entire contents of the node-token file supposed to go into the environment variable? This is never explained one way or another.

[brandond]

You should install without K3S_URL on the server, and then specify K3S_URL on the agents, with the URL pointing at the server node. Use the same K3S_TOKEN value for all nodes. If the agents are not showing up, check the logs on the agent, and confirm that port 6443 is open on the server.

[jdmarshall]

I went through the k3s-ansible setup with similar results.

From the logs, it's getting a connection reset trying to contact https://master:6443/cacerts, even though I can fetch the same URL from the command line on the agent machine.

Seems like maybe traefik is not behaving properly.

[brandond]

Traefik does not handle traffic to port 6443, this port is used by k3s itself. You're saying you can curl that URL from a shell on the agent, but the k3s agent itself is getting errors reaching it?

[jdmarshall]

syslog:Aug 13 04:48:46 localhost k3s[5715]: time="2020-08-13T04:48:46.269655934Z" level=info msg="Running load balancer 127.0.0.1:34187 -> [two.local:6443]"
syslog:Aug 13 06:54:18 localhost k3s[5715]: time="2020-08-13T06:54:18.364644384Z" level=error msg="failed to get CA certs at https://127.0.0.1:34187/cacerts: Get https://127.0.0.1:34187/cacerts: read tcp 127.0.0.1:45044->127.0.0.1:34187: read: connection reset by peer"

two.local:6443 works fine. localhost:34187 gets a connection reset, same as the logs.

[brandond]

Can you provide the exact commands that you're running to install on both the server and agent?

[jdmarshall]

The logs above are from running k3s-ansible, which stalls forever waiting to validate that everything is working. Of course this is Armbian, so there are some tweaks, so it doesn't run the raspbian role, or add the boot flags (there is no such file, and the file that exists has a different format, and the flags all appear to be on by default anyway).

Any logs you want to see?

[brandond]

Can you try running the uninstall scripts to clean k3s off both nodes, then run the exact steps from the Quick-Start guide and provide both the output from the install scripts, and the resulting logs?

[jdmarshall]

server:

curl -sfL https://get.k3s.io | sh -
[INFO]  Finding release for channel stable
[INFO]  Using v1.18.6+k3s1 as release
[INFO]  Downloading hash https://github.com/rancher/k3s/releases/download/v1.18.6+k3s1/sha256sum-arm64.txt
[INFO]  Downloading binary https://github.com/rancher/k3s/releases/download/v1.18.6+k3s1/k3s-arm64
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s

agent:

curl -sfL https://get.k3s.io | K3S_URL=https://two.local:6443 K3S_TOKEN=K10693ca5cac380465f1eeb788dd84fa3e0fb7e5b4b3df645c8846b0773b67abc19::server:bc6bea566731a735c5f9a33ca6ebf986 sh -

[INFO]  Finding release for channel stable
[INFO]  Using v1.18.6+k3s1 as release
[INFO]  Downloading hash https://github.com/rancher/k3s/releases/download/v1.18.6+k3s1/sha256sum-arm64.txt
[INFO]  Downloading binary https://github.com/rancher/k3s/releases/download/v1.18.6+k3s1/k3s-arm64
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s-agent.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s-agent.service
[INFO]  systemd: Enabling k3s-agent unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s-agent.service → /etc/systemd/system/k3s-agent.service.
[INFO]  systemd: Starting k3s-agent

back on server:

remote@two:~$ sudo kubectl get nodes
NAME        STATUS   ROLES    AGE    VERSION
two.local   Ready    master   6m1s   v1.18.6+k3s1

It doesn't seem like you get any diagnostic information to speak of on stdout or stderr. Anything other than good news is in /var/log.

find 'two.local' in /var/log:

./syslog:Aug 21 01:17:35 localhost k3s[33580]: time="2020-08-21T01:17:35.091644053Z" level=info msg="Running load balancer 127.0.0.1:34307 -> [two.local:6443]"
./syslog:Aug 21 01:19:16 localhost k3s[33842]: time="2020-08-21T01:19:16.146682382Z" level=info msg="Running load balancer 127.0.0.1:45349 -> [two.local:6443]"
./syslog:Aug 21 01:33:13 localhost k3s[34191]: time="2020-08-21T01:33:13.905829168Z" level=info msg="Running load balancer 127.0.0.1:34639 -> [two.local:6443]"

remote@t4:/var/log$ sudo find . -type f | xargs sudo grep 34639
90 lines of the form:

./syslog:Aug 21 01:39:36 localhost k3s[34191]: time="2020-08-21T01:39:36.371805738Z" level=error msg="failed to get CA certs at https://127.0.0.1:34639/cacerts: Get https://127.0.0.1:34639/cacerts: read tcp 127.0.0.1:59354->127.0.0.1:34639: read: connection reset by peer"

[jdmarshall]

After adding a static entry for the server in the agent's /etc/hosts file, the agent succeeds in registering itself.

it turns out Go does not like mDNS and it's currently listed as won't fix: golang/go#35067

Pretty much any other bash command, including curl, is going to honor nsswitch.conf. Go does not, meaning a whole lot of reasonable sanity checks don't provide useful data.

[brandond]

Ah, yes, that would do it. I was assuming you had actual unicast name resolution working for the two hostnames.

[jdmarshall]

I ended up relocating DHCP to another machine, configured to report itself as the DNS server for DHCP clients.

[jdmarshall]

The Go folks are suggesting that the DNS resolver that has this problem is not the Go default behavior, and that the default lookup mechanism works fine with nsswitch.conf.

golang/go#35067

[brandond]

The docs say:

By default the pure Go resolver is used, because a blocked DNS request consumes only a goroutine, while a blocked C call consumes an operating system thread. When cgo is available, the cgo-based resolver is used instead under a variety of conditions: on systems that do not let programs make direct DNS requests (OS X), when the LOCALDOMAIN environment variable is present (even if empty), when the RES_OPTIONS or HOSTALIASES environment variable is non-empty, when the ASR_CONFIG environment variable is non-empty (OpenBSD only), when /etc/resolv.conf or /etc/nsswitch.conf specify the use of features that the Go resolver does not implement, and when the name being looked up ends in .local or is an mDNS name.

Either way, I would not rely on mDNS for a variety of reasons. Get unicast DNS working, or set up hosts file entries.