Imcomplete gethostbyname response

[picnoir]

There's something wrong with the getaddrinfo operation.

Running hostname --fqdn and dumping the nscd socket:

NSNCD:

mars 07 14:24:18 framework sockdump[96606]: 14:24:18.768 >>> process hostname [98811 -> 70129] len 18(18)
mars 07 14:24:18 framework sockdump[96606]: 0000  02 00 00 00 0d 00 00 00  06 00 00 00 68 6f 73 74  ............host
mars 07 14:24:18 framework sockdump[96606]: 0010  73 00                                             s.
mars 07 14:24:18 framework sockdump[96606]: 14:24:18.768 >>> process hostname [98811 -> 70129] len 22(22)
mars 07 14:24:18 framework sockdump[96606]: 0000  02 00 00 00 05 00 00 00  0a 00 00 00 66 72 61 6d  ............fram
mars 07 14:24:18 framework sockdump[96606]: 0010  65 77 6f 72 6b 00                                 ework.


NSCD:

mars 07 14:25:53 framework sockdump[96606]: 14:25:53.784 >>> process hostname [101011 -> 100886] len 22(22)
mars 07 14:25:53 framework sockdump[96606]: 0000  02 00 00 00 05 00 00 00  0a 00 00 00 66 72 61 6d  ............fram
mars 07 14:25:53 framework sockdump[96606]: 0010  65 77 6f 72 6b 00                                 ework.
mars 07 14:25:53 framework sockdump[96606]: 14:25:53.784 >>> process nscd [100886 -> 101011] len 90(90)
mars 07 14:25:53 framework sockdump[96606]: 0000  02 00 00 00 01 00 00 00  1c 00 00 00 01 00 00 00  ................
mars 07 14:25:53 framework sockdump[96606]: 0010  0a 00 00 00 10 00 00 00  01 00 00 00 00 00 00 00  ................
mars 07 14:25:53 framework sockdump[96606]: 0020  66 72 61 6d 65 77 6f 72  6b 2e 61 6c 74 65 72 6e  framework.altern
mars 07 14:25:53 framework sockdump[96606]: 0030  61 74 69 76 65 62 69 74  2e 66 72 00 0a 00 00 00  ativebit.fr.....
mars 07 14:25:53 framework sockdump[96606]: 0040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 01  ................
mars 07 14:25:53 framework sockdump[96606]: 0050  66 72 61 6d 65 77 6f 72  6b 00                    framework.

Nixpkgs issue: NixOS/nixpkgs#196934

[picnoir]

We're responding with the request's hostname:

nsncd/src/handlers.rs

Line 128 in 47e580f

canon_name: hostname.to_string(),

. That's probably the root issue here.

There's likely more informations in the canonname field of the AddrInfo struct https://docs.rs/dns-lookup/latest/dns_lookup/struct.AddrInfo.html

[picnoir]

Wait a minute, it actually seems like nsncd is not responding to hostname at all actually!

Playing a bit more with nscd and this sock dump systemd service:

    systemd.services.sockdump = {
      wantedBy = [ "multi-user.target" ];
      path = [
        # necessary for bcc to unpack kernel headers and invoke modprobe
        pkgs.gnutar
        pkgs.xz.bin
        pkgs.kmod
      ];
      environment.PYTHONUNBUFFERED = "1";

      serviceConfig = {
        ExecStart = "${pkgs.sockdump}/bin/sockdump /var/run/nscd/socket";
        Restart = "on-failure";
        RestartSec = "1";
        Type = "simple";
      };
    };

I don't see any nsncd response on the socket (while I do see nscd's ones). I can still resolve test.localhost though, indicating the NSS setup still works. No error logs in nsncd.

Is it:

1. Nsncd being completely broken somehow (it used to work when we released the thing).
2. An issue with sockdump.

I need to instrument my nsncd a bit more to debug that further. I'll come back to this.

[picnoir]

So, it seems like hostname --fqdn is using the GETHOSTBYNAMEv6 operation in the end.

Sadly for us, the getaddrinfo function provided by the dns_lookup crate is not returning the canonical name :(

[picnoir]

Okay, I found the issue. In GETHOSTBYNAMEV6 (and v4), we're using getaddrinfo instead of gethostbyname (https://man7.org/linux/man-pages/man3/gethostbyname.3.html).

There's no bindings for that function in nix nor in dns_lookup. Looks like we need to write the ffi call ourselves.

[picnoir]

So, overall, to fix this:

1. write the gethostbyname FFI call to a Nix fork. EDIT: let's vendor that in to start with. We'd need to port that across different Unix archs if we were up to upstream that to Nix directly.
2. temporarily point Nsncd to this Nix fork.
3. when the Nix PR gets merged, move Nsncd back to mainline Nix.

Bonus points for:

• Write a getaddrinfo FFI to Nix.
• Drop the dns_lookup crate dependency.

[picnoir]

^ Yet another bug report in Nixpkgs, we procrastinated this long enough, it needs to be fixed.

Flokli and I booked a pairing session to fix that tomorrow afternoon.

[RaitoBezarius]

Just encountered this bug in production :<

[picnoir]

Erf. A fix is on the way there: https://github.com/nix-community/nsncd/tree/nin/ghbn

Doing this on the side, I can't give a clear ETA, but massive plot twist aside, we're almost there.