net: issue with DNS response > 512 bytes (cannot unmarshal DNS message)

[kvaps]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.6.2 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH=""
GORACE=""
GOROOT="/usr/lib/go-1.6"
GOTOOLDIR="/usr/lib/go-1.6/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT="1"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

What did you do?

I have get-ip.go with this code:

package main

import (
    "fmt"
    "net"
)

func main() {
    addr, err := net.LookupHost("storage.googleapis.com")
    fmt.Println(addr, err)
}

What did you expect to see?

# ./get-ip 
[216.58.201.112 2a00:1450:4014:801::2010] <nil>

What did you see instead?

# ./get-ip 
dial tcp: lookup storage.googleapis.com on 10.36.1.10:53: cannot unmarshal DNS message

Additional information:

It works with GODEBUG=netdns=cgo, but not with GODEBUG=netdns=go

# GODEBUG=netdns=cgo ./get-ip 
[172.217.23.208 2a00:1450:4014:80c::2010] <nil>

# GODEBUG=netdns=go ./get-ip 
dial tcp: lookup storage.googleapis.com on 10.36.1.10:53: cannot unmarshal DNS message

It also started working after flushing dns-cache on my dns-server.

So I made capture of answer from dns-server:

before: (got error)

Domain Name System (response)
    [Request In: 2]
    [Time: 0.000410000 seconds]
    Transaction ID: 0xc4c9
    Flags: 0x8180 Standard query response, No error
    Questions: 1
    Answer RRs: 2
    Authority RRs: 13
    Additional RRs: 14
    Queries
        storage.googleapis.com: type AAAA, class IN
    Answers
        storage.googleapis.com: type CNAME, class IN, cname storage.l.googleusercontent.com
        storage.l.googleusercontent.com: type AAAA, class IN, addr 2a00:1450:4014:80d::2010
    Authoritative nameservers
        com: type NS, class IN, ns h.gtld-servers.net
        com: type NS, class IN, ns c.gtld-servers.net
        com: type NS, class IN, ns k.gtld-servers.net
        com: type NS, class IN, ns b.gtld-servers.net
        com: type NS, class IN, ns d.gtld-servers.net
        com: type NS, class IN, ns f.gtld-servers.net
        com: type NS, class IN, ns g.gtld-servers.net
        com: type NS, class IN, ns m.gtld-servers.net
        com: type NS, class IN, ns i.gtld-servers.net
        com: type NS, class IN, ns j.gtld-servers.net
        com: type NS, class IN, ns e.gtld-servers.net
        com: type NS, class IN, ns l.gtld-servers.net
        com: type NS, class IN, ns a.gtld-servers.net
    Additional records
        storage.l.googleusercontent.com: type A, class IN, addr 216.58.201.112
        h.gtld-servers.net: type A, class IN, addr 192.54.112.30
        c.gtld-servers.net: type A, class IN, addr 192.26.92.30
        k.gtld-servers.net: type A, class IN, addr 192.52.178.30
        b.gtld-servers.net: type A, class IN, addr 192.33.14.30
        d.gtld-servers.net: type A, class IN, addr 192.31.80.30
        f.gtld-servers.net: type A, class IN, addr 192.35.51.30
        g.gtld-servers.net: type A, class IN, addr 192.42.93.30
        m.gtld-servers.net: type A, class IN, addr 192.55.83.30
        i.gtld-servers.net: type A, class IN, addr 192.43.172.30
        j.gtld-servers.net: type A, class IN, addr 192.48.79.30
        e.gtld-servers.net: type A, class IN, addr 192.12.94.30
        l.gtld-servers.net: type A, class IN, addr 192.41.162.30
        a.gtld-servers.net: type A, class IN, addr 192.5.6.30

after: (is working)

Domain Name System (response)
    [Request In: 1]
    [Time: 0.000746000 seconds]
    Transaction ID: 0x86fe
    Flags: 0x8180 Standard query response, No error
    Questions: 1
    Answer RRs: 2
    Authority RRs: 0
    Additional RRs: 0
    Queries
        storage.googleapis.com: type A, class IN
    Answers
        storage.googleapis.com: type CNAME, class IN, cname storage.l.googleusercontent.com
        storage.l.googleusercontent.com: type A, class IN, addr 216.58.201.112

And hexdump of package:

before: (got error)

0000   86 ea 80 1a 4b c3 9a cc 60 09 5f 0b 08 00 45 00  ....K...`._...E.
0010   02 2e 05 b1 00 00 40 11 5c 59 0a 24 01 0a 0a 24  ......@.\Y.$...$
0020   01 64 00 35 b2 cc 02 1a 18 e1 82 0a 81 80 00 01  .d.5............
0030   00 02 00 0d 00 0d 07 73 74 6f 72 61 67 65 0a 67  .......storage.g
0040   6f 6f 67 6c 65 61 70 69 73 03 63 6f 6d 00 00 01  oogleapis.com...
0050   00 01 c0 0c 00 05 00 01 00 00 0a af 00 1e 07 73  ...............s
0060   74 6f 72 61 67 65 01 6c 11 67 6f 6f 67 6c 65 75  torage.l.googleu
0070   73 65 72 63 6f 6e 74 65 6e 74 c0 1f c0 34 00 01  sercontent...4..
0080   00 01 00 00 00 63 00 04 d8 3a c9 70 c0 1f 00 02  .....c...:.p....
0090   00 01 00 01 52 3a 00 14 01 61 0c 67 74 6c 64 2d  ....R:...a.gtld-
00a0   73 65 72 76 65 72 73 03 6e 65 74 00 c0 1f 00 02  servers.net.....
00b0   00 01 00 01 52 3a 00 04 01 68 c0 70 c0 1f 00 02  ....R:...h.p....
00c0   00 01 00 01 52 3a 00 04 01 63 c0 70 c0 1f 00 02  ....R:...c.p....
00d0   00 01 00 01 52 3a 00 04 01 6b c0 70 c0 1f 00 02  ....R:...k.p....
00e0   00 01 00 01 52 3a 00 04 01 62 c0 70 c0 1f 00 02  ....R:...b.p....
00f0   00 01 00 01 52 3a 00 04 01 64 c0 70 c0 1f 00 02  ....R:...d.p....
0100   00 01 00 01 52 3a 00 04 01 66 c0 70 c0 1f 00 02  ....R:...f.p....
0110   00 01 00 01 52 3a 00 04 01 67 c0 70 c0 1f 00 02  ....R:...g.p....
0120   00 01 00 01 52 3a 00 04 01 6d c0 70 c0 1f 00 02  ....R:...m.p....
0130   00 01 00 01 52 3a 00 04 01 69 c0 70 c0 1f 00 02  ....R:...i.p....
0140   00 01 00 01 52 3a 00 04 01 6a c0 70 c0 1f 00 02  ....R:...j.p....
0150   00 01 00 01 52 3a 00 04 01 65 c0 70 c0 1f 00 02  ....R:...e.p....
0160   00 01 00 01 52 3a 00 04 01 6c c0 70 c0 6e 00 01  ....R:...l.p.n..
0170   00 01 00 00 ca ea 00 04 c0 05 06 1e c0 8e 00 01  ................
0180   00 01 00 00 ca ea 00 04 c0 36 70 1e c0 9e 00 01  .........6p.....
0190   00 01 00 00 ca ea 00 04 c0 1a 5c 1e c0 ae 00 01  ..........\.....
01a0   00 01 00 00 ca ea 00 04 c0 34 b2 1e c0 be 00 01  .........4......
01b0   00 01 00 00 ca ea 00 04 c0 21 0e 1e c0 ce 00 01  .........!......
01c0   00 01 00 00 ca ea 00 04 c0 1f 50 1e c0 de 00 01  ..........P.....
01d0   00 01 00 00 ca ea 00 04 c0 23 33 1e c0 ee 00 01  .........#3.....
01e0   00 01 00 00 ca ea 00 04 c0 2a 5d 1e c0 fe 00 01  .........*].....
01f0   00 01 00 00 ca ea 00 04 c0 37 53 1e c1 0e 00 01  .........7S.....
0200   00 01 00 00 ca ea 00 04 c0 2b ac 1e c1 1e 00 01  .........+......
0210   00 01 00 00 ca ea 00 04 c0 30 4f 1e c1 2e 00 01  .........0O.....
0220   00 01 00 00 ca ea 00 04 c0 0c 5e 1e c1 3e 00 01  ..........^..>..
0230   00 01 00 00 ca ea 00 04 c0 29 a2 1e              .........)..

after: (is working)

0000   86 ea 80 1a 4b c3 9a cc 60 09 5f 0b 08 00 45 00  ....K...`._...E.
0010   00 7e 07 00 00 00 40 11 5c ba 0a 24 01 0a 0a 24  .~....@.\..$...$
0020   01 64 00 35 93 32 00 6a 17 31 86 fe 81 80 00 01  .d.5.2.j.1......
0030   00 02 00 00 00 00 07 73 74 6f 72 61 67 65 0a 67  .......storage.g
0040   6f 6f 67 6c 65 61 70 69 73 03 63 6f 6d 00 00 01  oogleapis.com...
0050   00 01 c0 0c 00 05 00 01 00 00 0a 40 00 1e 07 73  ...........@...s
0060   74 6f 72 61 67 65 01 6c 11 67 6f 6f 67 6c 65 75  torage.l.googleu
0070   73 65 72 63 6f 6e 74 65 6e 74 c0 1f c0 34 00 01  sercontent...4..
0080   00 01 00 00 00 26 00 04 d8 3a c9 70              .....&...:.p

I presume that it connected with:

• net: builtin DNS stub resolver fails to parse responses from consul with "cannot unmarshal DNS message" #11070
• https://stackoverflow.com/questions/38770700

[ianlancetaylor]

Have you tried the current Go release, 1.8.3?

[kvaps]

Yes, first I caught this bug with kubeadm, it is compiled by 1.8.3:

# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.1", GitCommit:"1dc5c66f5dd61da08412a74221ecc79208c2165b", GitTreeState:"clean", BuildDate:"2017-07-14T01:48:01Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

After this, I installed golang from the ubuntu xenial repo, and tried to repeat it (as I reported)

[ianlancetaylor]

CC @mdempsky @iangudger

[iangudger]

Both of the hex dumps have prefixes before the message. As a result, neither is an RFC 1035 DNS message.

The first message does not match the text you provided, so lets focus on the second message which both matches the text and has the same problem.

The first two bytes of the DNS message should be the message ID (0x86fe). If you look closely, those two bytes don't show up until most of the way through line three. That is where the actual DNS message starts.

I am not sure what the prefix is, but it isn't compatible with the current DNS library.

[kvaps]

Hi @iangudger, thanks for answer. But provided hex dump is dump of tcp-package, not only dns reply.
If you want I can provide you full dump of tcp session.

[kvaps]

pcaps.zip

[iangudger]

@kvaps, what do you mean by tcp-package?

[kvaps]

@ianlancetaylor, I mean that this hex dump is contain the tcp/ip headers like source ip, dst ip addresses and other service information.

You can use wireshark for open cap-files attached to my previous message and see all session between dns-client and dns-server.

I can also attach the hex dump without tcp headers:

0000   5f fb 81 80 00 01 00 02 00 0d 00 0d 07 73 74 6f  _............sto
0010   72 61 67 65 0a 67 6f 6f 67 6c 65 61 70 69 73 03  rage.googleapis.
0020   63 6f 6d 00 00 01 00 01 c0 0c 00 05 00 01 00 00  com.............
0030   0a c1 00 1e 07 73 74 6f 72 61 67 65 01 6c 11 67  .....storage.l.g
0040   6f 6f 67 6c 65 75 73 65 72 63 6f 6e 74 65 6e 74  oogleusercontent
0050   c0 1f c0 34 00 01 00 01 00 00 00 75 00 04 d8 3a  ...4.......u...:
0060   c9 70 c0 1f 00 02 00 01 00 01 52 4c 00 14 01 69  .p........RL...i
0070   0c 67 74 6c 64 2d 73 65 72 76 65 72 73 03 6e 65  .gtld-servers.ne
0080   74 00 c0 1f 00 02 00 01 00 01 52 4c 00 04 01 6a  t.........RL...j
0090   c0 70 c0 1f 00 02 00 01 00 01 52 4c 00 04 01 65  .p........RL...e
00a0   c0 70 c0 1f 00 02 00 01 00 01 52 4c 00 04 01 6c  .p........RL...l
00b0   c0 70 c0 1f 00 02 00 01 00 01 52 4c 00 04 01 61  .p........RL...a
00c0   c0 70 c0 1f 00 02 00 01 00 01 52 4c 00 04 01 68  .p........RL...h
00d0   c0 70 c0 1f 00 02 00 01 00 01 52 4c 00 04 01 63  .p........RL...c
00e0   c0 70 c0 1f 00 02 00 01 00 01 52 4c 00 04 01 6b  .p........RL...k
00f0   c0 70 c0 1f 00 02 00 01 00 01 52 4c 00 04 01 62  .p........RL...b
0100   c0 70 c0 1f 00 02 00 01 00 01 52 4c 00 04 01 64  .p........RL...d
0110   c0 70 c0 1f 00 02 00 01 00 01 52 4c 00 04 01 66  .p........RL...f
0120   c0 70 c0 1f 00 02 00 01 00 01 52 4c 00 04 01 67  .p........RL...g
0130   c0 70 c0 1f 00 02 00 01 00 01 52 4c 00 04 01 6d  .p........RL...m
0140   c0 70 c0 6e 00 01 00 01 00 00 ca fc 00 04 c0 2b  .p.n...........+
0150   ac 1e c0 8e 00 01 00 01 00 00 ca fc 00 04 c0 30  ...............0
0160   4f 1e c0 9e 00 01 00 01 00 00 ca fc 00 04 c0 0c  O...............
0170   5e 1e c0 ae 00 01 00 01 00 00 ca fc 00 04 c0 29  ^..............)
0180   a2 1e c0 be 00 01 00 01 00 00 ca fc 00 04 c0 05  ................
0190   06 1e c0 ce 00 01 00 01 00 00 ca fc 00 04 c0 36  ...............6
01a0   70 1e c0 de 00 01 00 01 00 00 ca fc 00 04 c0 1a  p...............
01b0   5c 1e c0 ee 00 01 00 01 00 00 ca fc 00 04 c0 34  \..............4
01c0   b2 1e c0 fe 00 01 00 01 00 00 ca fc 00 04 c0 21  ...............!
01d0   0e 1e c1 0e 00 01 00 01 00 00 ca fc 00 04 c0 1f  ................
01e0   50 1e c1 1e 00 01 00 01 00 00 ca fc 00 04 c0 23  P..............#
01f0   33 1e c1 2e 00 01 00 01 00 00 ca fc 00 04 c0 2a  3..............*
0200   5d 1e c1 3e 00 01 00 01 00 00 ca fc 00 04 c0 37  ]..>...........7
0210   53 1e                                            S.

0000   3c f9 81 80 00 01 00 02 00 00 00 00 07 73 74 6f  <............sto
0010   72 61 67 65 0a 67 6f 6f 67 6c 65 61 70 69 73 03  rage.googleapis.
0020   63 6f 6d 00 00 01 00 01 c0 0c 00 05 00 01 00 00  com.............
0030   04 f3 00 1e 07 73 74 6f 72 61 67 65 01 6c 11 67  .....storage.l.g
0040   6f 6f 67 6c 65 75 73 65 72 63 6f 6e 74 65 6e 74  oogleusercontent
0050   c0 1f c0 34 00 01 00 01 00 00 00 0a 00 04 ac d9  ...4............
0060   17 d0                                            ..

[iangudger]

@kvaps, I got confused when you were talking about TCP. The hex dumps you posted did not look like TCP DNS packets. I have confirmed from your packet capture that these messages were all exchanged over UDP, and they do look like UDP DNS packets.

I am actually working on a solution to this problem. I have confirmed that the problematic message parses just fine with the new DNS library (golang.org/x/net/dns/dnsmessage). I am working on converting the standard net package to use it for DNS resolution. I am aiming to get this in Go 1.10. Once that lands, this problem should go away.

[kvaps]

@iangudger, No problem, thank you!