iPXE boot doesn't load cloud-config (DHCP based DNS is not being used)

[gizmotronic]

RancherOS Version: (ros os version)
0.9.2-rc2 and later

Where are you running RancherOS? (docker-machine, AWS, GCE, baremetal, etc.)
vSphere Hypervisor 6.5.0 (ESXi) virtual machine, iPXE, diskless

RancherOS loads on a diskless VM using iPXE, but starting with 0.9.2-rc2, it's unable to load the cloud-config specified using the rancher.cloud_init.datasources kernel command line parameter.

I've isolated the problem to changeset 79a7e59. Reverting this change allows 1.0.0 to work normally.

[SvenDowideit]

oh wonderful - that change was made to fix AWS cloud-init.

[SvenDowideit]

@gizmotronic - can you post the log file of boot with rancher.debug=true? I'm curious to see what's going on.

[gizmotronic]

@SvenDowideit It took me a bit to figure out how to get past the rate limited printk, but here it is: rancheros-dmesg.txt

This was really helpful. My cloud-config can't load because the host it's coming from is on my internal network, which can't be resolved by Google DNS. I don't expect it to be using Google because my DHCP server is providing the necessary (private) DNS configuration.

In case some future reader is wondering how I turned off the rate limiting, I added printk.devkmsg=on to my kernel command line parameters.

[SvenDowideit]

yes, that's not good. can you add your local dns server seting to the boot cmdline for now?

[SvenDowideit]

this might be the root cause of something I'm trcking down atm - thank you for the analysis

[gizmotronic]

I've confirmed that setting the local DNS server is an effective workaround.

[SvenDowideit]

argh! it works for my server (on an installed disk), guess I need to kick it with pxe-dust

[root@rancher rancher]# cat /etc/resolv.conf 
# Generated by dhcpcd from eth0.dhcp, eth1.dhcp
# /etc/resolv.conf.head can replace this line
domain home.org.au
nameserver 10.10.10.1
nameserver 8.8.8.8
# /etc/resolv.conf.tail can replace this line
[root@rancher rancher]# ros -v
ros version v1.0.1
[root@rancher rancher]# system-docker exec -it network dhcpcd -U eth0
broadcast_address=10.10.10.255
dhcp_lease_time=864000
dhcp_message_type=5
dhcp_server_identifier=10.10.10.1
domain_name=home.org.au
domain_name_servers='10.10.10.1 8.8.8.8'
ip_address=10.10.10.2
network_number=10.10.10.0
routers=10.10.10.1
subnet_cidr=24
subnet_mask=255.255.255.0
eth0: dhcp6_dump: No such file or directory

[SvenDowideit]

yup, and the dns servers are also set and working at cloud-init-save when I pixieboot it with a local dns only datasource url :(

DEBU[0035] framesize: 110                               
cloud-init_1 | > time="2017-05-17T05:04:13Z" level=info msg="Running DHCP on eth1: dhcpcd -MA4 -e force_hostname=true eth1" 
DEBU[0035] framesize: 110                               
cloud-init_1 | > time="2017-05-17T05:04:13Z" level=info msg="Running DHCP on eth0: dhcpcd -MA4 -e force_hostname=true eth0" 
DEBU[0035] framesize: 42                                
cloud-init_1 | sending commands to master dhcpcd process
[   66.748924] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[   67.130548] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   69.623691] tg3 0000:03:00.1 eth1: Link is up at 1000 Mbps, full duplex
[   69.946168] tg3 0000:03:00.1 eth1: Flow control is on for TX and on for RX
[   70.281179] tg3 0000:03:00.1 eth1: EEE is disabled
[   70.521612] tg3 0000:03:00.0 eth0: Link is up at 1000 Mbps, full duplex
[   70.843881] tg3 0000:03:00.0 eth0: Flow control is on for TX and on for RX
[   71.178719] tg3 0000:03:00.0 eth0: EEE is disabled
[   71.419367] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   71.724858] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   72.134437] time="2017-05-17T05:04:20Z" level=info msg="Apply Network Config SyncHostname" 
DEBU[00[   72.541819] time="2017-05-17T05:04:20Z" level=debug msg="datasources that will be consided: []string{\"url:http://mini/test.yml\"}" 
41] framesize: 82                                [   73.123418] time="2017-05-17T05:04:21Z" level=info msg="cloud-init: Checking availability of \"url\"\n" 

cloud-init_1 | > time="2017-05-17T05:04:20Z" level=info msg="Apply Network Config SyncHostname" 
DEBU[0042] framesize: 123                               
cloud-init_1 | > time="2017-05-17T05:04:20Z" level=debug msg="datasources that will be consided: []string{\"url:http://mini/test.yml\"}" 
DEBU[0042] framesize: 95                                
cloud-init_1 | > time="2017-05-17T05:04:21Z" level=info msg="cloud-init: Checking availability of \"url\"\n" 
[   74.122492] time="2017-05-17T05:04:22Z" level=info msg="cloud-init: Datasource unavailable, skipping: url: http://mini/test.yml (lastError: Unable to fetch data: Get http://mini/test.yml: dial tcp 10.10.10.14:80: getsockopt: connection refused)" 
DEBU[0043] framesize: 237                               
cloud-init_1 | > time="2017-05-17T05:04:22Z" level=info msg="cloud-init: Datasource unavailable, skipping: url: http://mini/test.yml (lastError: Unable to fetch data: Get http://mini/test.yml: dial tcp 10.10.10.14:80: getsockopt: connection refused)" 

[SvenDowideit]

@gizmotronic I guess the next question is - what do oyu get when you run system-docker exec -it network dhcpcd -U eth0