dual NIC netconfig breaks 0.9.1 -> 0.9.2 and 1.0.0

[joshuacox]

**RancherOS Version: (ros os version)**0.9.2-0.1.0

Where are you running RancherOS? (docker-machine, AWS, GCE, baremetal, etc.)
KVM VM with two virtual NICs (one public, one private)

I've been following #1477 and #1705 and have run into more woes with my dual NIC setup.

When I upgraded to -> rancheros 1.0.0 on my dual NIC VM, I found that again I had only one interface working (the one with the gateway), I rolled back to 0.9.2, and found the same issue there, so I kept rolling back to 0.9.1

That fixed it for me and I now have two working NICs in separate networks again.

relevant network section

rancher:
  network:
    interfaces:
      eth0:
        address: 192.168.0.38/24
        mtu: 1500
        dhcp: false
      eth1:
        address: 10.1.111.93/24
        gateway: 10.1.111.65

[SvenDowideit]

I thought I wrote a test for this :/

[SvenDowideit]

I did - https://github.com/rancher/os/pull/1666/files#diff-2f43c1a40bf9c492da522998bd0fb833

but ... @joshuacox I don't suppose you have a spare box you can use to show me the output of ip a, and the dmesg when booted with rancher.debug=true in the kernel cmdline?

[joshuacox]

Ok, I have figured out why mine failed, having the gateway on the second address kills the first, here is an ip a of it in a broken state

for example this is a broken config:

  network:
    interfaces:
      eth0:
        address: 192.168.0.93/24
        mtu: 1500
        dhcp: false
      eth1:
        address: 10.3.33.247/24
        gateway: 10.3.33.1
        mtu: 1500
        dhcp: false

However, for this next one, I changed out which networks the virtual nics were attached (the opposite of the first), and changed the order here:

  network:
    interfaces:
      eth0:
        address: 10.3.33.247/24
        gateway: 10.3.33.1
        mtu: 1500
        dhcp: false
      eth1:
        address: 192.168.0.93/24
        mtu: 1500
        dhcp: false

that worksfine, @SvenDowideit feel free to close this as I can fix mine by ensuring the public interface (the one with the gateway on it), has to come first

[SvenDowideit]

ah, yes, and my test uses a gateway on both :( nope, not closing, I'll add another test and fix it too - hopefully that'll resolve the problem for more future configs.

[joshuacox]

I've got another server on 1.0.0 where the above is not necessarily true, possibly just rebooting a few times or some other timeout happens the issue resolves itself. still testing here

[SvenDowideit]

I might just make a 1.0.1-rc3 today (depending on how much the flu is affecting me) - at which point, i'd be very interested in seeing the rancher.debug=true dmesg output on both a working and failing setup (assuming its not just the ordering issue you noted above)

[joshuacox]

@SvenDowideit I'm not certain if you ever made an rc3, but I did upgrade to 1.0.1 and a few of my dual nic setups immediately died upon reboot (no addresses show up on the console login screen, and they are all unresponsive on any net interface). I needed to get them back up pretty quickly so I cut a few new VMs using the latest (1.0.1) iso, Install worked great, they even took the first reboot fine (where I add in the public interface). But then I logged in set a new hostname, ros console switch ubuntu, and reboot a second time and dead again, but not on machines that do not receive a second public interface and merely connect to the private service network. continuing to test here. I'll sud ros config set rancher.debug=true as it says here and dig deeper.

EDIT: adding the relevant network configs

rancher:
  network:
    interfaces:
      eth0:
        address: 8.8.8.93/24
        gateway: 8.8.8.65
        mtu: 1500
        dhcp: false
      eth1:
        address: 192.168.88.42/24
        mtu: 1500
        dhcp: false

further EDITs:
I hate this kind of "fix", but after recreating one of the public facing rancher hosts, and rebooting it after it sat with the dead interface a bit, it 'fixed' itself (no clue if something was going on in the background I was not able to log in at that point). I'm not certain if this would have fixed the original upgraded machines as they are all gone. But I can test some more.

[SvenDowideit]

argh. those are the worst, and probably explain why I'm having so much pain producing a similar result.

[stffabi]

We are seeing the same issue on our servers with multiple nics, running 1.0.1. I try to get a debug output...

[stffabi]

I suspect it has something to do with DHCP being activated per default on startup on all interfaces. I've seen that the interface which has no ip assigned is the one which has always been assigned an IP from the DHCP.

For example in the following case, eth2 had been assigned an IP from DHCP (10.141.50.81/16). In that case eth2 had no IP assigned after rancher has been started.

[ 21.154500] time="2017-05-23T07:14:24Z" level=info msg="Apply Network Config"
[ 21.155473] time="2017-05-23T07:14:24Z" level=info msg="Applying 127.0.0.1/8 to lo"
[ 21.155518] time="2017-05-23T07:14:24Z" level=info msg="Applying ::1/128 to lo"
[ 21.155671] time="2017-05-23T07:14:24Z" level=info msg="Applying 10.139.241.111/24 to eth0"
[ 21.155757] time="2017-05-23T07:14:24Z" level=info msg="Set 10.139.241.111/24 on eth0"
[ 21.155776] time="2017-05-23T07:14:24Z" level=info msg="removing fe80::20c:29ff:fea5:d29e/64 from eth0"
[ 21.156091] time="2017-05-23T07:14:24Z" level=info msg="Removed fe80::20c:29ff:fea5:d29e/64 from eth0"
[ 21.156364] time="2017-05-23T07:14:24Z" level=info msg="Applying 10.139.18.68/20 to eth1"
[ 21.156435] time="2017-05-23T07:14:24Z" level=info msg="Set 10.139.18.68/20 on eth1"
[ 21.156453] time="2017-05-23T07:14:24Z" level=info msg="removing fe80::20c:29ff:fea5:d2a8/64 from eth1"
[ 21.156673] time="2017-05-23T07:14:24Z" level=info msg="Removed fe80::20c:29ff:fea5:d2a8/64 from eth1"
[ 21.158250] time="2017-05-23T07:14:24Z" level=info msg="Added default gateway 10.139.16.1"
[ 21.158352] time="2017-05-23T07:14:24Z" level=info msg="Applying 10.141.0.33/16 to eth2"
[ 21.158415] time="2017-05-23T07:14:24Z" level=info msg="Set 10.141.0.33/16 on eth2"
[ 21.158433] time="2017-05-23T07:14:24Z" level=info msg="removing 10.141.50.81/16 eth2 from eth2"
[ 21.158501] time="2017-05-23T07:14:24Z" level=info msg="Removed 10.141.50.81/16 eth2 from eth2"
[ 21.158519] time="2017-05-23T07:14:24Z" level=info msg="removing fe80::20c:29ff:fea5:d2b2/64 from eth2"
[ 21.158727] time="2017-05-23T07:14:24Z" level=info msg="Removed fe80::20c:29ff:fea5:d2b2/64 from eth2"
[ 21.158914] time="2017-05-23T07:14:24Z" level=info msg="Apply Network Config RunDhcp"
[ 21.158927] time="2017-05-23T07:14:24Z" level=debug msg=RunDhcp
[ 21.188548] time="2017-05-23T07:14:24Z" level=error msg="exit status 1"
[ 21.188748] time="2017-05-23T07:14:24Z" level=debug msg="dhcpcd -u eth2: "
[ 21.191272] time="2017-05-23T07:14:24Z" level=error msg="exit status 1"
[ 21.191664] time="2017-05-23T07:14:24Z" level=error msg="exit status 1"
[ 21.192036] time="2017-05-23T07:14:24Z" level=debug msg="dhcpcd -u eth1: "
[ 21.192039] time="2017-05-23T07:14:24Z" level=debug msg="dhcpcd -u eth0: "
[ 21.192107] time="2017-05-23T07:14:24Z" level=info msg="Apply Network Config SyncHostname"

Here you see a dmesg output after a successfull boot, in that case no IP has been assigned from DCHP.

[ 24.038581] time="2017-05-23T08:53:51Z" level=info msg="Apply Network Config"
[ 24.039612] time="2017-05-23T08:53:51Z" level=info msg="Applying 127.0.0.1/8 to lo"
[ 24.039665] time="2017-05-23T08:53:51Z" level=info msg="Applying ::1/128 to lo"
[ 24.039853] time="2017-05-23T08:53:51Z" level=info msg="Applying 10.139.18.68/20 to eth0"
[ 24.039911] time="2017-05-23T08:53:51Z" level=info msg="removing fe80::20c:29ff:fea5:d29e/64 from eth0"
[ 24.040198] time="2017-05-23T08:53:51Z" level=info msg="Removed fe80::20c:29ff:fea5:d29e/64 from eth0"
[ 24.041831] time="2017-05-23T08:53:51Z" level=info msg="Added default gateway 10.139.16.1"
[ 24.041935] time="2017-05-23T08:53:51Z" level=info msg="Applying 10.139.241.111/24 to eth1"
[ 24.042030] time="2017-05-23T08:53:51Z" level=info msg="Set 10.139.241.111/24 on eth1"
[ 24.042050] time="2017-05-23T08:53:51Z" level=info msg="removing fe80::20c:29ff:fea5:d2a8/64 from eth1"
[ 24.042301] time="2017-05-23T08:53:51Z" level=info msg="Removed fe80::20c:29ff:fea5:d2a8/64 from eth1"
[ 24.042562] time="2017-05-23T08:53:51Z" level=info msg="Applying 10.141.0.33/16 to eth2"
[ 24.042639] time="2017-05-23T08:53:51Z" level=info msg="Set 10.141.0.33/16 on eth2"
[ 24.042658] time="2017-05-23T08:53:51Z" level=info msg="removing fe80::20c:29ff:fea5:d2b2/64 from eth2"
[ 24.042865] time="2017-05-23T08:53:51Z" level=info msg="Removed fe80::20c:29ff:fea5:d2b2/64 from eth2"
[ 24.043052] time="2017-05-23T08:53:51Z" level=info msg="Apply Network Config RunDhcp"
[ 24.043064] time="2017-05-23T08:53:51Z" level=debug msg=RunDhcp
[ 24.112092] time="2017-05-23T08:53:51Z" level=error msg="exit status 1"
[ 24.112444] time="2017-05-23T08:53:51Z" level=debug msg="dhcpcd -u eth0: "
[ 24.112621] time="2017-05-23T08:53:51Z" level=error msg="exit status 1"
[ 24.112805] time="2017-05-23T08:53:51Z" level=debug msg="dhcpcd -u eth2: "
[ 24.112837] time="2017-05-23T08:53:51Z" level=error msg="exit status 1"
[ 24.112865] time="2017-05-23T08:53:51Z" level=debug msg="dhcpcd -u eth1: "
[ 24.112932] time="2017-05-23T08:53:51Z" level=info msg="Apply Network Config SyncHostname"

This would also explain why sometimes the ip addresses get correctly assigned and with another reboot it fails. This depends on the timing the DHCP server is able to lease an IP out or not, before the config has been changed to the static ip.

@joshuacox do you have DHCP servers running on your networks whose interfaces have this problem?