Docker should be only started after taupage-init is done

[aermakov-zalando]

If the instance is rebooted because of AWS instance autorecovery, the application will start before the disks are mounted (fsck takes time, and the image is already cached), which will obviously fail. This should be fixed by making the application only start after all (or at least important for the application) init scripts are done. See also #441.

[a1exsh]

Not sure why you see relation with #441?

[aermakov-zalando]

Because we should also not start the application if the init tasks fail.

[a1exsh]

Yes, but in this case init didn't fail, it was just done in the incorrect order. And I think if taupage-init fails a step, it doesn't go to the next one?

[aermakov-zalando]

The bug report has led me to believe that it'll just proceed to the next script. However you're right, it'll just abort and not do anything, and the issue with container starting isn't related to taupage-init.

[a1exsh]

The root cause here is that there 2 sets of init scripts in taupage: the ones shipped in the system packages that live in /etc/init.d, with all the dependencies to define proper startup order in place, and another set of custom script in /opt/taupage/init.d.

This issue is exactly the case when normal operation of a system init script (for docker) depends on running a custom init script from taupage (10-prepare-disks.py). But the order is the opposite of what it should be: docker server starts before disks are mounted. In normal startup scenarios it's not a problem, because docker starts w/o any pre-existing containers, and the container is created later, after disks are mounted. In case of autorecovery (or reboot of instance) you can have this race condition: if docker starts the pre-existing container before the disks are mounted, the application can fail.