Receiver crashes if DNs file is being written to and read from at the same time.

[gregcorbett]

As per this ELOG, bin/receiver.py seems to crashes if get_dns is called while the DNs file is being written to by the auth component of APEL.

It would be good to be able to reproduce this bug to determine the exact cause.

[tofu-rocketry]

It happened again.

[gregcorbett]

I did the tiniest amount of investigation to try and come up with a repeatable test for this and it seems (at least under Python 3) having the dns file open for writing while trying to read from it isn't sufficient to cause this. Maybe the cause is trying to read the exact moment the write is actually flushed to disk...

[tofu-rocketry]

Sounds like a simple retry would be sufficient after ~30 seconds, with a limit of, say, 3 attempts before barfing.

[gregcorbett]

It happened again

[tofu-rocketry]

So it's ending up here:

ssm/ssm/agents.py

Line 377 in 83e33c7

raise Ssm2Exception('No valid DNs found in %s. SSM will not start' % dn_file)

So it's getting a len of 0. So we could just add a couple of retries within that function to see if it can get len>0