Retry on errors from Dynamo

[jml]

When we attempt to write to dynamo, we will sometimes get errors. Currently, we abort on error.

This is almost never the right behaviour, because failing to write to dynamo (esp. during shut down) means loss of user data. Instead, we should retry.

Note that one major source of errors is being throttled by Dynamo. In this case, we should also retry, but we should be especially careful that we back off, lest we cause a cascading failure.

[jml]

@juliusv in #84 points out that we do retry in normal operation but not during shutdown.

[jml]

Fixed by #119

[tomwilkie]

Still not doing retries on shutdown

[jml]

Ah right. Kind of curious now as to what @juliusv meant when he said we were backing off already.

[tomwilkie]

retry, not backoff.

its because we don't drop the chunks if we fail to persist them, so they'll get retried on a later pass.