Investigate and fix 500s when writing to Dynamo

[jml]

We are getting a fairly constant stream of 500s from Dynamo when we attempt to write to it (~3-4qps on dev).

We should understand what's causing this & fix it. https://github.com/weaveworks/monitoring/issues/12 will help.

[jml]

Might be related to #61, but too early to say.

[tomwilkie]

According to loggly, 90% of the log lines (and therefore errors) are ProvisionedThroughputExceededException.

[tomwilkie]

Write capacity was set to 400/s, which we're almost always trying to do more. Tuning chunk size (#11) should help, but in the mean time I'll make it 1k/s.

[tomwilkie]

That seems to have had a small temporary reduction, but then back to usual - we're still being throttled. I suspect we're just trying to write to much, and the change in provisioned capacity just reset some throttling counter.

[tomwilkie]

Now the ongoing deployment is done, the number of 500s has dropped dramatically (and the dynamodb latency is much improved). So this did have an effect:

[tomwilkie]

Fixed by #85