Use a Retryer instead of retry loops for DynamoDB

[bboreham]

The DynamoDB client disables retries and then wraps everything in a loop to do retries.
Disabling retries came in #153; the backoff loop has gone through many iterations since #119.

The stated reason is that we want to monitor retries.

From my experiences using the DynamoDB API elsewhere, I think it would be far cleaner to implement a Retryer which adds the required monitoring, let the AWS SDK do retries, and take out the explicit loops.

Noted at #792 (comment).

[rlisagor]

There's a chance I can take this on, depending on our priorities. It seems like the logic in the DefaultRetryer from the AWS SDK should work, if wrapped with monitoring code. Would that be sufficient? Or do you envision some custom retry logic?

[csmarchbanks]

I believe the intention is to just wrap with monitoring code and stop trying to do any custom retry logic.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[pracucci]

@bboreham In the PR #2452 you mentioned it partially fixes this issue. What's left out?

[bboreham]

I believe all of pkg/chunk/aws/dynamodb_table_client.go could be changed from the loop in backoffAndRetry() to the retryer added in #2452.

BatchWrite() and getDynamoDBChunks() also implement their own retry loops, but these have complicated interactions with throttling so I suspect will have to stay.