Retry on 412 status codes

[tomhjp]

Fixes #332 - see issue for details.

412 means the token used has not yet been replicated to the performance replica queried, so we can append it to the list of retryable HTTP codes.

[swenson]

LGTM

[calvn]

Should we retry on 500's (except 501) too? This is what the Vault Go API client does via go-retryablehttp.

[tomhjp]

I believe 500 and a few other 5XX are already included in the defaults for the got dependency: https://github.com/sindresorhus/got/blob/HEAD/documentation/7-retry.md#retry

I'd like to give that a test to be sure we're preserving that default properly though.

[swenson]

That's interesting. I wouldn't recommend retrying on 500 on POST/PATCH messages, because they aren't guaranteed to be idempotent and it is possible that the server is in a weird state after the 500. GET, PUT, and DELETE should be safe to retry on 500 though.

But I would recommend retrying 400 and 404 for all verbs.

[tvoran]

How many times will this retry? Should we set a default limit? https://github.com/sindresorhus/got/blob/HEAD/documentation/7-retry.md#limit

We might want a configurable retry number as in this PR as well: #337

[tomhjp]

I wouldn't recommend retrying on 500 on POST/PATCH messages

By default it never retries on POST or PATCH, although vault-action doesn't support any methods other than GET anyway.

But I would recommend retrying 400 and 404 for all verbs.

Is this so we get reads after writes for pre-1.10? If we want to go down that route I think we should take advantage of the replication state headers from Vault instead, i.e. do similar to ReadYourWrites from the api package.

How many times will this retry? Should we set a default limit?

The default is up to 2 retries, for a total of 3 attempts. Yeah it would be nice to add a configurable for that, although I'll leave that feature for a separate PR.

[tomhjp]

I'm going to merge for now, but happy to address any more follow ups in another PR.