Drop dynamodb requirements for S3 locks

[dimoschi]

Terraform Version

Terraform v1.7.5
on darwin_arm64

Use Cases

• S3 for state management is one of the strongest alternatives to remote, however a dynamodb is required for state locking, which increases complexity, although not by much.

Attempted Solutions

Based on the current implementation of S3 there isn't an alternate solution.

Proposal

AWS S3 nowadays comes with an object lock mechanism, which could be utilized instead of relying to DynamoDB to lock the state. Per AWS docs (https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html#object-lock-legal-holds) the object lock uses WORM, that can be applied on demand (legal hold), which would prevent multiple runs from altering the state.

References

• Deprecate s3 remote backend lock table with new strong consistency #27070

[apparentlymart]

Thanks for sharing this, @dimoschi.

These new S3 features seem more motivated by legal/compliance than by being a synchronization mechanism, so I think we'll need to study more closely to find out what consistency guarantees it provides, but I agree that this does seem like an interesting alternative to using DynamoDB.

The S3 backend is maintained by the AWS provider team rather than by the Terraform Core team, so I'm going to label this to make it visible to them for consideration.

In the meantime, if you or anyone else has references to specific information about what consistency guarantees S3 offers for object lock -- for example, is it safe to assume that the lock is in effect for all clients as soon as the API server responds? -- that could help with determining whether this feature is sufficient to meet Terraform's needs for workspace state locking.

Thanks again!

[bschaatsbergen]

Taking a quick look at the S3 Object locking documentation, it appears that it's only accessible on buckets with versioning enabled.

Object Lock works only in buckets that have S3 Versioning enabled.

Source: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html

And unfortunately we are not able to rely on the read-after-write consistency for locking too.

Amazon S3 does not support object locking for concurrent writers. If two PUT requests are simultaneously made to the same key, the request with the latest timestamp wins. If this is an issue, you must build an object-locking mechanism into your application.

Source: https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html#ConsistencyModel

Also, I'm not sure if using object locking is a neat solution for this - it would have to be used in 'governance' mode, and you're required to specify a lock duration... which you then have to remove afterward.

[bschaatsbergen]

Hi @dimoschi,

For more context, please refer to #35625 and #35661, where we are actively tracking and prototyping an S3-native backend implementation. This effort instead builds on the recently released conditional writes feature. Thank you for your patience!

[bschaatsbergen]

Closing this in favor to focus on tracking the above referenced issue and PR.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.