Add support for SSE-KMS and bucket owner verification #18312

jed326 · 2025-05-15T22:13:24Z

Description

This PR makes 3 main changes

1. Remove the repository-s3 setting `server_side_encryption`

As of January 5, 2023, S3 now applies SSE-S3 as the base level of encryption (the AES256 encryption type), and there is no way to disable this. This means regardless of whether or not this setting is set or whatever value the setting is set to, it makes no difference in how objects are uploaded to S3.

For more details, see: https://docs.aws.amazon.com/AmazonS3/latest/userguide/serv-side-encryption.html

Fortunately, repository settings are fully backwards compatible ever after being removed, which means that even after removing the setting, I can still register a repository like so:

curl -XPUT "http://localhost:9200/_snapshot/vector-repo" -H 'Content-Type: application/json' -d'                                            
{
  "type": "s3",
  "settings": {
    "bucket": "sse-tester",
    "base_path": "test_sse_with_enc_ctx",
    "region": "us-east-1",
    "expected_bucket_owner": "759512025873",
    "server_side_encryption": true
  }
}
'

2. Add settings to support SSE-KMS

There are 4 new settings introduced:

server_side_encryption_type
server_side_encryption_kms_key_id
server_side_encryption_bucket_key_enabled
server_side_encryption_encryption_context

The server_side_encryption_type setting supports 3 values:

AES256 -- SSE-S3)
aws:kms -- SSE-KMS)
bucket_default -- this will make the request use the default encryption configuration on the S3 bucket

Example repository:

curl -XPUT "http://localhost:9200/_snapshot/vector-repo" -H 'Content-Type: application/json' -d'                                            
{
  "type": "s3",
  "settings": {
    "bucket": "sse-tester",
    "base_path": "test_sse_with_enc_ctx",
    "region": "us-east-1",
    "expected_bucket_owner": "759512025873",
    "server_side_encryption_type": "aws:kms",
    "server_side_encryption_kms_key_id": "arn:aws:kms:us-east-1:XXXXXX:key/XXXXXX",
    "server_side_encryption_encryption_context": "{\"domainARN\": \"my-test-domain\"}"
  }
}
'

3. Add bucket owner verification

The new setting expected_bucket_owner, when set, will be passed in all S3 bucket operation requests to verify that the bucket owner is the expected account.

See: https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucket-owner-condition.html#bucket-owner-condition-when-to-use

Testing

Added the new setting/fields to the existing SSE unit tests, but we are pretty limited without being able to actually call S3.

Note: Pasted actual KMS key in the test results below, but the key used for testing has been deleted for security purposes

Manual testing:

Register repo with valid kms values, take snapshot, check S3 getObject request for encryption values

curl -XPUT "http://localhost:9200/_snapshot/vector-repo" -H 'Content-Type: application/json' -d'                                            
{
  "type": "s3",
  "settings": {
    "bucket": "sse-tester",
    "base_path": "test_sse_with_enc_ctx",
    "region": "us-east-1",
    "expected_bucket_owner": "759512025873",
    "server_side_encryption": true,
    "server_side_encryption_type": "aws:kms",
    "server_side_encryption_kms_key_id": "arn:aws:kms:us-east-1:209972747865:key/b74e2711-1b42-4f5f-9831-aa8519f58dc8",
    "server_side_encryption_encryption_context": "{\"domainARN\": \"my-test-domain\"}",
    "server_side_encryption_bucket_key_enabled": true
  }
}
'

aws s3api get-object --bucket sse-tester --key test_sse_with_enc_ctx/snap-IzNPHJG8RUWOZWPtUWINpQ.dat ./tmp/downloadedFile.file
{
    "AcceptRanges": "bytes",
    "LastModified": "2025-05-20T01:19:04+00:00",
    "ContentLength": 306,
    "ETag": "\"04c6d13cd4c99c504db07b52d5865c95\"",
    "ChecksumCRC32": "OveSuA==",
    "ChecksumType": "FULL_OBJECT",
    "ContentType": "application/octet-stream",
    "ServerSideEncryption": "aws:kms",
    "Metadata": {},
    "SSEKMSKeyId": "arn:aws:kms:us-east-1:209972747865:key/b74e2711-1b42-4f5f-9831-aa8519f58dc8",
    "BucketKeyEnabled": true
}

Register repo with invalid KMS key

curl -XPUT "http://localhost:9200/_snapshot/vector-repo" -H 'Content-Type: application/json' -d'                                            
{
  "type": "s3",
  "settings": {
    "bucket": "sse-tester",
    "base_path": "test_sse_with_enc_ctx",
    "region": "us-east-1",
    "expected_bucket_owner": "759512025873",
    "server_side_encryption": true,
    "server_side_encryption_type": "aws:kms",
    "server_side_encryption_kms_key_id": "arn:aws:kms:us-east-1:209972747865:key/b74e2711-1b42-4f5f-9831-aa8519f58dc0",
    "server_side_encryption_encryption_context": "{\"domainARN\": \"my-test-domain\"}",
    "server_side_encryption_bucket_key_enabled": true
  }
}
'
{
    "error": {
        "root_cause": [{
            "type": "repository_verification_exception",
            "reason": "[vector-repo] path [test_sse_with_enc_ctx] is not accessible on cluster-manager node"
        }],
        "type": "repository_verification_exception",
        "reason": "[vector-repo] path [test_sse_with_enc_ctx] is not accessible on cluster-manager node",
        "caused_by": {
            "type": "i_o_exception",
            "reason": "Unable to upload object [test_sse_with_enc_ctx/tests-9lw1TTwWQjasbYQfuqhx3w/master.dat] using a single upload",
            "caused_by": {
                "type": "s3_exception",
                "reason": "User: ----- is not authorized to perform: kms:GenerateDataKey on this resource because the resource does not exist in this Region, no resource-based policies allow access, or a resource-based policy explicitly denies access (Service: S3, Status Code: 403, Request ID: H7MDGSGQG5JXZ1RN, Extended Request ID: TZzLwOcEl5ZvgF0PlKA9sWaRb6Ejkopr5CA9UeWhp9188GxSo3afo+CpGHW7pshersXYAdFMaxY=) (SDK Attempt Count: 1)"
            }
        }
    },
    "status": 500
}

Register repo with invalid encryption context

curl -XPUT "http://localhost:9200/_snapshot/vector-repo" -H 'Content-Type: application/json' -d'                                            
{
  "type": "s3",
  "settings": {
    "bucket": "sse-tester",
    "base_path": "test_sse_with_enc_ctx",
    "region": "us-east-1",
    "expected_bucket_owner": "759512025873",
    "server_side_encryption": true,
    "server_side_encryption_type": "aws:kms",
    "server_side_encryption_kms_key_id": "arn:aws:kms:us-east-1:209972747865:key/b74e2711-1b42-4f5f-9831-aa8519f58dc8",
    "server_side_encryption_encryption_context": "{\"domainARN\": \"my-test-domain\"}INVALID",
    "server_side_encryption_bucket_key_enabled": true
  }
}
'
{
    "error": {
        "root_cause": [{
            "type": "repository_verification_exception",
            "reason": "[vector-repo] path [test_sse_with_enc_ctx] is not accessible on cluster-manager node"
        }],
        "type": "repository_verification_exception",
        "reason": "[vector-repo] path [test_sse_with_enc_ctx] is not accessible on cluster-manager node",
        "caused_by": {
            "type": "i_o_exception",
            "reason": "Unable to upload object [test_sse_with_enc_ctx/tests-bv7KdLsNT-isoYU7xnYEBg/master.dat] using a single upload",
            "caused_by": {
                "type": "s3_exception",
                "reason": "The header 'x-amz-server-side-encryption-context' shall be Base64-encoded UTF-8 string holding JSON which represents a string-string map (Service: S3, Status Code: 400, Request ID: ECVM877F8ABXS1RX, Extended Request ID: 00vaUW6siUaCXv5y0YToPXZUyLc3sb9/lrIpE9FTKwzN0wrvnGoUTNZwUzGGAbutO9yLkzLu3v0=) (SDK Attempt Count: 1)"
            }
        }
    },
    "status": 500
}

Register repo with invalid bucket owner

curl -XPUT "http://localhost:9200/_snapshot/vector-repo" -H 'Content-Type: application/json' -d'                                            
{
  "type": "s3",
  "settings": {
    "bucket": "sse-tester",
    "base_path": "test_sse_with_enc_ctx",
    "region": "us-east-1",
    "expected_bucket_owner": "209972747865",
    "server_side_encryption": true,
    "server_side_encryption_type": "aws:kms",
    "server_side_encryption_kms_key_id": "arn:aws:kms:us-east-1:209972747865:key/b74e2711-1b42-4f5f-9831-aa8519f58dc8",
    "server_side_encryption_encryption_context": "{\"domainARN\": \"my-test-domain\"}",
    "server_side_encryption_bucket_key_enabled": true
  }
}
'
{
    "error": {
        "root_cause": [{
            "type": "repository_verification_exception",
            "reason": "[vector-repo] path [test_sse_with_enc_ctx] is not accessible on cluster-manager node"
        }],
        "type": "repository_verification_exception",
        "reason": "[vector-repo] path [test_sse_with_enc_ctx] is not accessible on cluster-manager node",
        "caused_by": {
            "type": "i_o_exception",
            "reason": "Unable to upload object [test_sse_with_enc_ctx/tests-wpGzDNqqSD-5z_t3qNoJrA/master.dat] using a single upload",
            "caused_by": {
                "type": "s3_exception",
                "reason": "Access Denied (Service: S3, Status Code: 403, Request ID: 59TPC5JJDGHR3J89, Extended Request ID: /yqpUj0dG4UhSsAmYnPEejJmWOpHdcy2hjv/hIYrRZjG0sSlPhG20fM9zUCHMnW7SDzdwBTEUJE=) (SDK Attempt Count: 1)"
            }
        }
    },
    "status": 500
}

Register repo with invalid encryption type

curl -XPUT "http://localhost:9200/_snapshot/vector-repo" -H 'Content-Type: application/json' -d'                                            
{
  "type": "s3",
  "settings": {
    "bucket": "sse-tester",
    "base_path": "test_sse_with_enc_ctx",
    "region": "us-east-1",
    "expected_bucket_owner": "759512025873",
    "server_side_encryption": true,
    "server_side_encryption_type": "invalid",
    "server_side_encryption_kms_key_id": "arn:aws:kms:us-east-1:209972747865:key/b74e2711-1b42-4f5f-9831-aa8519f58dc8",
    "server_side_encryption_encryption_context": "{\"domainARN\": \"my-test-domain\"}",
    "server_side_encryption_bucket_key_enabled": true
  }
}
'
{
    "error": {
        "root_cause": [{
            "type": "repository_exception",
            "reason": "[vector-repo] failed to create repository"
        }],
        "type": "repository_exception",
        "reason": "[vector-repo] failed to create repository",
        "caused_by": {
            "type": "illegal_argument_exception",
            "reason": "server_side_encryption_type must be one of [AES256, aws:kms, bucket_default]"
        }
    },
    "status": 500
}

Check that default encryption settings are used when default setting is set

aws s3api get-bucket-encryption --bucket sse-tester                                                                           
{
    "ServerSideEncryptionConfiguration": {
        "Rules": [
            {
                "ApplyServerSideEncryptionByDefault": {
                    "SSEAlgorithm": "aws:kms",
                    "KMSMasterKeyID": "arn:aws:kms:us-east-1:759512025873:alias/aws/s3"
                },
                "BucketKeyEnabled": true
            }
        ]
    }
}

curl -XPUT "http://localhost:9200/_snapshot/vector-repo" -H 'Content-Type: application/json' -d'                                            
{
  "type": "s3",
  "settings": {
    "bucket": "sse-tester",
    "base_path": "test_sse_with_enc_ctx",
    "region": "us-east-1",
    "expected_bucket_owner": "759512025873",
    "server_side_encryption": true,
    "server_side_encryption_type": "bucket_default" 
  }
}
'

aws s3api get-object --bucket sse-tester --key test_sse_with_enc_ctx/snap-GQmFCVglSPWzgntQWC08sA.dat ./tmp/downloadedFile.file
{
    "AcceptRanges": "bytes",
    "LastModified": "2025-05-20T01:27:57+00:00",
    "ContentLength": 306,
    "ETag": "\"7dd0810991c09eb7a2f6eb5e4e379913\"",
    "ChecksumCRC32": "kB1bXQ==",
    "ChecksumType": "FULL_OBJECT",
    "ContentType": "application/octet-stream",
    "ServerSideEncryption": "aws:kms",
    "Metadata": {},
    "SSEKMSKeyId": "arn:aws:kms:us-east-1:759512025873:key/2826bcd9-2686-4d65-8cf6-aa5846c0c36f",
    "BucketKeyEnabled": true
}

Check that setting bucket key enabled to false disables bucket key

curl -XPUT "http://localhost:9200/_snapshot/vector-repo" -H 'Content-Type: application/json' -d'                                            
{
  "type": "s3",
  "settings": {
    "bucket": "sse-tester",
    "base_path": "test_sse_with_enc_ctx",
    "region": "us-east-1",
    "expected_bucket_owner": "759512025873",
    "server_side_encryption": true,
    "server_side_encryption_type": "aws:kms",       
    "server_side_encryption_kms_key_id": "arn:aws:kms:us-east-1:209972747865:key/b74e2711-1b42-4f5f-9831-aa8519f58dc8",
    "server_side_encryption_encryption_context": "{\"domainARN\": \"my-test-domain\"}",
    "server_side_encryption_bucket_key_enabled": false
  }
}
'

aws s3api get-object --bucket sse-tester --key test_sse_with_enc_ctx/snap-8di6_WPiQeqOjh0IgnG5Fg.dat ./tmp/downloadedFile.file
{
    "AcceptRanges": "bytes",
    "LastModified": "2025-05-20T01:31:41+00:00",
    "ContentLength": 306,
    "ETag": "\"c9f4cdd1460751d676ace612c95a7ffe\"",
    "ChecksumCRC32": "LSLcpg==",
    "ChecksumType": "FULL_OBJECT",
    "ContentType": "application/octet-stream",
    "ServerSideEncryption": "aws:kms",
    "Metadata": {},
    "SSEKMSKeyId": "arn:aws:kms:us-east-1:209972747865:key/b74e2711-1b42-4f5f-9831-aa8519f58dc8"
}

BucketKeyEnabled is not present in the response above.

Check that AES256 encryption type still uses to SSE-S3

curl -XPUT "http://localhost:9200/_snapshot/vector-repo" -H 'Content-Type: application/json' -d'                                            
{
  "type": "s3",
  "settings": {
    "bucket": "sse-tester",
    "base_path": "test_sse_with_enc_ctx",
    "region": "us-east-1",
    "expected_bucket_owner": "759512025873",
    "server_side_encryption": true,
    "server_side_encryption_type": "AES256"        
  }                                                                                                                    
}                                                                                      
' 

aws s3api get-object --bucket sse-tester --key test_sse_with_enc_ctx/snap-D2HjatDxRhqWA6rJWFI0Og.dat ./tmp/downloadedFile.file
{
    "AcceptRanges": "bytes",
    "LastModified": "2025-05-20T01:34:56+00:00",
    "ContentLength": 306,
    "ETag": "\"e7545a369280ab6c6277f2a208629fdb\"",
    "ChecksumCRC32": "HusXpQ==",
    "ChecksumType": "FULL_OBJECT",
    "ContentType": "application/octet-stream",
    "ServerSideEncryption": "AES256",
    "Metadata": {}
}

Valid encryption settings on async multi-part upload
I used the kNN plugin's implementation for this, as it was a little easier to set up than remote store (ref: https://github.com/opensearch-project/k-NN/blob/b7fc5dd98072ea157a9b301e7f93d79e9700984d/src/main/java/org/opensearch/knn/index/codec/nativeindex/remote/DefaultVectorRepositoryAccessor.java#L81)

curl -XPUT "http://localhost:9200/_snapshot/vector-repo" -H 'Content-Type: application/json' -d'                                            
{
  "type": "s3",
  "settings": {
    "bucket": "sse-tester",
    "base_path": "test_sse_with_enc_ctx",
    "region": "us-east-1",
    "expected_bucket_owner": "759512025873",
    "server_side_encryption": true,
    "server_side_encryption_type": "aws:kms",
    "server_side_encryption_kms_key_id": "arn:aws:kms:us-east-1:209972747865:key/b74e2711-1b42-4f5f-9831-aa8519f58dc8",
    "server_side_encryption_encryption_context": "{\"domainARN\": \"my-test-domain\"}",
    "server_side_encryption_bucket_key_enabled": true
  }
}


aws s3api get-object --bucket sse-tester --key test_sse_vector_repo/l4bvV1PlQXiKOUE4EdefDQ_vectors/NI3v9JYBAeI-CJ5ZSygX_target_field__53.knnvec downloadedFile.file 
{
    "AcceptRanges": "bytes",
    "LastModified": "2025-05-21T22:20:46+00:00",
    "ContentLength": 307200000,
    "ETag": "\"05dff410cf57ca43906a34914cc62d04-19\"",
    "ChecksumCRC64NVME": "s3ZEszslDh8=",
    "ChecksumType": "FULL_OBJECT",
    "ContentType": "binary/octet-stream",
    "ServerSideEncryption": "aws:kms",
    "Metadata": {},
    "SSEKMSKeyId": "arn:aws:kms:us-east-1:209972747865:key/b74e2711-1b42-4f5f-9831-aa8519f58dc8",
    "BucketKeyEnabled": true
}

aws s3api get-object-attributes --bucket sse-tester --key test_sse_vector_repo/l4bvV1PlQXiKOUE4EdefDQ_vectors/NI3v9JYBAeI-CJ5ZSygX_target_field__53.knnvec --object-attributes ObjectParts
{
    "LastModified": "2025-05-21T22:20:46+00:00",
    "ObjectParts": {
        "TotalPartsCount": 19
    }
}

Related Issues

Resolves #14606

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.
- [DOC] SSE-KMS support for repository-s3 documentation-website#9929

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2025-05-15T22:20:58Z

❌ Gradle check result for d04e3d3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-05-15T23:01:41Z

❌ Gradle check result for e2a5d28: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-05-16T05:00:35Z

❕ Gradle check result for a3b50c3: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2025-05-16T05:01:26Z

Codecov Report

Attention: Patch coverage is 80.00000% with 24 lines in your changes missing coverage. Please review.

Project coverage is 72.48%. Comparing base (b53de2b) to head (64ccbce).
Report is 10 commits behind head on main.

Files with missing lines	Patch %	Lines
...g/opensearch/repositories/s3/utils/SseKmsUtil.java	69.69%	7 Missing and 3 partials ⚠️
...va/org/opensearch/repositories/s3/S3BlobStore.java	52.94%	6 Missing and 2 partials ⚠️
...a/org/opensearch/repositories/s3/S3Repository.java	68.42%	2 Missing and 4 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #18312      +/-   ##
============================================
- Coverage     72.61%   72.48%   -0.13%     
+ Complexity    67438    67374      -64     
============================================
  Files          5488     5489       +1     
  Lines        311067   311168     +101     
  Branches      45218    45222       +4     
============================================
- Hits         225872   225552     -320     
- Misses        66848    67265     +417     
- Partials      18347    18351       +4

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

github-actions · 2025-05-16T22:51:28Z

❌ Gradle check result for 5196b5d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-05-16T23:38:57Z

❌ Gradle check result for 457a26c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-05-19T18:27:00Z

❌ Gradle check result for 28ea988: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-05-19T19:26:19Z

❌ Gradle check result for ca7fc4d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-05-19T20:43:19Z

❌ Gradle check result for 7eacbe1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-05-19T21:30:59Z

❌ Gradle check result for adacc1a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-05-19T22:44:07Z

❌ Gradle check result for adacc1a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-05-19T23:56:37Z

❕ Gradle check result for adacc1a: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

jed326 · 2025-05-20T01:41:21Z

@Bukhtawar @gbbafna please take a look when you get a chance, thanks!

plugins/repository-s3/src/main/java/org/opensearch/repositories/s3/S3BlobContainer.java

github-actions · 2025-05-20T22:48:15Z

❌ Gradle check result for 65fd070: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

…dd support for bucket owner verification Signed-off-by: Jay Deng <[email protected]>

jed326 · 2025-05-21T00:15:45Z

Thanks for the first pass @gbbafna! I need to run some more manual tests on the multi-part upload scenarios as well. For the previous remote store PRs, did we have any ways to run tests directly against S3? I don't think it's possible for us to add functional coverage on any of the new settings without calling S3 so wanted to see if there was more I could do beyond manual tests.

github-actions · 2025-05-21T01:11:02Z

✅ Gradle check result for 64ccbce: SUCCESS

gbbafna · 2025-05-21T03:01:54Z

Thanks for the first pass @gbbafna! I need to run some more manual tests on the multi-part upload scenarios as well. For the previous remote store PRs, did we have any ways to run tests directly against S3? I don't think it's possible for us to add functional coverage on any of the new settings without calling S3 so wanted to see if there was more I could do beyond manual tests.

We created docker cluster locally which uses S3 . Yes, it is not possible to test this without actually calling S3 .

jed326 · 2025-05-21T04:56:57Z

We created docker cluster locally which uses S3 . Yes, it is not possible to test this without actually calling S3 .

Thanks @gbbafna. In that case I will work on adding some more manual tests that cover the async use cases and update the PR description when done

jed326 · 2025-05-21T23:58:53Z

@gbbafna I've added manual testing on the async cases to the PR overview. It's not really possible to do unhappy path testing on these either, since registering a repo does a test upload. I also manually tested the async delete path, but wasn't sure how to display the results for that. Please let me know if you think there is additional test coverage I can do, or if you think this PR is good to merge. Thanks!

gbbafna · 2025-05-22T11:06:10Z

@gbbafna I've added manual testing on the async cases to the PR overview. It's not really possible to do unhappy path testing on these either, since registering a repo does a test upload. I also manually tested the async delete path, but wasn't sure how to display the results for that. Please let me know if you think there is additional test coverage I can do, or if you think this PR is good to merge. Thanks!

LGTM . Thanks @jed326 for the changes.

Bukhtawar · 2025-05-25T05:55:44Z

plugins/repository-s3/src/main/java/org/opensearch/repositories/s3/S3BlobStore.java

+        this.serverSideEncryptionType = SERVER_SIDE_ENCRYPTION_TYPE_SETTING.get(repositoryMetadata.settings());
+        this.serverSideEncryptionKmsKey = SERVER_SIDE_ENCRYPTION_KMS_KEY_SETTING.get(repositoryMetadata.settings());
+        this.serverSideEncryptionBucketKey = SERVER_SIDE_ENCRYPTION_BUCKET_KEY_SETTING.get(repositoryMetadata.settings());
+        this.serverSideEncryptionEncryptionContext = SERVER_SIDE_ENCRYPTION_ENCRYPTION_CONTEXT_SETTING.get(repositoryMetadata.settings());
+        this.expectedBucketOwner = EXPECTED_BUCKET_OWNER_SETTING.get(repositoryMetadata.settings());


@jed326 : For my understanding would these settings become update-able?

@Bukhtawar yep, a user can change the encryption type / KMS key / encryption context at any time. On the download side these values are read from object metadata rather than user provided, so as long as the calling identity has proper permissions updating these settings won't break anything.

…dd support for bucket owner verification (opensearch-project#18312) Signed-off-by: Jay Deng <[email protected]>

…dd support for bucket owner verification (opensearch-project#18312) Signed-off-by: Jay Deng <[email protected]>Signed-off-by: TJ Neuenfeldt <[email protected]>

…dd support for bucket owner verification (opensearch-project#18312) Signed-off-by: Jay Deng <[email protected]>

jed326 force-pushed the sse-kms branch from d04e3d3 to e2a5d28 Compare May 15, 2025 22:37

jed326 force-pushed the sse-kms branch from e2a5d28 to a3b50c3 Compare May 16, 2025 03:53

github-actions bot added enhancement Enhancement or improvement to existing feature or request Plugins security Anything security related labels May 16, 2025

opensearch-ci-bot mentioned this pull request May 16, 2025

[AUTOCUT] Gradle Check Flaky Test Report for AzureBlobStoreRepositoryTests #14291

Open

jed326 force-pushed the sse-kms branch from a3b50c3 to 5196b5d Compare May 16, 2025 22:38

jed326 force-pushed the sse-kms branch 2 times, most recently from 7ccc13e to 457a26c Compare May 16, 2025 23:11

opensearch-ci-bot mentioned this pull request May 17, 2025

[AUTOCUT] Gradle Check Flaky Test Report for PemTrustConfigTests #17745

Open

jed326 force-pushed the sse-kms branch from 457a26c to 28ea988 Compare May 19, 2025 17:56

jed326 force-pushed the sse-kms branch from 28ea988 to ca7fc4d Compare May 19, 2025 19:01

jed326 force-pushed the sse-kms branch from ca7fc4d to 7eacbe1 Compare May 19, 2025 20:09

jed326 force-pushed the sse-kms branch from 7eacbe1 to adacc1a Compare May 19, 2025 21:01

opensearch-ci-bot mentioned this pull request May 20, 2025

[AUTOCUT] Gradle Check Flaky Test Report for RemoteStoreStatsIT #14310

Open

jed326 marked this pull request as ready for review May 20, 2025 01:40

jed326 requested a review from a team as a code owner May 20, 2025 01:40

jed326 added v3.1.0 Storage:Snapshots Storage:Remote labels May 20, 2025

github-project-automation bot added this to Storage Project Board May 20, 2025

jed326 mentioned this pull request May 20, 2025

[DOC] SSE-KMS support for repository-s3 opensearch-project/documentation-website#9929

Closed

4 tasks

gbbafna reviewed May 20, 2025

View reviewed changes

plugins/repository-s3/src/main/java/org/opensearch/repositories/s3/S3BlobContainer.java Outdated Show resolved Hide resolved

plugins/repository-s3/src/main/java/org/opensearch/repositories/s3/S3BlobContainer.java Outdated Show resolved Hide resolved

jed326 force-pushed the sse-kms branch from adacc1a to 65fd070 Compare May 20, 2025 22:22

jed326 force-pushed the sse-kms branch from 65fd070 to c9b0a1c Compare May 20, 2025 23:02

Add support for SSE-KMS, remove old server_side_encryption setting, a…

64ccbce

…dd support for bucket owner verification Signed-off-by: Jay Deng <[email protected]>

jed326 force-pushed the sse-kms branch from c9b0a1c to 64ccbce Compare May 21, 2025 00:01

gbbafna approved these changes May 22, 2025

View reviewed changes

github-project-automation bot moved this to 👀 In review in Storage Project Board May 22, 2025

jed326 merged commit ab0827a into opensearch-project:main May 22, 2025
32 checks passed

github-project-automation bot moved this from 👀 In review to ✅ Done in Storage Project Board May 22, 2025

Bukhtawar reviewed May 25, 2025

View reviewed changes

tandonks pushed a commit to tandonks/OpenSearch that referenced this pull request Jun 1, 2025

Add support for SSE-KMS, remove old server_side_encryption setting, a…

9b836db

…dd support for bucket owner verification (opensearch-project#18312) Signed-off-by: Jay Deng <[email protected]>

Gagan6164 pushed a commit to Gagan6164/OpenSearch that referenced this pull request Jun 8, 2025

Add support for SSE-KMS, remove old server_side_encryption setting, a…

26de985

…dd support for bucket owner verification (opensearch-project#18312) Signed-off-by: Jay Deng <[email protected]>

Gagan6164 pushed a commit to Gagan6164/OpenSearch that referenced this pull request Jun 8, 2025

Add support for SSE-KMS, remove old server_side_encryption setting, a…

867e8f8

…dd support for bucket owner verification (opensearch-project#18312) Signed-off-by: Jay Deng <[email protected]>

BrewTestBot mentioned this pull request Jun 24, 2025

opensearch 3.1.0 Homebrew/homebrew-core#228059

Merged

Jorgesnchz mentioned this pull request Jun 27, 2025

Compatibility with OpenSearch 3.1.0 wazuh/wazuh-indexer#1000

Closed

6 tasks

Add support for SSE-KMS and bucket owner verification #18312

Add support for SSE-KMS and bucket owner verification #18312

Uh oh!

Conversation

jed326 commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

1. Remove the repository-s3 setting server_side_encryption

2. Add settings to support SSE-KMS

3. Add bucket owner verification

Testing

Related Issues

Check List

Uh oh!

github-actions bot commented May 15, 2025

Uh oh!

github-actions bot commented May 15, 2025

Uh oh!

github-actions bot commented May 16, 2025

Uh oh!

codecov bot commented May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented May 16, 2025

Uh oh!

github-actions bot commented May 16, 2025

Uh oh!

github-actions bot commented May 19, 2025

Uh oh!

github-actions bot commented May 19, 2025

Uh oh!

github-actions bot commented May 19, 2025

Uh oh!

github-actions bot commented May 19, 2025

Uh oh!

github-actions bot commented May 19, 2025

Uh oh!

github-actions bot commented May 19, 2025

Uh oh!

jed326 commented May 20, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented May 20, 2025

Uh oh!

jed326 commented May 21, 2025

Uh oh!

github-actions bot commented May 21, 2025

Uh oh!

gbbafna commented May 21, 2025

Uh oh!

jed326 commented May 21, 2025

Uh oh!

jed326 commented May 21, 2025

Uh oh!

gbbafna commented May 22, 2025

Uh oh!

Uh oh!

Bukhtawar May 25, 2025

Choose a reason for hiding this comment

Uh oh!

jed326 May 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jed326 commented May 15, 2025 •

edited

Loading

1. Remove the repository-s3 setting `server_side_encryption`

codecov bot commented May 16, 2025 •

edited

Loading