NSFS | S3 | Versioning: Deleting object versions except the latest breaks ListObjectVersions

[hseipp]

Environment info

• NooBaa Version: 5.15.4
• Platform: CentOS 8 (deployed using https://github.com/IBM/StorageScaleVagrant

Actual behavior

ListObjectVersions does not return a VersionId for last remaining version after all previous versions of a key are deleted.
Instead, ListObjectVersions only returns:

{
    "RequestCharged": null
}

Expected behavior

ListObjectVersions should report correct versioning information for the object. Example:

{
    "Versions": [
        {
            "ETag": "\"mtime-d3qrmcqwi1og-ino-105s\"",
            "Size": 9,
            "StorageClass": "STANDARD",
            "Key": "testobj",
            "VersionId": "mtime-d3qrmcqwi1og-ino-105s",
            "IsLatest": true,
            "LastModified": "2024-08-27T14:31:45+00:00",
            "Owner": {
                "DisplayName": "NooBaa",
                "ID": "123"
            }
        }
    ],
    "RequestCharged": null
}

Steps to reproduce

Execute Ceph s3-tests, function test_versioning_obj_create_read_remove()

S3TEST_CONF=s3tests.conf tox -- s3tests_boto3/functional/test_s3.py::test_versioning_obj_create_read_remove

Alternatively run the following python script (adjust profile and certificate to your needs):

#!/usr/bin/python
import boto3
session = boto3.Session(profile_name='s3user1')
endpoint = 'https://localhost:6443'
cos = session.client('s3', endpoint_url=endpoint, verify='/home/vagrant/aws-cert/tls.crt')
bucket_name = "s3tests-gr7nhf7l22gctexhbn5kw-1"
response = cos.create_bucket(Bucket=bucket_name)
response = cos.put_bucket_versioning(Bucket=bucket_name, VersioningConfiguration={'Status': 'Enabled'})
read_status = None
expected_string  = 'Enabled'
for i in range(5):
  try:
    response = cos.get_bucket_versioning(Bucket=bucket_name)
    read_status = response['Status']
  except KeyError:
    read_status = None
  if (expected_string == read_status):
            break
  time.sleep(1)
assert expected_string == read_status
version_ids = []
for i in range(5):
  data = 'content-{i}'.format(i=i)
  response = cos.put_object(Bucket=bucket_name, Key='testobj', Body=data)
  version_id = response['VersionId']
  version_ids.append(version_id)

for i in range(5):
  rm_version_id = version_ids.pop(0)
  cos.delete_object(Bucket=bucket_name, Key='testobj', VersionId=rm_version_id)
  response = cos.list_object_versions(Bucket=bucket_name)
  print("Versions: {}".format(response['Versions']))

More information - Screenshots / Logs / Other output

Output of the Python script with the error:

Versions: [{'ETag': '"mtime-d3rk3k4prklc-ino-nbe"', 'Size': 9, 'StorageClass': 'STANDARD', 'Key': 'testobj', 'VersionId': 'mtime-d3rk3k4prklc-ino-nbe', 'IsLatest': True, 'LastModified': datetime.datetime(2024, 8, 28, 12, 50, 45, tzinfo=tzlocal()), 'Owner': {'DisplayName': 'NooBaa', 'ID': '123'}}, {'ETag': '"mtime-d3rk3k4fk2yo-ino-nbd"', 'Size': 9, 'StorageClass': 'STANDARD', 'Key': 'testobj', 'VersionId': 'mtime-d3rk3k4fk2yo-ino-nbd', 'IsLatest': True, 'LastModified': datetime.datetime(2024, 8, 28, 12, 50, 45, tzinfo=tzlocal()), 'Owner': {'DisplayName': 'NooBaa', 'ID': '123'}}, {'ETag': '"mtime-d3rk3k3xc6ps-ino-nbc"', 'Size': 9, 'StorageClass': 'STANDARD', 'Key': 'testobj', 'VersionId': 'mtime-d3rk3k3xc6ps-ino-nbc', 'IsLatest': True, 'LastModified': datetime.datetime(2024, 8, 28, 12, 50, 45, tzinfo=tzlocal()), 'Owner': {'DisplayName': 'NooBaa', 'ID': '123'}}, {'ETag': '"mtime-d3rk3k37arcw-ino-nba"', 'Size': 9, 'StorageClass': 'STANDARD', 'Key': 'testobj', 'VersionId': 'mtime-d3rk3k37arcw-ino-nba', 'IsLatest': True, 'LastModified': datetime.datetime(2024, 8, 28, 12, 50, 45, tzinfo=tzlocal()), 'Owner': {'DisplayName': 'NooBaa', 'ID': '123'}}]
Versions: [{'ETag': '"mtime-d3rk3k4prklc-ino-nbe"', 'Size': 9, 'StorageClass': 'STANDARD', 'Key': 'testobj', 'VersionId': 'mtime-d3rk3k4prklc-ino-nbe', 'IsLatest': True, 'LastModified': datetime.datetime(2024, 8, 28, 12, 50, 45, tzinfo=tzlocal()), 'Owner': {'DisplayName': 'NooBaa', 'ID': '123'}}, {'ETag': '"mtime-d3rk3k4fk2yo-ino-nbd"', 'Size': 9, 'StorageClass': 'STANDARD', 'Key': 'testobj', 'VersionId': 'mtime-d3rk3k4fk2yo-ino-nbd', 'IsLatest': True, 'LastModified': datetime.datetime(2024, 8, 28, 12, 50, 45, tzinfo=tzlocal()), 'Owner': {'DisplayName': 'NooBaa', 'ID': '123'}}, {'ETag': '"mtime-d3rk3k3xc6ps-ino-nbc"', 'Size': 9, 'StorageClass': 'STANDARD', 'Key': 'testobj', 'VersionId': 'mtime-d3rk3k3xc6ps-ino-nbc', 'IsLatest': True, 'LastModified': datetime.datetime(2024, 8, 28, 12, 50, 45, tzinfo=tzlocal()), 'Owner': {'DisplayName': 'NooBaa', 'ID': '123'}}]
Versions: [{'ETag': '"mtime-d3rk3k4prklc-ino-nbe"', 'Size': 9, 'StorageClass': 'STANDARD', 'Key': 'testobj', 'VersionId': 'mtime-d3rk3k4prklc-ino-nbe', 'IsLatest': True, 'LastModified': datetime.datetime(2024, 8, 28, 12, 50, 45, tzinfo=tzlocal()), 'Owner': {'DisplayName': 'NooBaa', 'ID': '123'}}, {'ETag': '"mtime-d3rk3k4fk2yo-ino-nbd"', 'Size': 9, 'StorageClass': 'STANDARD', 'Key': 'testobj', 'VersionId': 'mtime-d3rk3k4fk2yo-ino-nbd', 'IsLatest': True, 'LastModified': datetime.datetime(2024, 8, 28, 12, 50, 45, tzinfo=tzlocal()), 'Owner': {'DisplayName': 'NooBaa', 'ID': '123'}}]
Traceback (most recent call last):
  File "create-versioned-bucket-upload-object_multiple.py", line 33, in <module>
    print("Versions: {}".format(response['Versions']))
KeyError: 'Versions'

Noobaa log with "all" mode enabled:

noobaa_20240828_1455.log.gz

[romayalon]

@nadavMiz can you please take a look?

[hseipp]

Please note that the same error occurs when using Ceph s3-tests test_versioning_obj_create_versions_remove_all():

S3TEST_CONF=s3tests.conf tox -- s3tests_boto3/functional/test_s3.py::test_versioning_obj_create_versions_remove_all
...
client = <botocore.client.S3 object at 0x7f29f51579d0>, bucket_name = 's3tests-l07lrjq90dn6imxsfi5dj-1', key = 'testobj', version_ids = ['mtime-d3ro2a9o7f28-ino-1es0']
contents = ['content-9']

    def check_obj_versions(client, bucket_name, key, version_ids, contents):
        # check to see if objects is pointing at correct version
    
        response = client.list_object_versions(Bucket=bucket_name)
        versions = []
>       versions = response['Versions']
E       KeyError: 'Versions'

[hseipp]

And another variant of the same issue - Ceph s3-tests test_versioning_obj_create_versions_remove_special_names():

contents = ['content-9']

    def check_obj_versions(client, bucket_name, key, version_ids, contents):
        # check to see if objects is pointing at correct version
    
        response = client.list_object_versions(Bucket=bucket_name)
        versions = []
>       versions = response['Versions']
E       KeyError: 'Versions'

Please note that due to the empty ListBucketVersion response also the teardown() of the s3-tests framework is failing because it deletes the objects in the bucket based on the ListBucketVersion information before trying to delete the bucket. In our case, the bucket will not be emptied as the ListBucketVersion response contains nothing except the RequestCharged entry while one version of the object is still present. Thus we get a BucketNotEmpty exception:

s3tests_boto3/functional/__init__.py:303: in teardown
    nuke_prefixed_buckets(prefix=prefix)
s3tests_boto3/functional/__init__.py:159: in nuke_prefixed_buckets
    raise err
s3tests_boto3/functional/__init__.py:150: in nuke_prefixed_buckets
    nuke_bucket(client, bucket_name)
s3tests_boto3/functional/__init__.py:139: in nuke_bucket
    client.delete_bucket(Bucket=bucket)
.tox/py/lib/python3.8/site-packages/botocore/client.py:569: in _api_call
    return self._make_api_call(operation_name, kwargs)

....
        if http.status_code >= 300:
            error_info = parsed_response.get("Error", {})
            error_code = error_info.get("QueryErrorCode") or error_info.get(
                "Code"
            )
            error_class = self.exceptions.from_code(error_code)
>           raise error_class(parsed_response, operation_name)
E           botocore.exceptions.ClientError: An error occurred (BucketNotEmpty) when calling the DeleteBucket operation: The bucket you tried to delete is not empty. You must delete all versions in the bucket.