Skip to content
This repository was archived by the owner on Jun 29, 2025. It is now read-only.

Conversation

thereis
Copy link
Contributor

@thereis thereis commented Apr 23, 2025

This pull request enhances the S3FileService class by improving error handling, adding detailed logging, and validating inputs to ensure robustness and maintainability.

When trying the integration of Pingvin with Minio, I had the issue where the bucket content could not be deleted. I've added the path of the bucket as /, which caused an error in the backend while trying to delete the assets.

image

The error message is also vague, so I added the error object and some debug messages to give a better context of what is happening under the hood:

Error thrown on main when trying to delete the bucket contents on a Minio server.

2025-04-23T22:54:08.017219025Z [Nest] 56  - 04/23/2025, 10:54:08 PM   ERROR ;5;3m[ExceptionsHandler] Could not delete all files from S3
2025-04-23T22:54:08.017276391Z Error: Could not delete all files from S3
2025-04-23T22:54:08.017279026Z     at S3FileService.deleteAllFiles (/opt/app/backend/dist/src/file/s3.service.js:177:19)
2025-04-23T22:54:08.017280779Z     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2025-04-23T22:54:08.017282423Z     at async ShareService.remove (/opt/app/backend/dist/src/share/share.service.js:236:9)
2025-04-23T22:54:08.017284346Z     at async ShareController.remove (/opt/app/backend/dist/src/share/share.controller.js:69:9)

After cloning and adding the error object to the error message:

{
  error: XMinioInvalidObjectName: Object name contains unsupported characters.
      at throwDefaultError (C:\Projetos\pingvin-share\backend\node_modules\@smithy\smithy-client\dist-cjs\index.js:867:20)        
      at C:\Projetos\pingvin-share\backend\node_modules\@smithy\smithy-client\dist-cjs\index.js:876:5
      at de_CommandError (C:\Projetos\pingvin-share\backend\node_modules\@aws-sdk\client-s3\dist-cjs\index.js:4952:14)
      at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
      at async C:\Projetos\pingvin-share\backend\node_modules\@smithy\middleware-serde\dist-cjs\index.js:35:20
      at async C:\Projetos\pingvin-share\backend\node_modules\@aws-sdk\middleware-sdk-s3\dist-cjs\index.js:484:18
      at async C:\Projetos\pingvin-share\backend\node_modules\@smithy\middleware-retry\dist-cjs\index.js:320:38
      at async C:\Projetos\pingvin-share\backend\node_modules\@aws-sdk\middleware-sdk-s3\dist-cjs\index.js:110:22
      at async C:\Projetos\pingvin-share\backend\node_modules\@aws-sdk\middleware-sdk-s3\dist-cjs\index.js:137:14
      at async C:\Projetos\pingvin-share\backend\node_modules\@aws-sdk\middleware-logger\dist-cjs\index.js:33:22 {
    '$fault': 'client',
    '$metadata': {
      httpStatusCode: 400,
      requestId: '18391315A583A4E6',
      extendedRequestId: 'dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8',
      cfId: undefined,
      attempts: 1,
      totalRetryDelay: 0
    },
    Code: 'XMinioInvalidObjectName',
    BucketName: 'mishare',
    Resource: '/mishare/',
    Region: 'eu-west',
    RequestId: '18391315A583A4E6',
    HostId: 'dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8'
  },
  shareId: 'Eo7pQuzW'
}

Now

When uploading files to a s3 bucket:

[Nest] 15092  - 04/24/2025, 12:59:16 AM   DEBUG [S3FileService] Processing chunk 1/1 for file "vTnV6nMf/730_20240311181546_1.png"
[Nest] 15092  - 04/24/2025, 12:59:16 AM   DEBUG [S3FileService] Initialized multipart upload for "vTnV6nMf/730_20240311181546_1.png" with uploadId "ZDRhODY2OTAtZDExMS00MDhmLThmOWUtNjFlMGNjZmQxNDMxLmI2NDFkMGMyLWJkMGItNDRlZi05MmEwLWJkYWZhMTBlNmRiZHgxNzQ1NDQ5MTU2NDE3Mzk1NzE4"
[Nest] 15092  - 04/24/2025, 12:59:16 AM   DEBUG [S3FileService] Uploaded part 1 for "vTnV6nMf/730_20240311181546_1.png"
[Nest] 15092  - 04/24/2025, 12:59:16 AM   DEBUG [S3FileService] Completed multipart upload for "vTnV6nMf/730_20240311181546_1.png"
[Nest] 15092  - 04/24/2025, 12:59:16 AM   DEBUG [S3FileService] Created database record for file "730_20240311181546_1.png" with ID "1ee927dc-aa2b-480f-80cd-24afeaa8317e"

When deleting the shared content from pingvin:

[Nest] 15092  - 04/24/2025, 1:00:20 AM   DEBUG [S3FileService] Attempting to delete files with prefix "vTnV6nMf/" from bucket "mishare"
[Nest] 15092  - 04/24/2025, 1:00:20 AM   DEBUG [S3FileService] ListObjectsV2Command response: Found 1 objects
[Nest] 15092  - 04/24/2025, 1:00:20 AM   DEBUG [S3FileService] Object(2) {
  prefix: 'vTnV6nMf/',
  keyCount: 1
}
[Nest] 15092  - 04/24/2025, 1:00:20 AM   DEBUG [S3FileService] Deleting 1 objects
[Nest] 15092  - 04/24/2025, 1:00:20 AM   DEBUG [S3FileService] Object(1) {
  firstFew: [
    'vTnV6nMf/730_20240311181546_1.png'
  ]
}
[Nest] 15092  - 04/24/2025, 1:00:20 AM   DEBUG [S3FileService] Successfully deleted 1 files for share vTnV6nMf

…tions

- Added validation for share ID and file name formats in multiple methods to prevent invalid inputs.
- Improved error logging for multipart upload failures and file deletion operations.
- Added debug logging for key operations, including multipart upload initialization, part uploads, and file deletions.
- Ensured consistent error handling by throwing InternalServerErrorException for various failure scenarios.
@thereis thereis changed the title feat(s3.service): add support to minio file deletion feat(s3): add support to minio file deletion Apr 23, 2025
@thereis thereis changed the title feat(s3): add support to minio file deletion fix(s3): format path when is '/' Apr 23, 2025
@thereis thereis changed the title fix(s3): format path when is '/' fix(s3): format path when bucket path is '/' Apr 23, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant