Minio Client AWS Auth only supports Access Keys; Add Options to Credential Chain

[ghost]

Feature Description

Objective

We propose the following features added to the Gitea release cycle (release/v1.22):

• Merge the credential chain reference for the Minio client credentials from the main branch into the current release cycle
• Stretch goal: Add the Minio.Credential.IAM_AWS function getEKSPodIdentityCredentials to this credential chain at the end, this way native Kubernetes authentication is handled.

Details

We have deployed Gitea using the Helm Chart on our AWS EKS cluster. We’ve been able to stand it up with supporting infrastructure and figure out configurations, but we’ve discovered that the implementation of the Minio client under the Gitea release branch is only looking at AWS Access Keys (long term credentials) for authentication with AWS. The recommended best practice from AWS prefers relying on temporary credentials instead of long-term credentials, such as access keys where possible.

We hope to see implementation of short-lived tokens to authenticate the Minio client with AWS APIs, as this would provide better security and simplified implementation.

• The current latest release client, this only configures credentials using access keys - https://github.com/go-gitea/gitea/blob/v1.22.3/modules/storage/minio.go#L88-L92
• Compare this to the main branch where it calls the credential chain - https://github.com/go-gitea/gitea/blob/main/modules/storage/minio.go#L99-L105
  • This chain contains FileAwsCredentials, which would allow us to be able to move off of access keys - https://github.com/go-gitea/gitea/blob/main/modules/storage/minio.go#L174-L192
  • This is possible due to the credential_process, which thanks to this medium article we found is a working solution to grab aws_session_token via the Web Identity Token file

Expanding on the credential chain, this currently does not import other supported features from the minio-go SDK. Notably, as we are on AWS EKS, the ideal solution we were hoping to implement is using IRSA roles which would allow direct usage of the token file generated by the Service Account.

• This is the EKS function which would directly support short-lived access tokens using native Kubernetes features - https://github.com/minio/minio-go/blob/cca410398dd20409180d4ff2d017e34bea17c27c/pkg/credentials/iam_aws.go#L318
• To also confirm, the environment variable AWS_WEB_IDENTITY_TOKEN_FILE is set on the pods associated with our service account, pointing to the web identity token - https://github.com/minio/minio-go/blob/cca410398dd20409180d4ff2d017e34bea17c27c/pkg/credentials/iam_aws.go#L124-L127
• If this solution is implemented, then we do not need to manually configure any further logic into the Gitea deployment since all authentication will be natively handled by the Kubernetes Service Account

References

• Current release for minio client, only contains access keys not the credential chain - https://github.com/go-gitea/gitea/blob/v1.22.3/modules/storage/minio.go#L88-L92

• Main branch, this calls the credential chain - https://github.com/go-gitea/gitea/blob/main/modules/storage/minio.go#L99-L105

• Found here, and contains the FileAwsCredentials, which allows us to use credential_process to convert our web identity token -> session token - https://github.com/go-gitea/gitea/blob/main/modules/storage/minio.go#L174-L192

• This is the Minio-Go SDK code, where ideally we'd like to see this EKS function imported as well - https://github.com/minio/minio-go/blob/cca410398dd20409180d4ff2d017e34bea17c27c/pkg/credentials/iam_aws.go#L318

• Which directly looks at the env variable to grab our web identity token, as defined by associated a Kubernetes Service Account paired with an AWS IAM role - https://github.com/minio/minio-go/blob/cca410398dd20409180d4ff2d017e34bea17c27c/pkg/credentials/iam_aws.go#L124-L127

Screenshots

No response

[techknowlogick]

Thank you for your issue. We don't backport enhancements/features such as this, but we hope to release 1.23 this quarter, which would allow you to use this in a non-nightly version.

[ghost]

Hello, thank you for the quick response!

We have given the nightly-rootless image a try, and this helped us figure out some follow up issues as it wasn’t working right away, so we have one working solution via workaround but also another fix recommended to allow flexibility with AWS Auth using Minio for storage.

---

The first discovery is that the credential process workaround was not working immediately, but found it was due to this line the Minio SDK that is imported - https://github.com/minio/minio-go/blob/master/pkg/credentials/file_aws_credentials.go#L113

Since it’s checking for two arguments, we just added a dummy string afterwards and it worked perfectly to authenticate without the use of Access Keys. Working example:

credential_process = /data/gitea/git/gen-token dummy-text

Ideally it should be checking with < not <=, but that isn’t due to the implementation by Gitea and was easily worked around for us. So, this will work when it makes it into the 1.23 release for short term credentials, but this is ultimately only a workaround for native authentication, leading into the next section ->

---

Secondly, while reviewing the code again I realized I have misinterpreted what this line actually does, I now see this is a reference to the entire file from the Minio SDK, not just that one “EKSPodIdentity” function I mentioned originally.

However, this does not work out of the box when trying this on the nightly image (explicitly not passing the shared credential environment variable and access keys.) We discovered this is due to the EC2 Metadata endpoint being hard coded at this line which is passed to the IAM package here. Once the code proceeds to this function, at L150 of this switch case since the endpoint is provided it does not attempt to utilize the actual STS endpoint resulting in an error, and ultimately a "permission denied."

To summarize the issue here, it is trying to receive credentials using the normal EC2 Metadata endpoint with the identity file rather than the STS endpoint.

Proposed Solution

To work around the hard coded EC2 Metadata endpoint, we have the following idea:

1. Remove the credentials.DefaultIAMRoleEndpoint value here, as this is what determines usage of EC2 Metadata endpoint only - https://github.com/go-gitea/gitea/blob/main/modules/storage/minio.go#L100
2. Add a new config value for the storage block, which is accessed here - https://github.com/go-gitea/gitea/blob/main/modules/storage/minio.go#L187 (which calls the Minio SDK here https://github.com/minio/minio-go/blob/master/pkg/credentials/iam_aws.go)
   1. Possibly with the other storage configs here, it may make sense to call it something like MINIO_IAM_ENDPOINT to keep with consistency - https://docs.gitea.com/administration/config-cheat-sheet#storage-storage
3. This allows the EC2 endpoint to be passed for tests and still pass, but allows it to be empty by default for the SDK to decide for itself what endpoint it should use - https://github.com/go-gitea/gitea/pull/31051/files#diff-97ef7d5c55f1086fb6eb6bff28747793dd6dabfceadcda573db8c5ba0a0aa8e5R120

We’ll try and work on a PR that accomplishes our proposal, but wanted to get this update out first.

[techknowlogick]

Thanks for that detailed followup @vkice-at-ocp :)
I've re-opened this ticket to keep track of it.

[ghost]

Our PR was merged in to resolve our issue, closing!