aws rds generate-db-auth-token generating invalid token when using .aws/credentials to authenticate

[scott-vandevoorde]

Describe the bug

When using credentials files (.aws/credentials & .aws/config) to authenticate with aws, aws rds generate-db-auth-token generates an invalid token.

The invalid token is missing the field X-Amz-Security-Token

If I set environment variables AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN, the token is generated successfully and is usable.

Expected Behavior

I expect the cli to generate a working token

Current Behavior

an invalid token is generated. The db says password authentication failed for user "rdsproxyuser"

Reproduction Steps

1. create an iam user
2. add unrestricted access to the user { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "*", "Resource": "*" } ] }
3. run aws configure
4. enter your access key, secret acccess key and region
5. test your cli access to aws. For example, run aws s3 ls and ensure it connects ok
6. stand up an aurora serverless v2 postgres RDS instance
7. stand up an rds proxy and configure it for IAM authentication
8. verify you can connect and login to the RDS proxy
9. on the command line run aws rds generate-db-auth-token --hostname adhoc3-amber-engine.proxy-cfds9ixmayu4.us-east-1.rds.amazonaws.com --port 5432 --region us-east-1 --username rdsproxyuser (replace rds host name, and user name with the appropriate info)
10. copy the generated token into an RDS connection and it will fail
11. note that the token does not containt the field X-Amz-Security-Token
12. Create a role with the following policy: { "Action": [ "rds-db:connect" ], "Resource": "arn:aws:rds-db:us-east-1:*:dbuser:*/*", "Effect": "Allow" }
13. Update the trust policy so you can assume this role: { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::793478358630:user/scott" }, "Action": "sts:AssumeRole" }
14. run the following command on the command line to assume the role and set your environment variablesexport $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s" \ $(aws sts assume-role \ --role-arn arn:aws:iam::793478358630:role/Adhoc3BastionStack-BastionHostRole-JDL6Y8KJ69VB \ --role-session-name Adhoc3BastionSession \ --query "Credentials.[AccessKeyId,SecretAccessKey,SessionToken]" \ --output text))
15. run generate-db-auth-token again: aws rds generate-db-auth-token --hostname adhoc3-amber-engine.proxy-cfds9ixmayu4.us-east-1.rds.amazonaws.com --port 5432 --region us-east-1 --username rdsproxyuser
16. copy the generated token and paste it to your rds connection.
17. this time the token will work.
18. Note that the token now contains X-Amz-Security-Token.

Possible Solution

trace the cli workflows between the two authentication mechanisms and figure out why the X-Amz-Security-Token is not being included in the token. Also recommend generating an error message instead of a bad tokens.

Additional Information/Context

Two co-workers validated the same behavior. One on windows and one on mac.

CLI version used

aws-cli/2.10.3 Python/3.9.11 Windows/10 exe/AMD64 prompt/off

Environment details (OS name and version, etc.)

tested on windows 10

[aBurmeseDev]

Hi @scott-vandevoorde - thanks for reaching out and for well-detailed post.

I followed steps to reproduce the issue but was unable to and I'd like verify a couple things:

• on step 7, you mentioned to stand up RDS proxy and configure it for IAM auth. Could you share how your IAM auth setup is like? Here's how I enabled during my setup: (reference guide)

aws rds create-db-instance \
    --db-instance-identifier mydbinstance \
    --db-instance-class db.m3.medium \
    --engine MySQL \
    --allocated-storage 20 \
    --master-username masterawsuser \
    --manage-master-user-password \
    --enable-iam-database-authentication

• Step 12 and 13 is also mentioned here in these docs: here and here
• I'd also like to clarify that when you start RDS connection with generated token, how are you using the token to connect? Here's how I generate the token and connect:

#  save the token to an environment variable, and use that variable to connect
export RDSHOST="mypostgres-cluster.cluster-abcdefg222hq.us-east-1.rds.amazonaws.com"

export PGPASSWORD="$(aws rds generate-db-auth-token \
--hostname $RDSHOST \
--port 5432 \
--region us-east-1 \
--username mydbuser)"

psql "host=$RDSHOST port=5432 sslmode=verify-full sslrootcert=/sample_dir/rds-combined-ca-bundle.pem dbname=dbName user= mydbuser"

• I came across similar issue where a user commented the culprit was "found the issue it was due to an empty extra aws_session_token = and the end of ~/.aws/credentials" and another user suggested a "workaround" here.
• Found another similar issue where one of our members suggested running aws sts get-caller-identity before token generation could help.

If the issue persists, could you share full debug logs from generate-db-auth-token command by adding --debug to the command, as that would give us more insight into this behavior? Please make sure to redact any sensitive info before sharing.

Hope that helps,
John

[scott-vandevoorde]

Hi @aBurmeseDev , thanks for looking into this issue!

I think the main difference in our setups is I'm connecting to an IAM enabled RDS proxy. My RDS instance doesn't have IAM configured as the proxy handles it.

Here's more detail around my setup. Our instances are complex, so I stood up a slimmed down stack and I was able to reproduce with this configuration:

1. Create a db instance. Update the db-subnet-group-name to choose which VPC you provision into.

aws rds create-db-instance \
    --db-instance-identifier temp-db \
    --db-instance-class db.m5d.large \
    --engine postgres \
    --allocated-storage 100 \
    --master-username masterawsuser \
    --manage-master-user-password \
	--db-subnet-group-name adhoc3-amber-engine-rds-adhoc3rdssubnets93ef7f60-zkd5nnppgsot

2. Create a policy to allow RDS proxy to fetch secrets from secrets manager:

aws iam create-policy \
--policy-name my-rds-proxy-sample-policy \
--policy-document '{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"secretsmanager:GetResourcePolicy",
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret",
"secretsmanager:ListSecretVersionIds"
],
"Resource": [
"*"
]
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"secretsmanager:GetRandomPassword",
"secretsmanager:ListSecrets"
],
"Resource": "*"
}
]
}'

3. Create a role to allow the rds proxy to assume role:

aws iam create-role --role-name my-rds-proxy-sample-role --assume-role-policy-document '{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "rds.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}'

4. attach the policy to the role: (update your AWS account # in the arn)

aws iam attach-role-policy --role-name my-rds-proxy-sample-role --policy-arn arn:aws:iam::123456789:policy/my-rds-proxy-sample-policy

5. Create the RDS proxy. (update the secret arn to your RDS secret, your role arn and vpc subnet ids)

aws rds create-db-proxy \
--db-proxy-name temp-db-proxy \
--engine-family POSTGRESQL \
--auth '{
"AuthScheme": "SECRETS",
"SecretArn": "arn:aws:secretsmanager:us-east-1:123456789:secret:rds!db-d7bb886b-b548-4d21-b15d-694d34c3c9b1-MbGtmT",
"IAMAuth": "REQUIRED"
}' \
--role-arn arn:aws:iam::123456789:role/my-rds-proxy-sample-role \
--vpc-subnet-ids subnet-09c83b23370eeb245 subnet-04dff361ae0758434  \
--require-tls

6. Register proxy targets to point your proxy to the rds instance

aws rds register-db-proxy-targets --db-proxy-name temp-db-proxy --db-instance-identifiers temp-db

7. Go to your RDS proxy in the AWS console and find it's security group. Allow inbound traffic from wherever you'd like to connect from.

------ testing the newly provisioned setup ------

We have an ec2 bastion running in the VPC. When I run the following. it can connect to the RDS proxy via IAM token:

export RDSHOST="temp-db-proxy.proxy-cfds9ixmayu4.us-east-1.rds.amazonaws.com"
export PGPASSWORD="$(aws rds generate-db-auth-token --hostname $RDSHOST --port 5432 --region us-east-1 --username masterawsuser)"
psql -h $RDSHOST -p 5432 "sslmode=require dbname=postgres user=masterawsuser"

Tokens generated on my local workbench via the same commands don't work.

once I set the following environment variables, aws rds generate-db-auth-token starts to generate working tokens

export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s" \
 $(aws sts assume-role \
 --role-arn arn:aws:iam::123456789:role/Adhoc3BastionStack-BastionHostRole-JDL6Y8KJ69VB \
 --role-session-name Adhoc3BastionSession \
 --query "Credentials.[AccessKeyId,SecretAccessKey,SessionToken]" \
 --output text))

Note: Adhoc3BastionStack-BastionHostRole-JDL6Y8KJ69VB is the role on the working bastion instance. My stored credentials have the following permissions, so it should be an issue with privilege.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "*", "Resource": "*" } ] }

I checked ~/.aws/credentials to see if anything was out of place. no dangling aws_session_token =. that would have been a great catch though.

aws sts get-caller-identity is quite interesting. I unset my environment variables:

unset AWS_ACCESS_KEY_ID; 
unset AWS_SECRET_ACCESS_KEY; 
unset AWS_SESSION_TOKEN

then I called aws sts get-caller-identity and the token generation started working properly! The tokens are still missing the field X-Amz-Security-Token but still seem to work fine without it. Do you have more info on what that does? is it refreshing the client session? I read the link you provided, and it looks like my session might expire again, so I'm going to keep an eye on it to see if things break again.

[aBurmeseDev]

Glad to hear that it's working as expected now after running aws sts get-caller-identity. The command is to confirm and validate the identity of the caller and IAM role. Here's more about it for reference: https://docs.aws.amazon.com/STS/latest/APIReference/API_GetCallerIdentity.html

[scott-vandevoorde]

thanks for the detail!