eks update-kubeconfig command does not assume role

[carlosrodf]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• I've gone though the User Guide and the API reference
• I've searched for previous similar issues and didn't find any solution

Describe the bug
When running aws eks update-kubeconfig --name <CLUSTER_NAME> --role-arn <ROLE_ARN> the command returns an access denied error:

An error occurred (AccessDeniedException) when calling the DescribeCluster operation: User: arn:aws:iam::*****:user/****** is not authorized to perform: eks:DescribeCluster on resource: arn:aws:eks:***:*****:cluster/******

The IAM user has permissions to assume the role in question and the role has full access to EKS. I have confirmed the credentials in both user and role work by performing the steps manually:

1. assuming the role manually through CLI
2. exporting assumed role environment variables
3. running aws eks update-kubeconfig... again

I report this as a bug because this command has the expected behaviour using the same IAM user and role

aws eks get-token --cluster-name <CLUSTER_NAME> --role-arn <ROLE_ARN> 

SDK version number

Platform/OS/Hardware/Device
aws-cli/2.2.3 Python/3.8.8 Linux/5.11.0-34-generic exe/x86_64.ubuntu.20 prompt/off

To Reproduce (observed behavior)
Steps to reproduce the behavior

Expected behavior
The command should assume the role specified by --role-arn and update ~/.kube/config file

Logs/output

2021-09-09 15:02:33,431 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.2.3 Python/3.8.8 Linux/5.11.0-34-generic exe/x86_64.ubuntu.20
2021-09-09 15:02:33,432 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['eks', 'update-kubeconfig', '--name', '*****************', '--role-arn', 'arn:aws:iam::**************:role/****************', '--debug']
2021-09-09 15:02:33,438 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x7fd731bd7670>
2021-09-09 15:02:33,438 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x7fd731d92430>
2021-09-09 15:02:33,438 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
2021-09-09 15:02:33,438 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7fd731db8c10>
2021-09-09 15:02:33,438 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7fd731dc2d30>
2021-09-09 15:02:33,438 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x7fd731be90d0>
2021-09-09 15:02:33,438 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x7fd731d61280>
2021-09-09 15:02:33,438 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
2021-09-09 15:02:33,438 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x7fd731be2310>
2021-09-09 15:02:33,438 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.2.3/dist/awscli/data/cli.json
2021-09-09 15:02:33,440 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_types at 0x7fd731c90280>
2021-09-09 15:02:33,440 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function no_sign_request at 0x7fd731c90dc0>
2021-09-09 15:02:33,440 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x7fd731c90d30>
2021-09-09 15:02:33,440 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x7fd731c90ee0>
2021-09-09 15:02:33,440 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x7fd731c90e50>
2021-09-09 15:02:33,441 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x7fd731afed40>
2021-09-09 15:02:33,441 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.2.3 Python/3.8.8 Linux/5.11.0-34-generic exe/x86_64.ubuntu.20 prompt/off
2021-09-09 15:02:33,441 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['eks', 'update-kubeconfig', '--name', '*****************', '--role-arn', 'arn:aws:iam::**************:role/****************', '--debug']
2021-09-09 15:02:33,441 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x7fd731bd7ca0>
2021-09-09 15:02:33,441 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7fd73266df70>
2021-09-09 15:02:33,441 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x7fd731b44af0>
2021-09-09 15:02:33,441 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function no_pager_handler at 0x7fd732669430>
2021-09-09 15:02:33,441 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7fd7325e9c10>
2021-09-09 15:02:33,442 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2021-09-09 15:02:33,443 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7fd731d61160>
2021-09-09 15:02:33,443 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x7fd731d8f310>
2021-09-09 15:02:33,449 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.2.3/dist/botocore/data/eks/2017-11-01/service-2.json
2021-09-09 15:02:33,450 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.2.3/dist/botocore/data/eks/2017-11-01/service-2.sdk-extras.json
2021-09-09 15:02:33,453 - MainThread - botocore.hooks - DEBUG - Event building-command-table.eks: calling handler <function inject_commands at 0x7fd731c6dc10>
2021-09-09 15:02:33,454 - MainThread - botocore.hooks - DEBUG - Event building-command-table.eks: calling handler <function add_waiters at 0x7fd731be2310>
2021-09-09 15:02:33,459 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.2.3/dist/botocore/data/eks/2017-11-01/waiters-2.json
2021-09-09 15:02:33,460 - MainThread - botocore.hooks - DEBUG - Event building-command-table.eks_update-kubeconfig: calling handler <function add_waiters at 0x7fd731be2310>
2021-09-09 15:02:33,460 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.update-kubeconfig.name: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fd7312ba1f0>
2021-09-09 15:02:33,460 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.update-kubeconfig: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fd73261b970>
2021-09-09 15:02:33,460 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.update-kubeconfig.kubeconfig: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fd7312ba1f0>
2021-09-09 15:02:33,460 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.update-kubeconfig.role-arn: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fd7312ba1f0>
2021-09-09 15:02:33,460 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.update-kubeconfig: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fd73261b970>
2021-09-09 15:02:33,460 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.update-kubeconfig.dry-run: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fd7312ba1f0>
2021-09-09 15:02:33,460 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.update-kubeconfig: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fd73261b970>
2021-09-09 15:02:33,461 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.update-kubeconfig.verbose: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fd7312ba1f0>
2021-09-09 15:02:33,461 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.update-kubeconfig: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fd73261b970>
2021-09-09 15:02:33,461 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.update-kubeconfig.alias: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fd7312ba1f0>
2021-09-09 15:02:33,461 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2021-09-09 15:02:33,461 - MainThread - botocore.credentials - INFO - Found credentials in environment variables.
2021-09-09 15:02:33,461 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.2.3/dist/botocore/data/endpoints.json
2021-09-09 15:02:33,466 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fd733f42c10>
2021-09-09 15:02:33,467 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.eks: calling handler <function add_generate_presigned_url at 0x7fd733f70f70>
2021-09-09 15:02:33,471 - MainThread - botocore.endpoint - DEBUG - Setting eks timeout as (60, 60)
2021-09-09 15:02:33,472 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.eks.DescribeCluster: calling handler <function base64_decode_input_blobs at 0x7fd731b47280>
2021-09-09 15:02:33,472 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.eks.DescribeCluster: calling handler <function generate_idempotent_uuid at 0x7fd733f66ca0>
2021-09-09 15:02:33,472 - MainThread - botocore.hooks - DEBUG - Event before-call.eks.DescribeCluster: calling handler <function inject_api_version_header_if_needed at 0x7fd733eec550>
2021-09-09 15:02:33,472 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=DescribeCluster) with params: {'url_path': '/clusters/*****************', 'query_string': {}, 'method': 'GET', 'headers': {'User-Agent': 'aws-cli/2.2.3 Python/3.8.8 Linux/5.11.0-34-generic exe/x86_64.ubuntu.20 prompt/off command/eks.update-kubeconfig'}, 'body': b'', 'url': 'https://eks.********.amazonaws.com/clusters/*****************', 'context': {'client_region': '********', 'client_config': <botocore.config.Config object at 0x7fd730f2e790>, 'has_streaming_input': False, 'auth_type': None}}
2021-09-09 15:02:33,472 - MainThread - botocore.hooks - DEBUG - Event request-created.eks.DescribeCluster: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fd730f2e850>>
2021-09-09 15:02:33,472 - MainThread - botocore.hooks - DEBUG - Event choose-signer.eks.DescribeCluster: calling handler <function set_operation_specific_signer at 0x7fd733f66b80>
2021-09-09 15:02:33,472 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2021-09-09 15:02:33,472 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
GET
/clusters/*****************

host:eks.********.amazonaws.com
x-amz-date:20210909T210233Z

host;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2021-09-09 15:02:33,472 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20210909T210233Z
20210909/********/eks/aws4_request
3ce24536fc7e0fb589c5154391c8d2a8997a0646e8914ff47527cc9fbba1fec0
2021-09-09 15:02:33,472 - MainThread - botocore.auth - DEBUG - Signature:
20d646a326805f289bebce51f7b259b054818128db6eef6a9cabc28f20bcbebd
2021-09-09 15:02:33,473 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://eks.********.amazonaws.com/clusters/*****************, headers={'User-Agent': b'aws-cli/2.2.3 Python/3.8.8 Linux/5.11.0-34-generic exe/x86_64.ubuntu.20 prompt/off command/eks.update-kubeconfig', 'X-Amz-Date': b'20210909T210233Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIARCH5EZTP5IDSZ5N2/20210909/********/eks/aws4_request, SignedHeaders=host;x-amz-date, Signature=20d646a326805f289bebce51f7b259b054818128db6eef6a9cabc28f20bcbebd'}>
2021-09-09 15:02:33,473 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/v2/2.2.3/dist/botocore/cacert.pem
2021-09-09 15:02:33,473 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): eks.********.amazonaws.com:443
2021-09-09 15:02:33,815 - MainThread - urllib3.connectionpool - DEBUG - https://eks.********.amazonaws.com:443 "GET /clusters/***************** HTTP/1.1" 403 188
2021-09-09 15:02:33,816 - MainThread - botocore.parsers - DEBUG - Response headers: {'Date': 'Thu, 09 Sep 2021 21:02:33 GMT', 'Content-Type': 'application/json', 'Content-Length': '188', 'Connection': 'keep-alive', 'x-amzn-RequestId': '2f939d4a-ee77-4c97-8270-cd7c020d3f0a', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Headers': '*,Authorization,Date,X-Amz-Date,X-Amz-Security-Token,X-Amz-Target,content-type,x-amz-content-sha256,x-amz-user-agent,x-amzn-platform-id,x-amzn-trace-id', 'x-amzn-ErrorType': 'AccessDeniedException', 'x-amz-apigw-id': 'FadwjGnwoAMFzZw=', 'Access-Control-Allow-Methods': 'GET,HEAD,PUT,POST,DELETE,OPTIONS', 'Access-Control-Expose-Headers': 'x-amzn-errortype,x-amzn-errormessage,x-amzn-trace-id,x-amzn-requestid,x-amz-apigw-id,date', 'X-Amzn-Trace-Id': 'Root=1-613a7669-2f77bb8127e32a5956504c13'}
2021-09-09 15:02:33,816 - MainThread - botocore.parsers - DEBUG - Response body:
b'{"message":"User: arn:aws:iam::**************:user/************ is not authorized to perform: eks:DescribeCluster on resource: arn:aws:eks:********:**************:cluster/*****************"}'
2021-09-09 15:02:33,817 - MainThread - botocore.hooks - DEBUG - Event needs-retry.eks.DescribeCluster: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7fd730ef72e0>>
2021-09-09 15:02:33,817 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2021-09-09 15:02:33,817 - MainThread - botocore.hooks - DEBUG - Event after-call.eks.DescribeCluster: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7fd730f2ee20>>
2021-09-09 15:02:33,818 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "awscli/clidriver.py", line 459, in main
  File "awscli/clidriver.py", line 594, in __call__
  File "awscli/customizations/commands.py", line 191, in __call__
  File "awscli/customizations/eks/update_kubeconfig.py", line 122, in _run_main
  File "awscli/customizations/eks/update_kubeconfig.py", line 276, in get_cluster_entry
  File "awscli/customizations/eks/update_kubeconfig.py", line 258, in _get_cluster_description
  File "botocore/client.py", line 249, in _api_call
  File "botocore/client.py", line 568, in _make_api_call
botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the DescribeCluster operation: User: arn:aws:iam::**************:user/************ is not authorized to perform: eks:DescribeCluster on resource: arn:aws:eks:********:**************:cluster/*****************

An error occurred (AccessDeniedException) when calling the DescribeCluster operation: User: arn:aws:iam::**************:user/************ is not authorized to perform: eks:DescribeCluster on resource: arn:aws:eks:********:**************:cluster/*****************

[tim-finnigan]

Hi @carlosrodf, thanks for reaching out. Are you using a role with the IAM policy AmazonEKSClusterPolicy?

I created a Amazon EKS cluster IAM role following the steps documented here, and was able to successfully run the command aws eks update-kubeconfig --name <CLUSTER_NAME> --role-arn <ROLE_ARN>.

A similar issue was opened recently here: #5823. You could try following the steps suggested in this comment: #5823 (comment)

[carlosrodf]

Hi @tim-finnigan, in my case the IAM user is in a different AWS Account.

Account A:

• IAM User

Account B:

• IAM Role
• EKS Cluster

The trust relationship between the role and the user works fine. I have tested assuming the role manually and it works.

[tim-finnigan]

Hi @carlosrodf, thanks for following up. I found this blog post on enabling cross-account access to Amazon EKS cluster resources: https://aws.amazon.com/blogs/containers/enabling-cross-account-access-to-amazon-eks-cluster-resources/

Can you try following those steps and let us know if that works for you?

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.