GCS backend does not work with GCP identity federation

[hazcod]

Related ticket: google-github-actions/auth#17

The auth GitHub Action from Google allows us to federate identity to a GCP service account without having to export account credentials. Currently this does not work with the terraform GCS backend. Raw gcloud works fine.

Terraform Version

 1.0.7
Go runtime version: go1.16.4

Terraform Configuration Files

terraform {
  backend "gcs" {
    bucket = "foo"
  }
}

Debug Output

Crash Output

Expected Behavior

Terraform GCS backend working as usual.

Actual Behavior

Error: Failed to get existing workspaces: querying Cloud Storage failed: Get "https://storage.googleapis.com/storage/v1/b/xxx-terraform-state/o?alt=json&delimiter=%2F&pageToken=&prefix=terraform%2Fstate%2F&prettyPrint=false&projection=full&versions=false": oauth2/google: unable to generate access token: Post "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/terraform@xxx.iam.gserviceaccount.com:generateAccessToken": oauth2/google: status code 400: {"error":"invalid_target","error_description":"The target service indicated by the \"audience\" parameters is invalid. This might either be because the pool or provider is disabled or deleted or because it doesn't exist."}

Steps to Reproduce

Create a GitHub Action:

jobs:
  test:
    permissions:
      contents: read
      id-token: 'write'
    runs-on: ubuntu-latest
    steps: 
    -
      uses: actions/checkout@v2
    -
      uses: hashicorp/setup-terraform@v1
    -
      name: Authenticate to Google Cloud
      uses: google-github-actions/auth@v0.3.0
      with:
        workload_identity_provider: projects/xxx/locations/global/workloadIdentityPools/main-pool/providers/github
        service_account: terraform@xxx.iam.gserviceaccount.com
        create_credentials_file: true
        activate_credentials_file: true
    -
      name: Terraform Init
      run: terraform init

[sethvargo]

I think you just need to update to the latest cloud storage SDK and it'll work.

[jketcham]

Looks like cloud.google.com/go/storage is currently set to v1.10.0 here, and the latest version available is v1.18.2.

I'm running into an error with using google cloud storage as my state backend, where I get this error when using Workload Identity Federation for authentication:

Error: storage.NewClient() failed: dialing: google: error getting credentials using GOOGLE_APPLICATION_CREDENTIALS environment variable: unknown credential type: "external_account"

Would love to see this resolved. I'd be glad to provide additional info beyond what I shared in this issue to help.

[zotrix]

I rebuilt locally for tests the teraform with the new gcs version (v1.18.2) and the plan was successful.

pr: #30276

[hkobayash]

It looks like it is already fix #28296 .

[uportalis]

Are you sure this is resolved?

Here is how I created the Google Cloud resources:

gcloud iam workload-identity-pools create "my-pool" \
  --project="${PROJECT_ID}" \
  --location="global" \
  --display-name="Demo pool"


gcloud iam workload-identity-pools providers create-oidc "my-provider" \
  --project="${PROJECT_ID}" \
  --location="global" \
  --workload-identity-pool="my-pool" \
  --display-name="Demo provider" \
  --attribute-mapping="google.subject=assertion.sub,attribute.repository=assertion.repository" \
  --issuer-uri="https://token.actions.githubusercontent.com"


gcloud iam service-accounts add-iam-policy-binding "sb-github-actions@${PROJECT_ID}.iam.gserviceaccount.com" \
  --project="${PROJECT_ID}" \
  --role="roles/iam.workloadIdentityUser" \
  --member="principalSet://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/my-pool/attribute.repository/mycompany/myrepo"

(I even tried with principalSet://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/my-pool/*.)

Here is my workflow:

jobs:
  terraform:
    permissions:
      id-token: write
      contents: read
    runs-on: [self-hosted, linux, x64, on-prem, terraform]
    timeout-minutes: 45
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Authenticate to Google Cloud
        uses: google-github-actions/auth@v0
        with:
          workload_identity_provider: '${{secrets.IDENTITY_PROVIDER_ID}}'
          service_account: '${{secrets.SA_EMAIL}}'
          create_credentials_file: true
          export_environment_variables: true

      - name: Terraform init
        run: |
          terraform init -input=false

Here is my backend:

terraform {
  backend "gcs" {
    bucket = "myuniqueidentifier"
    prefix = "terraform/state/something"
  }
}

Here is my provider config:

terraform {
  required_version = "1.1.6"
  required_providers {
    google      = {
      source  = "hashicorp/google"
      version = "~> 3.90"
    }
  }
}

provider "google" {
  region = var.region
  zone   = var.zone
}

I double-checked and retried several times from scratch following documentation at https://cloud.google.com/blog/products/identity-security/enabling-keyless-authentication-from-github-actions and at https://cloud.google.com/iam/docs/using-workload-identity-federation#generate-automatic.

I tested granting my service account roles Storage Object Admin, then Storage Admin, then Owner, on the project that contains the bucket (which is not same project as the one containing the service account).

And I'm still getting the following error :

│ Error: Failed to get existing workspaces: querying Cloud Storage failed: Get "https://storage.googleapis.com/storage/v1/b/myuniqueidentifier/o?alt=json&delimiter=%2F&pageToken=&prefix=terraform%2Fstate%2Fsomething%2F&prettyPrint=false&projection=full&versions=false": impersonate: unable to generate access token: Post "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/***:generateAccessToken": oauth2/google: status code 403: {
│   "error": {
│     "code": 403,
│     "message": "The caller does not have permission",
│     "status": "PERMISSION_DENIED"
│   }
│ }

Any idea what I am doing wrong?

[crw]

Hi @uportalis, you may not get an answer in this thread if it is not an expansion of the original issue (which it may be!) Just in case, you can also try getting help from the community forums, there are more people there to help with these types of questions. Thanks!

[NathanielRose]

This error also occurs if your pool is disabled. Please check for this.