Skip to content

fix(bigquery): Add retry predicates for dataset create and update#26430

Draft
ElliotSwart wants to merge 1 commit intohashicorp:mainfrom
ElliotSwart:fix/bigquery-dataset-iam-retry-on-create-update
Draft

fix(bigquery): Add retry predicates for dataset create and update#26430
ElliotSwart wants to merge 1 commit intohashicorp:mainfrom
ElliotSwart:fix/bigquery-dataset-iam-retry-on-create-update

Conversation

@ElliotSwart
Copy link

Summary

  • Adds IamServiceAccountNotFound and IsBigqueryIAMQuotaError retry predicates to BigQuery dataset Create (POST) and Update (PUT) operations
  • Fixes transient 400 errors when datasets have inline access blocks referencing service accounts or groups that haven't fully propagated
  • Companion to fix(bigquery): Add retry predicates for BigQuery dataset IAM operations #26429 which adds the same predicates to BigQuery dataset IAM policy operations

Problem

When a google_bigquery_dataset with inline access blocks and a google_service_account are created in the same terraform apply, the BigQuery API can fail with:

Error creating Dataset: googleapi: Error 400: Service account sa@project.iam.gserviceaccount.com does not exist, invalid

or during updates:

Error updating Dataset: IAM setPolicy failed: googleapi: Error 400: Group bq-users@example.com does not exist, invalid

This happens because GCP IAM is eventually consistent — the service account or group exists but isn't yet visible to the BigQuery API (typically 30-120 seconds).

Currently, resourceBigQueryDatasetCreate and resourceBigQueryDatasetUpdate call SendRequest without any ErrorRetryPredicates, so these transient errors fail immediately.

Fix

Add two existing retry predicates to both Create and Update SendRequest calls:

  • IamServiceAccountNotFound: retries on 400 errors with "Service account does not exist"
  • IsBigqueryIAMQuotaError: retries on 403 errors from BigQuery IAM quota limits

Test plan

  • Run provider acceptance tests for google_bigquery_dataset
  • Create dataset with inline access referencing a new service account — should retry and succeed
  • Verify non-retryable errors (invalid location, malformed config) still fail immediately
  • Verify retry behavior with debug logging

References

🤖 Generated with Claude Code

BigQuery dataset create (POST) and update (PUT) operations currently
do not pass any ErrorRetryPredicates to SendRequest. When a dataset
has inline access blocks referencing service accounts or groups created
in the same apply, the API can return Error 400 "Service account does
not exist" or "Group does not exist" due to IAM propagation delays.

This adds IamServiceAccountNotFound and IsBigqueryIAMQuotaError retry
predicates to both Create and Update operations, matching the pattern
used in the companion fix for BigQuery dataset IAM policy operations.

Addresses 230 observed failures from IAM setPolicy errors during
dataset creation/update with inline access blocks.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@github-actions
Copy link

This repository is generated by https://github.com/GoogleCloudPlatform/magic-modules. Any changes made directly to this repository will likely be overwritten. If you have further questions, please feel free to ping your reviewer or, internal employees, reach out to one of the engineers. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant