Production-grade batch ETL platform built entirely on GCP, provisioned with Terraform as Infrastructure as Code (IaC). Demonstrates enterprise patterns including credential management via Secrets Manager and KMS for key rotation, end-to-end CI/CD with Cloud Build, containerised deployment on Cloud Run, and scheduled execution via Cloud Scheduler. Data is persisted to PostgreSQL via Cloud SQL.
| Tool | Usage |
|---|---|
| Cloud Provider |  |
| Platform Building |  |
| Pipeline Construction |  |
- Service: Google Cloud Run (HTTP-based)
- Trigger: Cloud Scheduler (Cron)
- Region:
us-central1(recommended) - Networking: Public Ingress (IAM-authenticated)
- Terraform >= 1.0.0
- Google Cloud SDK (gcloud)
- Permissions (for your personal account or SA):
roles/run.admin,roles/cloudscheduler.admin,roles/iam.serviceAccountUser - Via web console, create a new Project and attach it to a Billing Account.
- Create a Bucket to be used as a backed bucket to store the Terraform status files.
- Replace this bucket name in the file /terraform/envs/dev/backend.tf
- Go to the Cloud Build Triggers page.
- Click Manage Repositories -> Connect Repository.
- Select GitHub (Cloud Build GitHub App).
- Follow the prompts to authorise your account and select your ducks-pipeline repository.
- Create the repository in the same region you have created the project (
us-central1).
- Choose the account you want to use for this configuration.
gcloud init
- Pick the cloud project to use
- Log in to the cloud by following the link provided after executing the command below
gcloud auth login
- Select your Google account
- Allow Google Cloud to access your account by clicking Allow
gcloud auth configure-docker <Project_region>-docker.pkg.dev
The deployment process involves the steps below:
Note
A circular reference will be created between Cloud Run Service and Artefact Registry. The first, require an image that has not been created yet, and the second requires a Service to crete an image. To avoid this circular reference, execute step 0. Set Up only the first time or after executing terraform destroy
- Set Up Uncomment lines below in the terraform/modules/services/main.tf file. This will create the service using a basic, ready to use google image.
#image = "us-docker.pkg.dev/cloudrun/container/hello" <-- Uncomment this line
image = var.image_path <-- Comment this line
- Cloud login and Authorization Execute the command below to login the Cloud provider
gcloud auth login
- Set the default project
gcloud config set project <YOUR_PROJECT_ID>
- Initialize Terraform:
terraform init
- Plan the infrastructure
terraform plan -out=tfplan
- Apply changes
apply -var="project_id=<YOUR_PROJECT_ID>"
To test the ETL logic locally without deploying to Cloud Run:
- Build the Docker image:
docker build -t ducks-etl .
- Run the container:
docker run -p 8080:8080 ducks-etl
- Trigger the process:
curl localhost:8080
- The job is configured to run automatically via Cloud Scheduler.
Target URI: https://duck-pipeline-service-dev-248136157540.us-central1.run.app
- Manual Trigger. To manually invoke the production pipeline with authentication:
gcloud scheduler jobs run daily-ducks-etl-job-dev --location=us-central1
Possible improvements to this project:
- Add monitoring and alerting
- Add functionality to promote to other environments (TST, PRE, PRD) 🚧