In this project, we will be building a data pipeline using the GCP Dataflow service. We will be creating a Flex template in which you can pass API endpoint response, extract keys in the response, and then export it to BigQuery.
Repository with Full Code: https://github.com/amandeepsaluja/gcp-dataflow-pipeline/tree/master/01-flex-template
Steps in the Project:
- Create the Pipeline in Python
- Create the Docker Image
- Setup Terraform configuration files
- Build & deploy Docker Image of the Flex template on Artifact Registry (Terraform + GitHub Actions)
- Create and Deploy the Dataflow job (GitHub Actions)