Cloud · OrchestrationComing SoonData Engineer
End-to-End Cloud Data Pipeline
API ingest → PySpark transform → BigQuery load → Airflow orchestration. Full DE stack on GCP.
PySparkBigQueryAirflowGCPPython
GCP
Cloud platform
Airflow
Orchestration
BigQuery
Data warehouse
01
Problem
Demonstrate end-to-end cloud pipeline engineering using production-grade tools: orchestration, distributed transformation, and cloud data warehousing.
02
Approach
Ingest a public API, transform with PySpark on Dataproc, load to BigQuery, orchestrate with Airflow on Cloud Composer. Full medallion architecture with data quality checks at each layer.
03
Outcome
This project is in progress — check back soon for results.
Data Pipeline
API IngestPySpark TransformQuality CheckBigQuery LoadDashboard
Interested in this kind of work? Let's talk.
Get in touch