•
Prefect·
dbt·
BigQuery·
Google Cloud Storage·
Python
ELT pipeline for loading and transforming my Strava activities data into simple Looker dashboard.Data is fetched from Strava API as JSONL files and stored in GCS bucket with BigQuery external table on...
About this project
ELT pipeline for loading and transforming my Strava activities data into simple Looker dashboard.
Data is fetched from Strava API as JSONL files and stored in GCS bucket with BigQuery external table on top.
Silver and gold layer datasets and tables - BigQuery. Silver layer contains the data with proper units of measure and data types. Gold layer is the table with metrics aggregated on a weekly level using ISO weeks.
Extraction and merging of the data is going to be done once a week, probably on Monday.
Data transformation (silver and gold layer)- DBT Core.
Data orchestration - Prefect.
Dashboard - Looker (will publish it publicly soon).
It is planned to have a Prefect server running locally on a server, once it is configured. Until then it will be manually triggered.
Stack:
Prefectdbt