DPH

Discover the Best
Data and AI Engineering Projects

Explore curated data and AI engineering projects from the community. Be recognized for your work, vote for your favorites, and share your own creations.

Week of May 17

May 17 - 23

0votes

1.Airflow Local Debug

Local execution and debugging toolkit for Apache Airflow DAGs that runs entirely in a single Python

Looking for contributorsLooking for feedback+1Actively maintained
Data PlatformETL/ELT PipelineData Quality & Testing
Apache AirflowPython
by Alexey Potemkin

Week of May 10

May 10 - 16

2votes

2.Playroom Diet

Multi-agent AI meets child development. Just like a diet needs balance, so does play.

Looking for feedbackPrototype+2Actively maintained
AI AgentsML/AI PipelineAI Application
Apache AirflowGoogle GeminiPostgreSQLPython+1
by Volker Janz

Week of May 3

May 3 - 9

1votes

3.AQI-DATA-PIPELINE

End-to-end AQI data engineering pipeline using Apache Airflow, dbt, Snowflake, and AWS S3 with Bronz

Looking for contributorsLooking for feedback+3Actively maintained
ETL/ELT PipelineAnalytics & BI
AWS S3Apache AirflowSnowflakedbt
by Kirtan Soni

4.Spotify Azure Data Pipeline

End-to-end batch pipeline: Azure SQL to Delta Lake Gold with SCD Type 2

Looking for feedbackOpen source
Batch ProcessingETL/ELT PipelineData Platform
AzureDatabricksAzure Blob StorageDelta Lake+1
by vikneshwararb

Week of Apr 26

Apr 26 - May 2

2votes

5.football-lakehouse

football analytics lakehouse using Apache Spark, Apache Iceberg, Project Nessie, MinIO, and Dremio.

Data PlatformBatch ProcessingETL/ELT Pipeline
Apache AirflowApache SparkApache IcebergMinIO
by كريم جلال

Week of Apr 19

Apr 19 - 25

1votes

6.VSFlow

YAML-driven task automation for VS Code

Experiments
AWS
by Rahul Rajasekharan

7.Spotify API ELT

End-to-end ELT pipeline for Spotify data with orchestration, testing & CI/CD

ETL/ELT Pipeline
Apache AirflowSodaPythonPostgreSQL
by qlbr913

Week of Apr 5

Apr 5 - 11

0votes

8.Minimalist Data Stack - Nasa data viz

Production-ready data visualization for NASA dataset

Analytics & BIETL/ELT PipelineData Platform
Apache AirflowSnowflakedbtGrafana+1
by thoutmose

Week of Mar 29

Mar 29 - Apr 4

1votes

9.airflow-ai-ops

A self-healing data pipeline platform built on Airflow

Data PlatformExperiments
Apache AirflowPostgreSQLSQLPython
by Rahul Rajasekharan

10.Basalt

A synthetic data generator that guarantees deterministic output

Data Platform
Python
by phanindra562

Week of Mar 22

Mar 22 - 28

0votes

11.Asphalt

Real-Time Vehicle Data Processing Pipeline

Real-time StreamingETL/ELT Pipeline
Apache FlinkApache KafkaJavaPython
by markov-ngz

Week of Mar 15

Mar 15 - 21

2votes

12.Instacart-Analytics-PIpeline

Production-grade analytics pipeline

ETL/ELT PipelineAnalytics & BIData Platform
Apache AirflowdbtPostgreSQLSnowflake+1
by otienoduncan99

13.Airflow Schedule Insights

Predict, simulate, and debug Airflow schedules before they fail.

ETL/ELT PipelineBatch ProcessingData Quality & Testing
Apache AirflowPostgreSQLSQLPython
by egor

Week of Mar 8

Mar 8 - 14

5votes

14.airflow-watcher

An Airflow UI plugin for monitoring DAG failures and SLA misses/delays.

Data Quality & TestingData PlatformExperiments
Apache AirflowPython
by ramsolaimalai

15.WikiStream Event Data Analytics Pipeline in AWS

Content monitoring analytics service using latest AWS S3 Tables along with MSK, EMR (SLA=20 mins)

ETL/ELT PipelineData Quality & TestingReal-time Streaming
Apache SparkAWSdeequSQL+1
by mrshihabullah

Week of Mar 1

Mar 1 - 7

0votes

16.airflow-schedule-heatmap

Visualize how your Airflow DAG schedules are distributed across the day with an interactive heatmap

ExperimentsBatch ProcessingData Platform
Apache AirflowPython
by Rahul Rajasekharan

17.PySemantic

A lightweight, graph-based Semantic Layer. Define metrics once, Generate SQL everywhere.

Analytics & BIData PlatformData Quality & Testing
Python
by nitesh8j

Week of Feb 22

Feb 22 - 28

4votes

18.Flowrs - a TUI for Airflow

Monitor, inspect and trigger Airflow DAGs from the comforts of your terminal

ExperimentsData Platform
Apache Airflow
by jan.vanbuel

Week of Dec 28, 2025

Dec 28 - Jan 3

4votes

19.UFC Data Warehouse

UFC Data Warehouse and dashboards based on up-to-date data

Analytics & BIETL/ELT Pipeline
dbt
by reshefsharvit21

20.Drift Detective

Drift Detective is a Python library for tracking schema evolution using versioned JSON snapshots

Data PlatformData Quality & Testing
Python
by varga.dani6

Week of Dec 21, 2025

Dec 21 - 27

4votes

21.Silism Commerce 360 - Local AI Native Lakehouse

AI-native e-commerce data platform you can run locally (Airflow + dbt + MCP)

Data PlatformETL/ELT PipelineAnalytics & BI
Apache KafkaApache AirflowdbtDuckDB+1
by Silism

22.Real-Time-Sales-Streaming-Pipeline

Modern Lakehouse Architecture with Kafka + Spark Structured Streaming + Delta Lake

Batch ProcessingReal-time Streaming
Apache KafkaApache Spark
by imen.bnamar

23.AIRFLow Medical Data Pipeline

Enterprise-grade ETL pipeline transforming medical XML data into actionable business intelligence

Analytics & BIETL/ELT Pipeline
Apache AirflowPostgreSQLPandasPython+1
by imen.bnamar

24.Bluesky NBA Real-Time Sentiment Analysis

A real-time data streaming pipeline that captures live posts from Bluesky regarding the NBA, perform

Batch ProcessingETL/ELT PipelineReal-time Streaming
Apache KafkaApache SparkElasticsearchGrafana
by imen.bnamar

Week of Dec 14, 2025

Dec 14 - 20

7votes

25.Airflow Python React Widgets

Python React Experiment

ExperimentsData Platform
Apache AirflowPython
by Rahul Rajasekharan

26.Yelp Batch ETL Pipeline

A batch ETL pipeline that processes Yelp business raw data to generate analytics and insights

ETL/ELT PipelineBatch ProcessingAnalytics & BI
Apache AirflowApache SparkScalaMongoDB+1
by darracq.aurelien

27.Airflow and DBT Analytics Accelerator

An Open-source accelerator for a ready-to-run, end-to-end analytics platform

Data PlatformETL/ELT PipelineAnalytics & BI
Apache AirflowdbtClickHouseSQL+1
by gsaipurushoth7

28.Cricket Analytics Data Pipeline

CAP is an end-to-end cricket analytics platform built on Cricsheet ball-by-ball data

Analytics & BIML/AI PipelineExperiments
SnowflakeApache AirflowPythonSQL+1
by Rahul Rajasekharan

Week of Dec 7, 2025

Dec 7 - 13

2votes

29.Yelp Medallion Batch Pipeline

Building medallion architecture for crowd-sourced reviews using Snowflake native features

ETL/ELT PipelineBatch Processing
SnowflakeSQL
by vikneshwararb

Week of Nov 23, 2025

Nov 23 - 29

1votes

30.Smart Wardrobe Suggestion

LLM Based Smart Clothing Suggestion

Batch ProcessingETL/ELT PipelineML/AI Pipeline
Apache Airflow
by Rahul Rajasekharan