Macro Agents Economic Data Platform
From FRED to Forecasts: A Modern Data Stack for Economic Intelligence
5.0
1 reviews•
Dagster·
dbt·
DuckDB·
Polars
A production-grade data platform that orchestrates the full lifecycle of economic and financial market data: from API ingestion through AI-powered analysis. Built on Dagster, dbt, and DSPy, this proje...
About this project
A production-grade data platform that orchestrates the full lifecycle of economic and financial market data: from API ingestion through AI-powered analysis. Built on Dagster, dbt, and DSPy, this project demonstrates how to architect a composable, testable pipeline that transforms raw economic indicators into actionable investment insights.
The Stack:
Orchestration: Dagster manages 70+ partitioned assets with weekly schedules, sensors, and eager automation for downstream dbt models
Transformation: dbt handles staging → marts → analysis layers with SQL-based modeling against DuckDB/MotherDuck
AI Agents: DSPy-powered agents perform economic cycle detection, asset allocation recommendations, and strategy backtesting
Compute: Polars for high-performance dataframe operations; DuckDB for embedded OLAP with MotherDuck cloud sync
Data Sources: Ingests from FRED (GDP, inflation, employment, housing, financial conditions), BLS, Census Bureau, Market Stack (indices, sectors, commodities, currencies), Treasury yields.
Architecture Highlights:
Clean separation between ingestion, transformation, and analysis layers
Partitioned assets by series code, ticker, and time period for incremental processing
Resource-based dependency injection for API clients and database connections
Comprehensive test coverage including asset/model description validation
AI Analysis Pipeline: The DSPy agents operate on transformed data to identify economic phases (expansion, peak, contraction, trough), generate portfolio allocation strategies, and backtest recommendations against historical returns. Model evaluation metrics drive continuous improvement.