Macro Agents Economic Data Platform

A production-grade data platform that orchestrates the full lifecycle of economic and financial market data: from API ingestion through AI-powered analysis. Built on Dagster, dbt, and DSPy, this project demonstrates how to architect a composable, testable pipeline that transforms raw economic indicators into actionable investment insights.

The Stack:

• Orchestration: Dagster manages 70+ partitioned assets with weekly schedules, sensors, and eager automation for downstream dbt models

• Transformation: dbt handles staging → marts → analysis layers with SQL-based modeling against DuckDB/MotherDuck

• AI Agents: DSPy-powered agents perform economic cycle detection, asset allocation recommendations, and strategy backtesting

• Compute: Polars for high-performance dataframe operations; DuckDB for embedded OLAP with MotherDuck cloud sync

Data Sources: Ingests from FRED (GDP, inflation, employment, housing, financial conditions), BLS, Census Bureau, Market Stack (indices, sectors, commodities, currencies), Treasury yields.

Architecture Highlights:

• Clean separation between ingestion, transformation, and analysis layers

• Partitioned assets by series code, ticker, and time period for incremental processing

• Resource-based dependency injection for API clients and database connections

• Comprehensive test coverage including asset/model description validation

AI Analysis Pipeline: The DSPy agents operate on transformed data to identify economic phases (expansion, peak, contraction, trough), generate portfolio allocation strategies, and backtest recommendations against historical returns. Model evaluation metrics drive continuous improvement.