Pipeline Forecaster

ML forecaster for Airflow DAGs. Trains XGBoost models on synthetic run history to predict runtime (regression) and failure probability (classification). Manages champion/challenger promotion via the MLflow Model Registry. Retrains itself on a schedule. Visualizes everything in a Streamlit dashboard.

The fifth project in the portfolio and the second targeting ML Engineer roles specifically.

Project	Track	Demonstrates
lineage-oracle	AI eng	retrieval + agents
dq-test-generator	AI eng	structured generation
dbt-sentinel	AI eng	structured critique
dq-watchdog	ML eng	classical + DL anomaly detection
pipeline-forecaster	ML eng	regression + classification + MLOps lifecycle (registry, champion/challenger, scheduled retraining)

What it does

1. Generates synthetic Airflow run records with 5 planted patterns (operator effects, time-of-day, day-of-week, input scale, users-table dependency).
2. Trains two XGBoost models — XGBRegressor for runtime and XGBClassifier for failure — sharing one feature pipeline.
3. Registers trained models to MLflow at stage Staging.
4. Runs a champion/challenger gate: a new model only promotes to Production if it's strictly better than the current champion by tolerance margins.
5. Serves both behind one FastAPI /predict endpoint.
6. Retrains on a schedule via APScheduler — automatic if enough new actuals have been logged, gated by the same promotion logic.
7. Visualizes champion model card + retraining history + recent predictions in a Streamlit dashboard.

Quickstart

# 1. Install
uv sync

# 2. Generate synthetic data
uv run forecaster generate

# 3. Train (registers as Staging)
uv run forecaster train

# 4. Promote Staging → Production
uv run forecaster promote

# 5. Serve
uv run forecaster serve

# 6. Predict
curl -X POST :8000/predict -H 'content-type: application/json' \
  -d '{"dag_id":"user_etl","task_id":"load","operator_type":"PythonOperator","start_hour":3,"day_of_week":1,"upstream_count":3,"input_row_count":10000}'
# → {"expected_duration_seconds":248.3,"p_failure":0.07,"model_version":"..."}

# 7. Open the dashboard (separate terminal)
uv run forecaster ui

# 8. Start the scheduled retraining loop (separate terminal)
uv run forecaster schedule

Eval — three layers

FORECASTER_RUN_EVALS=1 uv run pytest evals/ -v -s

• Layer 1: accuracy thresholds (R² ≥ 0.60, MAPE ≤ 20% on runtime; AUC ≥ 0.80, F1 ≥ 0.65 on failure)
• Layer 2: pattern recovery — the model actually predicts the planted patterns
• Layer 3: promotion-gate correctness on three scenarios (better/worse/within-tolerance challengers)

Docker

docker build -t pipeline-forecaster .
docker run -p 8000:8000 pipeline-forecaster

Documentation

• docs/ARCHITECTURE.md — seven-component pipeline + design choices
• docs/EVALS.md — the three-layer eval story
• docs/superpowers/specs/2026-05-21-pipeline-forecaster-design.md — full spec
• docs/superpowers/plans/2026-05-22-pipeline-forecaster.md — implementation plan

Tech stack

• Tabular ML: XGBoost
• Preprocessing: scikit-learn (TargetEncoder, OneHotEncoder)
• Experiment tracking + model registry: MLflow (local SQLite)
• Serving: FastAPI + uvicorn
• Scheduler: APScheduler (in-process)
• Dashboard: Streamlit
• Container: Docker
• Tests: pytest