df Demand Forecasting
MLOps reference implementation

Forecasts that stay honest after deployment.

A demand forecasting platform for NYC taxi data. Contracts validate the input, a gate decides what reaches production, the API reports on itself, and a monitor files a ticket when the model starts to slip.

Python 3.12 LightGBM FastAPI GitHub Actions 66 tests mypy strict
WAPE, holdout
21.8%
14 days, 176,736 rows
Interval coverage
79.4%
p10 to p90, nominal 80%
p99 serve latency
4.3 ms
one client, 48 slot forecast
Throughput
570 req/s
4 concurrent clients, one process
Infrastructure cost
$0
free tiers only
01

The system

Five stages, one repository, no always-on infrastructure. Every arrow below runs as a command locally and as a workflow in CI.

NYC TLC monthly parquet + SHA-256 manifest Panel contract dense zone x slot no nulls, no gaps forecast panel Training time split, lags p50, p10, p90 forecast train Registry gate champion vs challenger 2% WAPE margin production alias Forecast API FastAPI, Prometheus request IDs, JSON logs Monitoring WAPE, bias, PSI · opens an issue on a breach forecast monitor

A breach re-enters the pipeline as a retraining trigger, so monitoring is part of the loop rather than a report at the end of it.

02

Results

Held out window, 17 to 30 September 2026. One run per configuration, no tuning on the test window. Errors below the 5% noise floor are not claimed.

MetricValueReading
WAPE0.218total absolute error over total demand
MAE2.32trips per 30 minute slot
RMSE6.92heavy tail from high volume zones
Bias−2.5%inside the ±10% gate
Coverage p10–p900.794nominal 0.80, band is calibrated
Max PSI0.017no drift against the training reference

WAPE by hour runs from 17.6% at 18:00 to 44.4% at 04:00, where a handful of trips moves the ratio. The dashboard carries the profile.

03

What makes it production

01
Promotion is a gate

Every retrain is registered, but only a challenger that beats the champion by 2% WAPE takes the production alias. The alias decides what serves, not the code.

02
Monitoring has teeth

A breach fails the workflow and opens a GitHub issue with the metric, the window and the runbook section. Regressions become tickets, not dashboards.

03
Contracts before models

The panel is validated for density, nulls, duplicates and zone count before training. A silently missing slot would corrupt every lag above it.

04
Observability by default

Request ID, model version and latency on every response and in every log line. Prometheus counters, a latency histogram and a batch size histogram.

05
Rollback is one line

Aliases are data. Point production at a previous version, redeploy, confirm with the version header. No rebuild, no migration.

06
One feature path

Training and serving reshape the same dense panel into the same cube, so a served lag cannot drift from a trained one.

04

API and operation

# forecast one zone, one day
curl "localhost:8000/v1/zones/161/forecast?slots=48"

# batch, all zones, no band
curl -X POST localhost:8000/v1/forecast/batch \
  -H 'content-type: application/json' \
  -d '{"slots":48,"quantiles":false}'

# what is deployed and what it scored
curl localhost:8000/v1/model

# operational surface
curl localhost:8000/healthz localhost:8000/readyz
curl localhost:8000/metrics | grep forecast_
Health/healthz
Readiness/readyz
Model metadata/v1/model
Single forecast/v1/zones/{id}/forecast
Batch (50 zones)/v1/forecast/batch
Metrics/metrics
Retraintrain.yml, monthly
Monitormonitor.yml, daily
Rollbackalias + redeploy