Fraud models decay silently. A classifier that scored a strong F1 in validation can keep reporting healthy accuracy for weeks after the behaviour underneath it has shifted — because the labels that let you measure performance arrive late. Performance is a lagging indicator. By the time the F1 score visibly drops, the losses are already booked. The question this project answers: can we detect that decay before it ever reaches a performance metric?
A production-style credit-card fraud model was scored against a streaming feature set, replayed week over week. Partway through the replay a deliberate distribution shift was injected — a known, datable event — so the monitor could be held to an objective test: did it catch the shift, and how early?
Each cycle computes a Population Stability Index (PSI) and a Kolmogorov–Smirnov test on every input feature, with Evidently generating the drift reports and MLflow logging every run so the history is queryable. An Airflow DAG orchestrates the loop on a schedule, and a Streamlit dashboard renders the drift timeline so a reviewer can read the model's health at a glance — no notebook required.
PSI on the key feature climbed from 0.004 to 2.96 — roughly an 800× jump, far past the 0.2 line that conventionally marks significant drift — and the monitor flagged it at Week 5.
Over that same window the model's F1 never moved: it held between 0.91 and 1.00. The drift signal led the performance signal by about two weeks — early warning while accuracy still looks fine. That gap is the entire value of the system.
Promote a PSI breach above 0.2 to an automated retrain trigger and an on-call page, rather than waiting on the weekly performance review. For a fraud model, two weeks of lead time is the difference between a controlled retrain and a quarter of mislabelled transactions. Keep the KS tests and Evidently reports as the diagnostic layer an engineer opens after the alert.