Problem
Input distributions can change while a model's performance metrics remain above their alert thresholds. This experiment asks whether feature-level monitoring detects a deliberately injected shift that an F1-only alert would miss.
Data
A credit-card fraud model was scored against features replayed week over week. Partway through the replay a deliberate distribution shift was injected — a known, datable event — so the monitor could be evaluated on whether and when it caught the shift.
Method
Each cycle computes a Population Stability Index (PSI) and a Kolmogorov–Smirnov test on input features, with Evidently generating drift reports and MLflow logging the runs. A Streamlit dashboard shows the replay results. The repository also includes an Airflow DAG for scheduled orchestration; running it requires Linux, WSL, or Docker.
Finding
Maximum feature PSI climbed from 0.004 to 2.96 — more than a 700× increase across the replay. The first warning fired at Week 5 (PSI 0.106, warning threshold 0.10), followed by a critical alert at Week 6 (PSI 0.201, critical threshold 0.20).
F1 remained between 0.91 and 1.00 over all 12 weeks and never crossed the 0.80 alert threshold. This demonstrates a difference between feature-drift and performance alerts; it does not establish a measured lead time to performance degradation.
Recommendation
Use a PSI breach as a prompt to investigate feature changes alongside KS tests and Evidently reports. Validate alert thresholds and model performance before deciding whether retraining is warranted.
Limitations and next steps
These results come from replayed data with injected feature shifts. Feature drift alone does not prove performance degradation. Next steps are testing other shift patterns and seasonality, measuring false alerts, and validating thresholds with delayed labels before automating responses.