Fraud is a needle-in-a-haystack problem — here, roughly 1 in 600 transactions. Accuracy is a trap: a model that approves everything is 99.8% accurate and catches zero fraud. The real questions are how well the model ranks fraud above legitimate activity, and where to set the cutoff — because every threshold trades missed fraud against false alarms, and both cost money.
A public dataset of about 284,000 real credit-card transactions with anonymised features, of which only 0.17% are fraudulent — a severe class imbalance that mirrors production reality and makes naive accuracy meaningless.
Five combinations of model and resampling were benchmarked — logistic regression and tree ensembles, with and without SMOTE oversampling — and compared on precision-recall AUC rather than ROC, the right metric when the positive class is rare. SHAP values explain which features drive each flag, and a cost model assigns dollar values to false negatives and false positives to locate the economically optimal threshold.
XGBoost with SMOTE led at a PR-AUC of 0.859. But the sharper result is economic: instead of defaulting to a 0.5 cutoff, optimising the threshold against the cost function lands at 0.49 and a projected $2,615 per day in combined fraud-and-review cost.
That's a number a manager can budget against — derived from the business cost, not assumed from a default.
Ship XGBoost + SMOTE at the cost-optimal threshold, and recompute that threshold on a schedule as fraud patterns and volume drift — the optimal cutoff is not static. Keep the SHAP explanations in the review queue so analysts see why each transaction was flagged, which speeds review and builds the audit trail a regulator will ask for.