Problem
Fraud is a needle-in-a-haystack problem — here, roughly 1 in 600 transactions. Accuracy is a trap: a model that approves everything is 99.8% accurate and catches zero fraud. The real questions are how well the model ranks fraud above legitimate activity, and where to set the cutoff — because every threshold trades missed fraud against false alarms, and both cost money.
Data
A public dataset of about 284,000 real credit-card transactions with anonymised features, of which only 0.17% are fraudulent — a severe class imbalance that makes naive accuracy meaningless.
Method
Five combinations of model and resampling were benchmarked — logistic regression and tree ensembles, with and without SMOTE oversampling — and compared on precision-recall AUC rather than ROC, the right metric when the positive class is rare. SHAP values explain which features drive each flag, and a cost model assigns dollar values to false negatives and false positives to locate the economically optimal threshold.
Finding
XGBoost with SMOTE led at a PR-AUC of 0.859. But the sharper result is economic: instead of defaulting to a 0.5 cutoff, optimising the threshold against the cost function lands at 0.49 and a projected $2,615 per day in combined fraud-and-review cost.
This is a cost-model projection to evaluate against realistic review costs and transaction volumes before using it for planning.
Recommendation
Evaluate XGBoost + SMOTE and the selected threshold on later data under realistic cost assumptions before deployment. Revisit the threshold as fraud patterns and volume change, and assess whether SHAP explanations help analysts review flagged transactions.
Limitations and next steps
The reported daily cost is a projection under the project's cost and volume assumptions, not measured savings. Anonymised features also limit business interpretation. Next steps are time-based validation, sensitivity analysis for review and fraud costs, and threshold evaluation on new transaction data.