Imbalanced Classification Benchmark (Fraud Detection)

The Problem

Fraud detection, rare disease diagnosis, equipment failure prediction. Every one of these problems shares the same core challenge: the event that matters is rare, often under 1% of the data, and standard evaluation quietly fails without anyone noticing.

This project does not ship a single fraud model. It ships a benchmark: nine techniques for handling extreme class imbalance, tested on the same data, the same split, and the same metrics, using a real credit card fraud dataset where fraud represents 0.173% of transactions, roughly 1 in every 578.

What the Benchmark Found

The most expensive technique tested, SMOTE combined with Edited Nearest Neighbors, took about 15 minutes to run and generated hundreds of thousands of synthetic rows. It finished sixth out of nine techniques.

The winning technique, a moderated class weight adjustment, changes nothing about the training data and adds a single parameter to the model. It trains in the same time as a model with no imbalance treatment at all, and still beat every resampling method tested.

Key Terms

PR-AUC

Area under the Precision-Recall curve. The primary metric for this benchmark, since it stays meaningful under severe imbalance, unlike Accuracy or ROC-AUC.

Class weight

An algorithm level adjustment that penalizes errors on the minority class more heavily, without touching the training data.

SMOTE

Synthetic Minority Oversampling Technique — generates synthetic examples of the rare class by interpolating between existing ones.

Threshold tuning

Adjusting the probability cutoff used to convert a model's output into a decision, instead of changing the data or the algorithm.

Pipeline Overview

Imbalanced Classification Benchmark Architecture

Fig. 1 · EDA → baseline → resampling, algorithm-level, anomaly detection and threshold tuning tested in parallel → benchmark → explainability on the winning technique.

Key Decisions

PR-AUC over Accuracy and ROC-AUC

A model predicting the majority class 100% of the time scores above 99.8% accuracy on this dataset. ROC-AUC also looks deceptively strong under extreme imbalance. PR-AUC is far more sensitive to how the model performs on the rare class, which is the one that matters.

Test set is never resampled

Every resampling technique is applied only to the training data. The test set always reflects the real world distribution, so every technique is compared on equal, realistic footing.

Same base model across every technique

XGBoost with fixed hyperparameters is used for every technique tested. This isolates the effect of the imbalance treatment itself, rather than mixing in the effect of different model choices.

Threshold tuning tested as a first resort

It reuses the already trained baseline model and searches for a better decision boundary. It costs nothing extra to compute and is often overlooked in favor of more complex resampling.

Isolation Forest as a contrast, not a candidate

It represents the alternative of ignoring the label entirely. Its weak performance makes a concrete point: when labeled examples of the rare class exist, even a small number of them, using that signal outperforms ignoring it.

Repository

fraud-001-imbalanced-classification-benchmark github.com →

fraud-001-imbalanced-classification-benchmark/
├── README.md
├── requirements.txt
├── .gitignore
├── config/
│ └── imbalance_config.py
├── src/
│ ├── __init__.py
│ ├── eda.py
│ ├── target_analysis.py
│ ├── baseline.py
│ ├── resampling_techniques.py
│ ├── algorithm_techniques.py
│ ├── anomaly_detection.py
│ ├── threshold_tuning.py
│ ├── benchmark.py
│ └── explainability.py
├── data/
│ └── creditcard.csv
├── models/
│ └── baseline_model.pkl
├── outputs/
│ ├── benchmark_final.parquet
│ ├── shap_summary.png
│ └── confusion_matrix.png
└── notebooks/
└── quickstart.ipynb

Stack: XGBoost, imbalanced-learn, scikit-learn, SHAP. Python 3.13. Uses the Credit Card Fraud Detection dataset (ULB), publicly available on Kaggle. The dataset exceeds GitHub's file size limit and is downloaded separately, with instructions in the README.