Real-Time Fraud Detection & Risk Scoring Platform
A streaming platform that scores financial transactions for fraud risk in near real time, with explainable output for investigators.
Fraud is rare — roughly 1–2% of transactions — and rules-based systems age badly against patterns that keep changing. Batch scoring finds the fraud after the money has moved. I built a streaming platform that consumes transactions from Kafka, validates and enriches them in Spark Structured Streaming, engineers behavioural features on the fly, and scores them with gradient-boosted models. A decision layer sits downstream of the model, because a fraud probability is not a decision: policy, thresholds, and context decide whether a transaction is approved, rejected, or routed to an analyst.
The harder problems were not modelling ones. Severe class imbalance makes accuracy meaningless, so evaluation runs on precision, recall and PR-AUC instead. Because fraud is time-dependent, training, validation and test sets are split on time rather than at random — a random split lets future information leak backwards and flatters the model into a number that production will not reproduce. MLflow tracks experiments and versions the models, Airflow orchestrates the training and promotion workflows separately from the streaming path, and SHAP produces per-transaction attributions so an investigator can see why a transaction was flagged, not just that it was.
Because this was a professional project, implementation details here are generalized and sanitized to respect confidentiality.
- Apache Kafka
- Spark Structured Streaming
- XGBoost
- LightGBM
- MLflow
- Airflow
- Redis
- SHAP
- Docker