Building a Real-Time Fraud Detection Platform with Kafka, Spark, XGBoost and MLflow

Because this was a professional project, implementation details have been generalized and sanitized to respect confidentiality. Where I use numbers to make a point concrete, I label them as illustrative.

Fraud is a latency problem before it is a modelling problem.

By the time a suspicious transaction surfaces in a nightly batch report, the money has moved, the card has been used again, and detection has quietly become recovery. The model may have been perfectly good. It just ran too late to matter.

So I built the detection path as a stream. Transactions arrive on Kafka, Spark Structured Streaming validates and enriches them, features are computed in flight, a gradient-boosted model scores each one, and a decision engine decides what to do about it.

One calibration up front, because it shapes everything: a Kafka plus Spark micro-batch path is near-real-time, not a synchronous authorisation hop. It does not sit inside a sub-100ms card-authorisation round trip. It targets decisions that stay open for hundreds of milliseconds to a few seconds — holds, step-up verification, review routing, intervention before settlement. If a score had to be returned inline to an authorisation request, you would put a dedicated synchronous serving layer in front of this.

Architecture

Solid arrows are the real-time data path. Dashed arrows are metadata and control flow — orchestration, monitoring, model management.

Kafka is the front door. It decouples producers from consumers, absorbs bursts without backpressuring the transaction itself, partitions for parallelism, and — the property that matters most operationally — lets you replay. When a downstream bug corrupts a day of features, replay is the difference between a fix and an incident.

Spark Structured Streaming consumes, validates, transforms, and computes features. Checkpointing and watermarking handle the parts nobody enjoys: recovery after failure, and late-arriving events that would otherwise silently corrupt windowed aggregates.

Redis holds state that has to be read fast. Scoring needs both the current transaction and the account’s recent behaviour, and the recent-behaviour lookup cannot be a warehouse query. To be precise about what this is: Redis here is a low-latency store for streaming state, not a full feature-store product with lineage and point-in-time guarantees.

XGBoost and LightGBM do the scoring. Transaction data is structured and tabular with heterogeneous numeric features and strong interactions, which is exactly where gradient-boosted trees still beat deep models — with far less tuning and far better inference latency.

MLflow, Airflow, MinIO/S3, PostgreSQL, Docker and SHAP form the layer around the model: tracking and registry, orchestration, artifact and metadata storage, reproducible packaging, explanations.

Why streaming, and why not batch

Batch is not inaccurate. It is late. It operates on a completed past, after the decision window has closed.

	Batch	Streaming
When scoring happens	On a schedule, after the fact	While the transaction is in flight
Feature freshness	As of the last window	Reflects very recent activity
What detection enables	Investigation, reporting	Approve / review / reject
Velocity signals	Visible after the burst ends	Visible as the burst happens

Streaming is not free — it is heavier to run, harder to debug, and demands more operational discipline. Batch is still the right tool for retraining, reporting and offline analysis, and it still does exactly that here. The argument is narrower than “streaming is better”: batch is structurally unable to support a decision that must be made in the moment.

Streaming also changes what a feature can be. “Average transaction amount” computed nightly is a different feature from the same thing computed over the last few minutes. Fraud is overwhelmingly behavioural — a transaction is suspicious relative to what this account normally does. Freshness is not a nice-to-have. It is most of the signal.

Class imbalance: accuracy is the wrong scoreboard

Fraud rates vary by institution, channel and product. I am not publishing this project’s base rate; for the argument below, use an illustrative one to two percent.

The obvious consequence: if 99% of transactions are legitimate, a model that predicts “legitimate” for everything scores about 99% accuracy while catching exactly zero fraud. Accuracy measures the majority class and tells you nothing.

The less obvious consequence matters more in production. Because the negative class is enormous, a tiny false-positive rate still produces a huge absolute number of false positives. A rate that looks negligible in a confusion matrix becomes a review queue no team can clear and a stream of declines hitting customers who did nothing wrong.

So the real problem is not “detect fraud.” It is detect fraud at a precision the review process and the customer base can absorb.

That pushes you to precision, recall, F1 and PR-AUC. PR-AUC is the honest one under heavy imbalance: its baseline is the positive rate itself, whereas ROC-AUC’s baseline is a fixed 0.5 and can look respectable on a model that is not usable. On the training side, class weighting via scale_pos_weight is the first lever. Note that LightGBM’s is_unbalance and scale_pos_weight are alternatives, not complements — is_unbalance just sets the weight for you.

Threshold selection is then a business decision, not a modelling one: it trades recall against how many alerts a human team can actually work.

Temporal validation: the failure mode that flatters you

This is the part I would want to be asked about in an interview.

Fraud is time-dependent, and a random train/test split quietly leaks the future into the past. Shuffle transactions and the model trains on next week while being tested on last week. It learns patterns it could not have known at decision time, scores beautifully offline, and underperforms in production — the worst kind of failure, because it is invisible until it costs money.

The fix is to split on time:

Historical period  →  Training
Later period       →  Validation
Future period      →  Test
Production period  →  Inference

Every split boundary is a point in time, and every feature must be computed using only data available before that boundary. This is point-in-time correctness, and it is easy to violate by accident:

• Aggregates computed across the whole dataset. An “account average amount” calculated over all data encodes future transactions into a training row.
• Label-derived features. Anything downstream of the fraud outcome — chargeback flags, investigation status — will not exist at scoring time.
• Entity overlap. The same account appearing in train and test lets the model memorise the account rather than learn the behaviour.

One clarification worth making, because it is commonly stated loosely: the problem with KFold is not that it shuffles — it does not shuffle by default. The problem is that every fold except the last trains on data that comes after some of what it validates on. Ordinary cross-validation is structurally wrong here; you want forward-chaining splits.

Temporal validation gives a lower number than a random split. That is the point. It is the number production will reproduce.

Model prediction is not a business decision

A model output of 0.93 — illustrative — does not mean “reject.”

The model produces a probability. The decision engine produces an outcome, and it takes more than the score: policy, thresholds, customer context, transaction type, and the operational cost of being wrong in each direction. The same 0.93 can mean reject on one product and step-up verification on another.

Keeping these separate is an architectural decision, not a philosophical one. It means thresholds and policy can change without retraining, and the model can be retrained without renegotiating business rules. Collapse them and every policy tweak becomes an ML release.

The review branch has its own problem: a queue of alerts, each needing context an investigator assembles by hand. I wrote about the retrieval layer built for that in From Fraud Alert to Investigation Summary with RAG and an LLM.

Explainability

Analysts do not act on a number. They act on a reason.

SHAP attributes a prediction across its input features additively, so each scored transaction carries a per-feature breakdown of what pushed it toward “suspicious.” For an illustrative score of 0.93, contributing factors might be an unusual amount, elevated velocity, a new device, a geographic deviation — illustrative examples, not the project’s implemented feature set.

Two things worth stating precisely. TreeSHAP is exact with respect to the conditional expectation implied by the tree structure — “exact” is a claim about the computation, not about ground truth. And SHAP explains the model, not reality: it tells you what the model used, which is exactly what you want when auditing a decision or debugging drift.

Without this, a flagged transaction is an assertion. With it, it is a starting point for an investigation.

Lifecycle and operations

MLflow tracks experiments, then registers, versions and promotes models. Versioning is not bureaucracy here — when scores shift in production, the first question is always “what changed,” and a registry is what makes that answerable. Multiple model versions (V1, V2, V3) were evaluated during development; those are experiment iterations, not production performance claims.

Airflow orchestrates the batch side: data preparation, training, scheduled evaluation, promotion. It is worth being explicit that Airflow is not the streaming engine — Kafka and Spark own the real-time path, Airflow owns the workflow path. Conflating the two is a common reading of diagrams like the one above.

Docker packages inference and services so the environment that ran in development is the one that runs in production.

Monitoring spans four layers, and the model layer is the one people forget: data quality (schema changes, invalid events, delays), streaming health (consumer lag, processing latency, throughput), model behaviour (prediction distribution, fraud-rate shifts, feature and concept drift), and infrastructure. Model performance degrades even when no code changes — fraud patterns move, and a model trained on last quarter’s behaviour is a depreciating asset.

What I would build next

An online feature store with real lineage and point-in-time guarantees, rather than Redis carrying that weight. Automated drift detection feeding automated retraining. Champion/challenger evaluation so a new model proves itself on live traffic before promotion. Schema governance on the Kafka topics. Further out, graph-based detection to catch fraud rings that look unremarkable transaction by transaction.

What this project taught me

Real-time ML is mostly not about the model. The model was the fastest part to build. The time went into temporal correctness, feature freshness, the prediction/decision boundary, and the machinery that lets you change a model safely once people depend on it.

And the metric you optimise decides what you build. Chase accuracy and you get a system that catches nothing. Chase recall alone and you get a queue nobody can clear. The useful target is narrower and less satisfying: catch as much as you can at a precision the business can live with, and be able to explain every call you make.

Technology stack

Layer	Tools
Streaming	Apache Kafka, Spark Structured Streaming
Models	XGBoost, LightGBM
Lifecycle	MLflow, Apache Airflow
Storage	MinIO / S3-compatible, PostgreSQL, Redis
Explainability	SHAP
Packaging	Docker
Languages	Python, PySpark, SQL