From Fraud Alert to Investigation Summary with RAG and an LLM
Assembling case context automatically so investigators spend their time deciding, not gathering
Because this was a professional project, implementation details have been generalized and sanitized to respect confidentiality. Where I use numbers to make a point concrete, I label them as illustrative.
In the previous post I described the scoring path: transactions stream through Kafka and Spark Structured Streaming, a gradient-boosted model produces a fraud probability, and a decision engine applies policy to approve, reject or route for review.
This post is about what happens on that third branch.
A score that clears the review threshold becomes an alert, and an alert becomes a human’s problem. Everything below the threshold approves normally and nobody ever looks at it. So the scoring model’s real output is not a decision — it is a queue. And the cost of that queue is measured in investigator time.
The problem is assembly, not judgement
An investigator opening a fresh alert starts from an ID and a number.
What follows is an hour of gathering. Pull the customer’s transaction history and look for the shape of normal. Check the device and location against past activity. Search for similar cases someone else already worked, which requires guessing the words they used to describe it. Find and re-read the policy that governs this transaction type and this product.
Then the actual judgement — is this fraud — takes a couple of minutes.
That ratio is the problem. Experienced investigators are good at the judgement and are spending most of their day on the retrieval in front of it. Worse, the retrieval is lossy: the “similar case” that would have settled the question exists, but only in someone else’s notes from eight months ago, described in words this investigator would not think to search for.
Context collection
An alert triggers two fetches in parallel.
The structured side pulls what the platform already knows and can query directly: transaction history, customer profile, the risk score and the specific fraud signals that drove it, device, IP and location, velocity and behavioural aggregates. This is the same data the scoring path used, now assembled for a human rather than a model.
The unstructured side reaches for institutional memory — the material that normally lives in people’s heads and in documents nobody can find: closed fraud cases, investigator notes, documented fraud patterns, policies, SOPs and guidelines.
The second half is where the value is, and it is the half that is hard to retrieve.
Making the corpus retrievable
Unstructured sources pass through cleaning and normalisation, PII masking, and metadata enrichment before anything is embedded.
The masking step is not housekeeping. Investigation notes are written by humans under time pressure and are dense with personal detail, and once text has been embedded and indexed it is genuinely difficult to take back — you cannot reliably un-learn a vector. Masking has to happen before embedding, not as a filter on the way out.
Metadata enrichment matters for a less obvious reason. Product, channel, case type, outcome and date are what let retrieval be constrained later. An embedding alone cannot express “only closed cases, only this product, only the last eighteen months,” and those constraints are usually the difference between a relevant result and a plausible one.
Cleaned, masked, enriched documents are then embedded with a sentence-transformer model and stored in a vector database alongside their metadata.
Retrieval: why pure vector search is not enough
Semantic search is good at this reads like that other case. It is bad at exact identifiers, policy clause numbers, scheme names and specific merchants — which is precisely what investigators search for.
Ask a vector index for a policy clause by number and it will happily return four clauses that are semantically adjacent and none that are the one you asked for. That failure is quiet, which makes it worse: the results look reasonable.
So retrieval is hybrid, and each part is there for a reason:
| Component | What it is for |
|---|---|
| Vector search | Semantic recall — cases that rhyme with this one |
| Keyword search | Exact precision on identifiers, clauses, merchant and scheme names |
| Metadata filtering | Constrain by product, channel, case type, date range |
| Re-ranking | Order the top-K by actual relevance rather than raw vector distance |
Re-ranking earns its place because nearest-neighbour distance is a weak proxy for usefulness. The closest vector is often a case that shares vocabulary rather than substance.
Generation, and what the model is allowed to do
A context builder assembles the prompt from the alert itself, the structured history, the retrieved cases and notes, and the applicable business rules.
The LLM then produces a structured summary rather than prose: what happened, why it was flagged, the key risk indicators, the supporting evidence, similar historical cases with links back to them, a recommended next action, and a confidence score. That summary is attached to the case automatically, so an investigator opens a case whose context is already gathered.
Then a human decides.
This is the same boundary the scoring system draws, one level up. In the scoring path, a probability is not a decision — policy is. Here, a generated summary is not a decision either. The investigation associate reviews the summary, checks the evidence against the cited sources, looks at the similar cases, adds their own notes, and makes the call. Approve, escalate, reject, close.
Treating the summary as the decision would be the worst mistake available in a system like this. It is a retrieval-grounded draft. It can be wrong in the specific way retrieval systems are wrong — confidently, fluently, and anchored on a “similar case” that is similar in wording but not in substance.
Two properties keep it reviewable rather than a black box:
- It cites what it used. The summary points at the cases and policies it drew on, so an analyst can check the source instead of trusting the paraphrase. A summary that cannot be traced back is not evidence.
- Outcomes feed back. Case decisions and analyst corrections return to the knowledge base, so later retrievals draw on a corpus that reflects what investigators actually concluded rather than what was written in the first draft of a note.
What I would build next
Evaluation is the honest gap. Retrieval quality is measurable — recall on a labelled set of case-to-case matches — and summary quality is harder, needing investigator ratings collected as part of the workflow rather than as a side project. Beyond that: making the confidence score a calibrated signal rather than a self-report, and detecting when retrieval returns nothing genuinely relevant so the system can say so instead of summarising four weak matches.
The honest framing
This does not make anyone a better investigator. The judgement was never the bottleneck.
It removes the assembly work in front of the judgement, and it surfaces prior cases a person would only find if they already knew to look for them. That is a smaller claim than “AI fraud investigation,” and it is the one the system actually supports.
Technology stack
| Layer | Tools |
|---|---|
| Retrieval | Sentence-transformer embeddings, vector database, hybrid search, re-ranking |
| Generation | LLM inference served behind a model-serving layer |
| Data | Data lake / lakehouse, streaming ingestion |
| Orchestration | Workflow orchestration for indexing and refresh |
| Governance | PII masking, RBAC, audit logging |