Round nine: “stop the fraud”
Every previous round dealt with systems that fail. This one deals with people who are trying to make it fail, and who adapt when you stop them. That changes the engineering: a static rule decays the moment it becomes known, a model trained on last month's fraud misses this month's, and the cost of a false positive is a real customer locked out of their own money.
The pressure: adversaries, not just load
“Someone is draining accounts through a mule network, and someone else is testing stolen cards against your authorisation endpoint. Stop them, without freezing legitimate customers on salary day.”
The structuring insight: fraud is not one problem. Each pattern has a different detection signal, a different acceptable latency, and a different cost of being wrong.
- Synchronous scoring adds under 150 ms at p99, fitting inside the card budget from Part 8.
- False positive rate under 0.1% on the block action. At 10M transactions a day, 0.1% is 10,000 wrongly blocked customers daily.
- Detect and contain a draining attack within 60 seconds.
- The scoring path fails open, because a fraud outage must not become a payments outage.
- Every decision is explainable, for the customer and for the regulator.
Rules engine on the synchronous path
Rules are unfashionable and they are where every real system starts, for three reasons worth being able to state.
- Explainable. "Blocked because a new beneficiary received more than ₦500,000 within an hour of a device change" is a sentence a customer, an agent and a regulator can all act on.
- Instantly deployable. A new attack pattern can be blocked in minutes, where retraining a model takes days.
- Cheap and predictable. Microseconds, with no inference infrastructure and no tail latency.
// rules are DATA, versioned and effective-dated, exactly like the
// compliance policies in Part 7. same reasoning: they change on an
// attacker's timetable, not a release schedule.
interface Rule {
id: string;
version: number;
when: Condition; // a tree over features, no arbitrary code
then: 'allow' | 'review' | 'block' | 'step_up';
weight: number; // contribution when combined with a model score
// shadow mode: evaluate and record, but do not act. every new rule
// runs here first so its true false-positive rate is measured on
// real traffic BEFORE it can decline anybody.
mode: 'shadow' | 'active';
effectiveFrom: Date;
effectiveTo?: Date;
}The rules that earn their place on the hot path
| Rule | Catches | Action |
|---|---|---|
| New beneficiary plus amount above a threshold | Account takeover, APP fraud | Step up authentication, rather than block |
| Device change within 24 h of a large outbound transfer | Account takeover | Step up |
| More than N declined authorisations per card in 10 minutes | Card testing | Block the card, narrow and safe |
| Amount just below a reporting threshold, repeatedly | Structuring | Review. Never auto-block, since it is often legitimate |
| Beneficiary on an internal mule list | Mule networks | Block |
| Transfer to a newly created account from many sources | Mule collection | Review |
Note how few of these block. Step-up authentication is usually the right action: it stops the fraudster, who cannot pass it, and mildly inconveniences the genuine customer, who can. Reaching for block first is the most common design error in this space.
Feature stores and the freshness problem
A rule or a model needs features, and features have wildly different freshness requirements and computation costs. That mismatch is the core engineering problem of a fraud platform.
| Feature | Freshness needed | Where it lives |
|---|---|---|
| Transactions in the last 60 seconds | Sub-second | Redis counters, incremented on the write path |
| Is this beneficiary new to this customer? | Sub-second | Redis set, checked and updated inline |
| Average transaction size over 90 days | Daily is fine | Batch-computed, loaded into the online store |
| Device reputation score | Minutes | Streaming aggregation from the event log |
| Graph distance to a known mule | Hours | Batch graph job, materialised per account |
| Account age, KYC tier | Static | Read from the account record |
The training and serving skew problem
a model trained on "average transaction size over 90 days"
computed by a batch job over the warehouse…
…is served in production by a different query, written by a
different engineer, against a different store, at a different time.
if the two definitions differ even slightly, the model sees
features it was never trained on and its accuracy silently collapses.
this is the single most common cause of an ML system that
performs well offline and badly in production.- One definition of each feature, used for both training and serving. This is the whole point; the rest is plumbing.
- Point-in-time correctness for training: the feature values as they were when the decision was made, never as they are now.
- An online store for low-latency serving and an offline store for training, populated from the same computation.
- Versioning, so a model is pinned to the feature versions it was trained against.
Behavioural baselines per account
Population-level thresholds cannot work in a bank. A ₦2,000,000 transfer is alarming for a student and routine for a business, so "normal" has to be defined per account.
// a compact per-account baseline, cheap to store for 20M accounts
interface Baseline {
accountId: string;
// distribution, not just a mean: the percentiles are what matter
amountP50: bigint; amountP95: bigint; amountP99: bigint;
txPerDayP95: number;
// the shape of their week, learned rather than assumed
activeHours: number[]; // 24 buckets of relative frequency
activeDays: number[]; // 7 buckets
knownBeneficiaries: number;
knownDevices: number;
typicalChannels: string[]; // app | ussd | pos | card | api
// a baseline built on 3 transactions is not a baseline. say so.
sampleSize: number;
confidence: 'low' | 'medium' | 'high';
computedAt: Date;
} deviation score, per dimension:
amount_z = (amount − p50) ÷ (p95 − p50)
hour_unusual = 1 − activeHours[hour_of_day]
new_beneficiary = beneficiary ∉ known ? 1 : 0
new_device = device ∉ known ? 1 : 0
combined, weighted, and scaled by confidence
a low-confidence baseline must not drive a block- Cold start. A new account has no baseline, so it cannot be scored against one. New accounts get tier-based limits instead, until enough history exists.
- Drift. People legitimately change: a new job, a new city, a business that grows. The baseline must decay old data and re-fit continuously, or every genuine life change becomes a fraud alert.
- Poisoning. A patient fraudster establishes a "normal" of increasing transfers over weeks, so the eventual drain looks in-distribution. Mitigated by absolute caps that no baseline can override, and by weighting recent behaviour less on young accounts.
Velocity checks in a sliding window
The cheapest high-value signal in fraud, and the implementation detail matters because the obvious approach has an exploitable hole.
Fixed window, and the boundary attack
fixed hourly window, limit 5 transactions:
09:59 → 5 transactions allowed
10:00 → counter resets
10:01 → 5 more allowed
10 transactions in 2 minutes, and the limit was never breached.
every attacker knows where your window boundary is.Sliding window with a sorted set
-- one atomic Lua script: trim the window, count, then add.
-- doing this as separate calls is the Part 2 lost update, again.
local key, now, window, limit = KEYS[1], ARGV[1], ARGV[2], ARGV[3]
-- drop everything older than the window
redis.call('ZREMRANGEBYSCORE', key, 0, now - window)
local count = redis.call('ZCARD', key)
if count >= tonumber(limit) then
return {0, count} -- denied
end
-- member must be unique per event, or two events at the same
-- millisecond collapse into one and the count under-reports.
redis.call('ZADD', key, now, ARGV[4])
redis.call('EXPIRE', key, math.ceil(window / 1000) + 60)
return {1, count + 1}| Approach | Memory per key | Accuracy | Use for |
|---|---|---|---|
| Fixed window counter | One integer | Exploitable at the boundary | Nothing security-relevant |
| Sliding window log | One entry per event | Exact | Low-volume, high-value checks. Our default |
| Sliding window counter | A few integers | Approximate, weighted across buckets | High-volume checks where exactness is not required |
| Token bucket | Two values | Exact for rate, permits bursts | API rate limiting rather than fraud velocity |
The dimensions worth counting
vel:acct:{id}transactions per account. Draining.vel:card:{token}authorisations per card. Card testing.vel:dev:{deviceId}across accounts. One device operating many accounts is the strongest mule signal available.vel:ben:{beneficiary}inbound sources. Mule collection points.vel:ip:{subnet}for onboarding and login. Automated account creation.vel:bin:{bin}per card BIN. A breach at another issuer being tested against us.
The third one deserves emphasis. Transaction-level signals see each mule transfer as normal; a single device fingerprint appearing across forty accounts is unambiguous, and it is a counter rather than a model.
Graph signals: money mules and rings
Some fraud is invisible at the transaction level by construction. Mule networks fragment money precisely so that no individual movement looks unusual, which means the signal lives in the structure rather than in any row.
| Graph signal | What it indicates | Cost to compute |
|---|---|---|
| Fan-out then fan-in | Classic layering: one source splits, then reconverges | Moderate. A bounded traversal |
| Shared device across accounts | Strongest single signal. One operator, many identities | Cheap. A Redis set |
| Shared beneficiary details | Same phone, address or next-of-kin across unrelated accounts | Cheap. An index lookup |
| Short path to a confirmed mule | Guilt by proximity, useful within 2 hops | Moderate, if materialised per account |
| Velocity of new edges | An account suddenly transacting with many new counterparties | Cheap. A counter |
| Community detection | Dense clusters that transact mostly internally | Expensive. Batch only |
The architecture: cheap signals online, expensive ones batch
// ONLINE, on the synchronous path, single-digit milliseconds.
// these are precomputed or counter-based, never traversals.
interface OnlineGraphFeatures {
beneficiaryIsFlaggedMule: boolean; // set membership
deviceAccountCount: number; // counter
hopsToNearestConfirmedMule: number; // materialised nightly
beneficiaryInboundSources24h: number; // counter
}
// BATCH, hours, over the whole graph. results are materialised back
// into the online store so the fast path never traverses anything.
// - connected components over the transfer graph
// - community detection, e.g. Louvain
// - PageRank-style centrality to find collection points
// - path distances to every confirmed muleScoring: rules, gradient boosting, and LLM triage
Three mechanisms with genuinely different strengths. The design is to use each where it wins rather than to pick one.
| Rules | Gradient boosted trees | LLM | |
|---|---|---|---|
| Latency | Microseconds | 1 to 10 ms | Seconds |
| On the sync path? | Yes | Yes | No |
| Explainability | Perfect | Good, via SHAP values | Plausible narrative, not a true explanation |
| Handles novel patterns | No | Somewhat | Yes |
| Handles unstructured input | No | No | Yes. Chat logs, documents, narratives |
| Determinism | Total | Total | Variable, needs pinning and low temperature |
| Best role | Hard blocks and step-ups | The primary score | Analyst assistance in case review |
Why gradient boosted trees rather than a neural network
- They win on tabular data. For structured features with no spatial or sequential structure, boosted trees consistently match or beat deep models.
- Fast inference on CPU, which matters at 5,600 decisions per second.
- Feature importance and SHAP values come nearly free, and explainability is a hard requirement here.
- Robust to unscaled, mixed-type, partially missing features, which describes real fraud features exactly.
- They handle extreme class imbalance well, and fraud is often under 0.1% of transactions.
Where the LLM genuinely helps
- Case summarisation. Turn 200 transactions, 3 device changes and a support thread into a paragraph an analyst reads in 20 seconds instead of 10 minutes.
- Unstructured evidence. Read the chat transcript where a customer was socially engineered, and identify the APP fraud pattern no tabular feature encodes.
- Narrative generation for suspicious activity reports, drafted for human review and sign-off.
- Pattern discovery across confirmed cases, proposing candidate rules for a human to evaluate in shadow mode.
Three-tier decisions: allow, review, block
A binary decision forces a bad tradeoff. Adding intermediate actions is what makes the false positive budget achievable.
score 0.00 → 0.70 ALLOW ~99.2% of traffic
score 0.70 → 0.90 STEP UP ~0.6%
score 0.90 → 0.97 REVIEW ~0.15%
score 0.97 → 1.00 BLOCK ~0.05%
only 0.05% is auto-blocked, which is how a 0.1% false
positive budget on blocks becomes achievable at all.| Action | Customer experience | Fraudster experience |
|---|---|---|
| Allow | Nothing | Succeeds |
| Step up | One extra factor: OTP, biometric, in-app confirm. Mild friction | Usually stopped. They lack the second factor |
| Delay plus review | "Processing, up to 30 minutes." Tolerable for a large transfer | Stopped, and the delay itself is the defence: it gives humans time |
| Block | Cannot use their own money. A call, a complaint, possibly a lost customer | Stopped |
Choosing the thresholds honestly
expected cost of a threshold t:
cost(t) = FP(t) × cost_of_false_positive
+ FN(t) × average_fraud_loss
cost_of_false_positive is NOT just a support call. it includes
churn probability, reputational effect and regulatory complaints.
the threshold is a business decision with an engineering input,
and presenting it that way is the senior move.Automated blocking and the blast radius
Automated blocking is necessary and dangerous. A bug or a poisoned signal can lock out a large fraction of the customer base in minutes, and that has happened to real banks.
- A global rate limit on the block action itself. If the system tries to block more than N accounts per minute, it stops blocking and pages a human. This single control prevents the worst outcome available.
- Proportionality. Block the narrowest thing that works: the transaction, then the card, then the channel, then the account. Full account freeze is the last resort.
- Automatic expiry. An automated block expires after a defined period unless a human confirms it. Failure mode becomes unblocking rather than permanent lockout.
- A kill switch. One flag disables automated blocking entirely, leaving rules in shadow. Part 13 covers the operational side.
- Never block the whole bank. No rule may match on a property shared by all customers. This must be validated when the rule is authored, not discovered in production.
// the circuit breaker on our own enforcement. it is the difference
// between a bad rule and a bank-wide incident.
async function enforceBlock(target: Target, reason: Reason) {
const rate = await counters.increment('blocks:global', { window: 60 });
if (rate > BLOCK_RATE_CEILING) {
// we are probably the problem, not the customers.
await alerts.page('fraud.block_rate_exceeded', { rate });
await flags.disable('fraud.auto_block');
return { blocked: false, reason: 'enforcement_suspended' };
}
// narrowest effective scope, and an expiry by default
return blocks.create({
scope: narrowestScopeFor(reason),
target,
expiresAt: addHours(now(), 24), // requires human confirmation to persist
createdBy: 'automated',
reason
});
}Account and wallet flagging state machine
"Blocked" is not one state. Different restrictions serve different purposes, and conflating them either over-restricts customers or fails to contain attacks.
| State | Credits in | Debits out | Used for |
|---|---|---|---|
| active | Yes | Yes | Normal |
| watch | Yes | Yes | Elevated monitoring, no customer impact. The most useful state, and the most underused |
| debit_restricted | Yes | No | Suspected compromise. Contains the attack while salary still arrives |
| credit_restricted | No | Yes | Suspected mule collection point. Stops it receiving more |
| frozen | No | No | Confirmed fraud, or a legal instruction |
| closed | No | No | Terminal. Residual balance handled by a defined process |
CREATE TABLE account_restrictions ( id UUID PRIMARY KEY, account_id UUID NOT NULL, state TEXT NOT NULL, -- WHY, in a form that can be shown to a customer and an auditor reason_code TEXT NOT NULL, reason_detail TEXT, -- WHO: automation, an analyst, or a court. drives who may lift it. imposed_by TEXT NOT NULL, -- automated | analyst:id | legal -- automated restrictions EXPIRE. legal ones do not. expires_at TIMESTAMPTZ, lifted_at TIMESTAMPTZ, lifted_by TEXT, created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); -- the restriction is checked on the posting path, and it is checked -- against the DIRECTION of the entry, not merely its existence. CREATE INDEX restr_active ON account_restrictions (account_id) WHERE lifted_at IS NULL;
- Automation may impose a restriction; only a human may lift a confirmed one.
- A legal restriction cannot be lifted by an analyst at all, and the system must enforce that rather than rely on process.
- Four-eyes on lifting anything above a value threshold, because lifting restrictions is a prime insider-abuse vector.
- Every imposition and lifting is immutably logged, with the actor and the reason. This log is itself monitored for insider anomalies.
Sanctions, PEP, AML transaction monitoring
Adjacent to fraud and legally distinct. Fraud protects the bank and its customers from loss. Financial crime compliance is a statutory obligation whose failure brings fines and criminal liability, not just losses.
| Control | What it checks | When |
|---|---|---|
| Sanctions screening | Names against OFAC, UN, EU, UK and local lists | Onboarding and every cross-border payment. Blocking, not advisory |
| PEP screening | Politically exposed persons and their associates | Onboarding and periodically. Triggers enhanced due diligence, not a block |
| Adverse media | Public reporting of financial crime | Onboarding and periodically |
| Transaction monitoring | Structuring, rapid movement, high-risk corridors, unexplained activity | Continuous |
| Threshold reporting | Transactions above a statutory amount | Per jurisdiction, per Part 7's policy engine |
Fuzzy name matching, and why it is genuinely hard
sanctions list: "Mohammed Al-Hassan"
must also match: Muhammad Al Hassan · Mohamad AlHassan
Hassan, Mohammed · محمد الحسن
must not match every one of the millions of people
legitimately named Mohammed Hassan.
false negative = sanctions violation, fines, criminal exposure
false positive = a real customer blocked from their money
so the tuning is deliberately conservative and the review queue
is staffed accordingly. this is a people problem as much as a
technical one, and the design must acknowledge that.Case management and the feedback loop
The part that is always underbuilt, and the part that determines whether the system improves. Without labels there is no learning, and labels come from human decisions.
detection → case → analyst decision → label → training data
↓
retrain, re-tune
without the loop, the model degrades from the day it ships
because attackers adapt and the model does not.
- The triggering decision, its score, and the top contributing features by SHAP value.
- Which rules fired, at which versions, including those in shadow mode.
- The account's baseline and precisely how this deviated from it.
- A transaction timeline with the graph neighbourhood rendered visually.
- Linked cases on the same device, beneficiary or network cluster.
- An LLM-drafted summary, clearly marked as assistance rather than a finding.
The label problem, which is subtler than it looks
- Blocked transactions have no outcome. We prevented it, so we never learn whether it was truly fraud. Mitigated by letting a small random sample of high-score transactions through and observing, which is uncomfortable and necessary.
- Confirmation lag. Fraud is often confirmed weeks later via a dispute, so recent data is systematically under-labelled and a model trained on it underestimates fraud.
- Analyst inconsistency. Different analysts label the same case differently. Needs calibration sets and inter-rater measurement, exactly like any other annotation pipeline.
Sketch v9: risk in the path, not beside it
The two paths, and why both are necessary
| Fast path | Deep path | |
|---|---|---|
| Latency budget | 150 ms | Seconds to hours |
| Inputs | Online features, counters, precomputed scalars | The whole event log, the graph, unstructured evidence |
| Techniques | Rules plus boosted trees | Graph algorithms, baselining, LLM triage |
| Can block a live transaction? | Yes | No, but it can restrict the account for the next one |
| Failure posture | Fails open | Degrades: detection is delayed, not lost |
| Catches | Card testing, known patterns, velocity | Mule networks, slow-burn fraud, novel patterns |
What changed, and the cost accepted
| Change | Driven by | Cost accepted |
|---|---|---|
| Rules engine, rules as versioned data | Attackers adapt faster than releases | A rule authoring and governance process |
| Mandatory shadow mode | Rules that look obvious have terrible precision | Days of delay before a rule can enforce |
| Feature store, online and offline | Training and serving skew silently destroys accuracy | Real infrastructure, and feature governance |
| Graph analysis in batch, materialised | Mule structure is invisible per transaction | Hours of detection latency on that signal |
| Four-tier actions including step-up and delay | A 0.1% false positive budget on blocks | Step-up and delay flows to build in the product |
| Rate limit on our own blocking | A bad rule can lock out the bank | Under a real attack, enforcement may suspend and page |
| Six-state restriction model | Freezing an account harms customers unnecessarily | Every money path must check direction-aware restrictions |
| Sanctions fails closed | A statutory obligation, unlike fraud | Cross-border payments queue when screening is down |