Your Fraud Model Is Not a Product: Hard Lessons From Shipping ML in Fintech

Bygaz

Why this matters right now

In fintech, “we’ll add ML for fraud later” used to be a reasonable lie you told investors.

It isn’t anymore.

Three things have converged:

  • Payment rails are faster (instant payouts, RTP, SEPA Instant) → fraud loss windows shrink from days to seconds.
  • Fraud and AML/KYC pressure is higher → regulators expect continuous monitoring, explainability, and auditable decisions.
  • Attackers are running playbooks at scale → account takeovers, money mules, synthetic identities, social engineering, and refund abuse are industrialized.

The result: if you run a payments, lending, or neobank stack and your risk & compliance is still mostly rules + dashboards + manual reviews, you are already paying for it:

  • Higher chargeback and fraud losses.
  • More “false positive” user friction at onboarding and checkout.
  • Ballooning operations teams doing case review.
  • Failing regulatory expectations around ongoing monitoring and suspicious activity detection.

But “add ML” is an overloaded instruction. In practice, ML in fintech infrastructure (fraud, AML, KYC, risk, compliance, open banking) is less about clever models and more about building a reliable decisioning system under hard constraints:

  • Latency (tens of milliseconds).
  • Data quality (messy, late, multi-tenant).
  • Adversaries (actively probing your decision boundary).
  • Regulation (need to explain why you blocked or allowed something).

This post is about those systems, not about model architectures.

What’s actually changed (not the press release)

The story isn’t “transformers for fraud” or “GenAI for KYC.” The real shift is more boring and more operational.

1. Real-time infrastructure has caught up

Historically:

  • Core banking and card processors were batchy.
  • You got chargeback data and “confirmed fraud” labels days/weeks later.
  • Building streaming features (velocity checks, device graph, merchant graph) was a heavy lift.

What’s changed:

  • Real-time feature stores and streaming pipelines (e.g., Kafka + Flink + a feature store) are now standard in serious fintech teams.
  • Low-latency decision APIs are being treated as first-class products.
  • Integration with card networks, open banking APIs, and device intelligence is more straightforward.

This enables online models that update risk in-session, not just as nightly jobs.

2. Data exhaust is richer and more structured

  • Device fingerprints, behavioral biometrics, and browser signals.
  • Open banking / PSD2 transaction histories.
  • Identity data from KYC providers (document verification, liveness, sanctions lists, PEP, adverse media).
  • Merchant, BIN, and IP intelligence.

Instead of just “card number + amount + merchant,” you now have a graph of entities and behaviors. The differentiator is whether you can actually turn this into stable, governed ML features.

3. Regulators are slowly getting more model-aware

Regulators (depending on jurisdiction) are:

  • Asking pointed questions about model governance, validation, and drift.
  • Expecting ongoing transaction monitoring for AML, not just onboarding checks.
  • Focusing on explainability and fairness in credit and risk decisions.

They’re not asking “are you using AI?”; they’re asking “how do you know this risk policy works, and how do you monitor it?”

4. Adversaries are probing your ML boundary

Once you deploy ML into payments or fraud:

  • Botnets and fraud rings can gradient-hack you in the wild by iterating on small changes and observing outcomes.
  • They can weaponize your false-positive blind spots (e.g., “high trust” device traits, cooperative merchants, naive velocity checks).

Your fraud/risk system is now an attacked system, not a static classifier. This pushes you toward defense-in-depth, randomized checks, and continuous policy changes.

How it works (simple mental model)

Forget models for a minute. Think of your fraud / AML / risk stack as this loop:

  1. Observe

    • Collect all signals about identities, devices, accounts, transactions, merchants, counterparties, behavioral flows.
    • Normalize and join them into a decision context in real-time.

  2. Decide

    • A policy engine that combines:
      • Deterministic rules (hard business constraints, regulatory checks).
      • Risk scores from ML models.
      • Case-specific workflows (step-up auth, extra KYC, manual review).

  3. Act

    • Approve, decline, hold, step-up, file SAR, trigger KYB refresh, delay payout, flag for enhanced due diligence.
    • Must be low latency and traceable.

  4. Learn

    • Log decisions, outcomes (chargebacks, disputes, law enforcement responses, account closures, SAR feedback), and reviewer labels.
    • Periodically retrain models and refine rules using these labels.
    • Monitor performance and drift.

You can overlay any concrete use case on this:

  • Card-not-present payment fraud.
  • Account takeover / social engineering.
  • AML transaction monitoring and SAR detection.
  • KYC/KYB risk scoring (identity mismatch, document fraud).
  • Credit risk and line management.

Where ML fits in this loop

ML is best used as risk estimation at specific points, not as an end-to-end oracle:

  • Pre-authorization: device risk, behavioral risk, identity risk.
  • At transaction: probability of fraud/chargeback, money-muling, sanctioned party involvement.
  • Post-transaction: unusual patterns across accounts / merchants, AML anomalies, synthetic identity clustering.

Model flavors:

  • Supervised models on labeled fraud/chargeback/AML cases.
  • Anomaly detection / peer group deviation for AML and emerging fraud patterns.
  • Graph-based models for rings, mules, collusion networks.

The product surface, however, is always a policy: “if risk score > X and involves high-risk country, route to manual review; else approve but delay settlement by N hours,” etc.

Where teams get burned (failure modes + anti-patterns)

Patterns seen repeatedly across fintech infra teams.

1. Treating the model as the product

Anti-pattern:

  • A team trains a high-AUC fraud model offline.
  • They deploy it as a scoring API.
  • They expect fraud to magically drop after “integrating ML.”

What happens:

  • The model is bolted onto a rules engine with ad hoc thresholds.
  • Analysts can’t interpret or override decisions easily.
  • No clear feedback loop from outcomes to retraining.
  • Security/compliance treat it as a black box and get nervous.

Fix:

  • Make the decision engine the product:
    • Strong policy DSL or UI that mixes rules, lists, and scores.
    • Versioned policies with approval and rollback.
    • Full audit trail and explanations (“this was declined because…”).
    • Tight integration with ops tooling for manual review.

2. Ignoring label bias and feedback loops

Anti-pattern:

  • Only declined or chargebacked transactions are labeled as fraud.
  • Approved transactions that were never disputed are treated as “safe.”
  • New model is trained on this skewed data.

What happens:

  • Model overfits to what you already decline.
  • Blind spots around new fraud patterns or underserved geos/segments.
  • Active learning from fraudsters’ probing worsens this.

Fix:

  • Explicitly track:
    • Observed positives (confirmed fraud/AML events).
    • Observed negatives (transactions with strong evidence of good behavior).
    • Unlabeled (approved but never conclusively verified).
  • Use strategies like:
    • Periodic random exploration (low-volume random sampling sent to manual review).
    • Post-hoc retrospective reviews on model’s lowest-confidence approvals.
    • Separate metrics for coverage (where model has seen enough data) vs global performance.

3. Latency and reliability surprises

Anti-pattern:

  • Model is trained and tested offline.
  • In production, feature computation involves heavy joins, cross-service calls, or cold-start cache misses.

What happens:

  • P99 latency blows your SLA to the card network or payment rail.
  • Engineers add fallbacks that skip most features when slow → behavior divergence, silent risk gap.
  • On-call teams now own a brittle, distributed decisioning path.

Fix:

  • From day one, design:
    • Online feature store with precomputed features keyed by account, device, merchant, etc.
    • Strict budget for inference (e.g., < 20ms at P99 end-to-end in the risk service).
    • Degradation modes that are explicit policies, not ad hoc (e.g., “if model unavailable, fall back to rules-only policy vX”).

4. Non-robust to adversaries

Anti-pattern:

  • Team treats fraud model like a spam classifier.
  • All features are transparent and deterministic.
  • No thought about strategic behavior.

What happens:

  • Fraudsters E2E test against your system:
    • Trying amounts, IP regions, device attributes, velocity.
    • Tuning up to your decision threshold.
  • Major breach when one combination sails through.

Fix:

  • Assume adaptive adversaries:
    • Limit the use of trivially spoofable single signals as hard gates.
    • Use feature families (device, behavior, entity network) instead of single rules.
    • Add some non-determinism for high-risk bands (e.g., random sampling to review).
    • Continuous red-teaming / purple-teaming of your risk system.

5. “AML is just fraud with different labels”

Anti-pattern:

  • Reusing the fraud stack directly for AML alerts or SAR triage.
  • Minimal involvement from compliance/legal.

What happens:

  • High false-positive volume from poorly tuned rules, plus:
  • ML that’s not aligned with regulatory typologies or filing standards.
  • Examiners find gaps in coverage or undocumented rationales → pain.

Fix:

  • Treat AML/KYC/KYB as a sibling domain, not a subset:
    • Co-design models with compliance teams.
    • Map features and risks to known typologies (structuring, sanctions evasion, mule activity, layering).
    • Keep explanation layers and playbooks tightly documented.
    • Separate metrics: regulatory risk vs financial loss.

Practical playbook (what to do in the next 7 days)

Concrete moves for a team running or planning fintech infra with ML.

1. Map your current decision loop

Create a one-page diagram for one high-impact flow (e.g., online card payment, onboarding KYC):

  • Inputs: what data is available at decision-time? Latency constraints?
  • Current logic: which rules, scorecards, vendors are used?
  • Outcomes: approve/decline/review/step-up? Who can override?
  • Feedback: how do you know if it was wrong?

This will reveal where ML would actually matter vs where you’re blindly “adding AI.”

2. Identify your “Ground Truth” sources

For that same flow, list:

  • What constitutes a confirmed bad outcome?
    • Fraud chargeback codes.
    • Confirmed account takeovers.
    • SAR filings or law enforcement feedback.
  • What constitutes a confirmed good outcome?
    • Long-lived accounts with healthy usage.
    • Merchants with low dispute ratios.
  • How fast do you learn each?

If you can’t answer this, your ML roadmap should start with instrumentation and labelling.

3. Baseline your metrics

Agree on 3–5 clear metrics per use case. Examples:

  • Fraud:
    • Fraud loss as bps of processed volume, by segment.
    • Fraud capture rate (what % of fraud is stopped before loss).
    • False positive rate at checkout/onboarding (good users impacted).
  • AML:
    • Alert volume and true positive rate (alerts leading to SAR).
    • Time from suspicious activity to escalation and review.
  • Ops:
    • Manual review cases per 1,000 transactions.
    • Average handling time and queue delay.

Even if they’re rough, make them explicit. You need this to evaluate any ML change.

4. Harden the decision surface

Regardless of model plans, you can usually improve safety quickly:

  • Add basic entity linking:
    • Cluster accounts via device, IP, bank details, card tokens.
    • Flag cross-entity velocity (multiple accounts → one dest).
  • Make policy configuration versioned and auditable:
    • Every change to rules or thresholds should be reviewable and revertible.
  • Implement structured logging for decisions:
    • Inputs, features (or hashes), rules fired, scores, final outcome, latency.

This sets the stage for future ML while improving your current rules system.

5. Design your first ML integration as a bounded experiment

Pick one scoped risk question, e.g.:

  • “What is the probability this transaction will be fraudulently charged back?”
  • “What is the probability this new account is a mule?”

Then:

  • Define a single risk score API with clear SLAs (latency, uptime).
  • Deploy it in shadow mode:
    • Log scores but keep production decisions unchanged.
    • Run for several weeks to understand drift, calibration, and weird edge cases.
  • Add the score into the decision engine for a specific segment only (e.g., new devices in risky geos) with a flag to roll back.

Treat this as product development, not a data science demo.

Bottom line

In fintech, “ML for fraud and AML” is not a differentiator by itself anymore. Owning a robust decisioning system is.

Teams that win in payments, open banking, and regtech tend to:

  • Treat risk, fraud, and compliance systems as first-class products, not bolt-on models.
  • Invest early in data plumbing, feature governance, and decision explainability.
  • Design for adversarial pressure and regulatory scrutiny, not just ROC curves.
  • Iterate with tightly scoped, measurable experiments rather than big-bang “AI transformations.”

If your current roadmap says “add ML to reduce fraud,” rewrite it to:

  • “Instrument ground truth;”
  • “Build a versioned decision engine;”
  • “Ship a bounded risk score into one policy with explicit success metrics.”

Everything else can follow from there.

Similar Posts