Your Fraud Model Is Not a Product: The Real Work In Fintech ML

Bygaz

Why this matters right now

If you run a fintech stack today—payments, lending, wallets, brokerage, BaaS—you’re already in the machine learning business, whether you like it or not.

You don’t get to opt out:

  • Card networks and processors are pushing more liability and chargeback risk downstream.
  • Regulators expect explainable, auditable decisions in AML/KYC and credit underwriting.
  • Fraud rings iterate faster than your quarterly roadmap.
  • Margins are thin enough that 20–40 bps of avoidable loss is a board-level problem.

The pitch you’ve likely heard: “Just plug in an ML fraud/AML/KYC/risk engine and watch loss rates fall.” Reality is messier:

  • A “good” model with bad data plumbing is worse than no model.
  • Incremental fraud detection often trades off hard against conversion and revenue.
  • ML systems drift silently, then fail catastrophically.
  • Regulatory and compliance constraints turn neat Kaggle solutions into useless artifacts.

This post is about the boring, mechanical details that separate “we tried ML for fraud once” from “we reduced fraud losses by 30% without tanking approval rates.”

Assume a reader who cares about:

  • measurable impact in bps and basis of approval,
  • provable controls for audit and regulators,
  • latency budgets in milliseconds,
  • and real cost of ownership (inference + people + process).

What’s actually changed (not the press release)

Three shifts in the last ~3 years materially changed how to build fintech infrastructure around ML, especially for fraud, AML, and risk.

1. Feature engineering moved into infrastructure

Old world:
Risk features were ad-hoc SQL queries and Python scripts:

  • “Count of failed logins in past 24h”
  • “Cards used on more than 3 devices in 7 days”
  • “Merchant MCC risk score lookup”

Every team had its own definitions; every pipeline broke differently in prod.

New world:

  • Feature stores (homegrown or off-the-shelf) are becoming part of the infra layer.
  • Online/offline consistency is an explicit design goal, not a hope.
  • Real-time aggregations over event streams (Kafka/Kinesis/PubSub) are standard.

What changed is not that “feature stores exist” (marketing); it’s that:

  • You can treat features as versioned, testable, reusable APIs.
  • Risk, payments, and product teams can share the same semantic definitions.
  • You can run A/B tests on feature sets, not just model weights.

2. Graph-based and sequence models are actually deployable

Fraud and AML are graph problems:

  • Devices ↔ users ↔ cards ↔ merchants ↔ bank accounts ↔ IPs
  • Transactions form sequences with patterns (time, location, merchant, channel).

What’s new is not the math; it’s the operational ability to:

  • Maintain near-real-time device/user/merchant graphs in memory or low-latency stores.
  • Run efficient graph-based features (centrality, shared attributes, cluster flags).
  • Deploy sequence models (GRU/LSTM/Transformers-lite) under a 50–100 ms SLA.

You don’t need to ship a giant graph neural network. For many teams, the shift is:

From “single-row tabular model per transaction”
To “model with state: features over the customer’s, device’s, and counterparty’s history and connections.”

That change alone often beats fancier model architectures.

3. Regulators and partners care about process, not your ROC curve

In AML/KYC, underwriting, and risk-based pricing:

  • You will need to explain why you rejected/flagged an entity.
  • You will need to show stable performance and fairness over time.
  • You will need to reconstruct the decision path months or years later.

What’s new in practice:

  • Model governance and MLOps controls are now table stakes when negotiating with banks, card schemes, or regulators.
  • “We use XGBoost and SHAP values” is not enough; you need:
    • versioned model + feature sets,
    • documented risk appetite,
    • challenge models,
    • independent validation.

The delta: the non-code parts (documentation, process, monitoring) now decide whether your ML stack is an asset or a liability.

How it works (simple mental model)

Strip away the vendor names. A production-grade ML system for payments/fraud/AML in fintech has five moving parts:

  1. Event ingestion
  2. Stateful feature computation
  3. Scoring decision service
  4. Policy engine
  5. Feedback and monitoring loop

Keep this picture in your head:

Streams → Features → Model → Policy → Outcome → Feedback

1. Event ingestion

All relevant events are captured as immutable logs:

  • Payments: auths, clears, reversals, chargebacks, disputes.
  • Identity: signups, KYC checks, document uploads, device fingerprints.
  • Behavioral: logins, failed attempts, session metadata.
  • External: sanctions lists, watchlists, bureau data, consortium signals.

Principles:

  • Append-only; no in-place updates.
  • Strict schemas with evolution control.
  • Idempotent ingestion.

If this layer is messy, everything else is noise.

2. Stateful feature computation

Features are not just “columns”; they’re stateful functions over events:

  • Per-entity counters: num_tx_last_5m, sum_tx_amount_24h
  • Velocity features: unique_device_count_7d
  • Graph features: num_shared_cards_with_known_fraud_accounts
  • KYC/risk features: kyc_doc_mismatch_score, business_category_risk_bucket

Key properties:

  • Online store for real-time inference (e.g., sub-10 ms reads).
  • Offline store for training and backtests.
  • Strong guarantees that the same code computes both.

In practice, this is often:

  • A stream processor (Flink/Spark/Beam/custom Go/Java service) computing aggregates.
  • A key-value store (Redis, Scylla, Dynamo, etc.) for online features.
  • A columnar warehouse (Snowflake/BigQuery/Redshift) for offline features.

3. Scoring decision service

This is the ML inference service:

  • Inputs: features + raw request context.
  • Outputs: scores (fraud probability, AML risk, income estimate, etc.).

Operational needs:

  • Latency budget: often 10–50 ms at P95.
  • Strict SLAs: degraded mode (fallback) when features or model service fail.
  • Versioning: ability to run multiple models in parallel for A/B and shadowing.

This is where you decide whether to:

  • Approve/decline/hold a transaction.
  • Step-up KYC (ask for more docs).
  • Trigger manual review or SAR workflow.

4. Policy engine

This is where business and compliance logic lives:

  • Thresholds: if score > 0.9 then decline.
  • Combinatorial logic: if (score > 0.7 and amount > $500) then hold.
  • Jurisdiction rules: different actions by country/region/product.
  • Regulatory constraints: e.g., mandatory screening actions for certain lists.

Explicitly separate:

  • Model (statistical signal)
  • Policy (risk appetite + regulation + business trade-offs)

This lets you:

  • Change thresholds and rules without touching model code.
  • Run simulations: “What if we lower this threshold by 0.05?”

5. Feedback and monitoring

Closed-loop is what makes ML worth it:

  • Labeling: fraud chargebacks, confirmed AML cases, KYC pass/fail, user feedback.
  • Performance: precision/recall at operational thresholds, not global AUC.
  • Bias/fairness: group-level metrics under regulatory scrutiny.
  • Drift: input distributions, data quality, and concept drift.

For fintech, you need two layers of monitoring:

  • Risk metrics: loss rates, false positive rate, manual review rates per segment.
  • Tech metrics: latency, feature availability, model response rate, error budgets.

Where teams get burned (failure modes + anti-patterns)

Patterns seen in production fintech systems.

1. Treating the model as the product

Anti-pattern:

  • Build “the best” fraud/AML model offline.
  • Hand it to the platform team with “please deploy this.”
  • Assume performance in prod will match validation metrics.

Failure modes:

  • Feature leakage in training that doesn’t exist in real-time.
  • Silent feature drift breaks calibration and thresholds.
  • Latency and timeouts cause unobserved fallbacks to legacy rules.

Better pattern:

  • Co-design data, infra, and model.
  • Start from evaluating decisions, not maximizing offline metrics.
  • Make model owners accountable for production KPIs and alerting.

2. Ignoring label quality and delays

In payments and fraud:

  • Chargebacks can take 60–120 days to finalize.
  • AML cases can take months to resolve.
  • Many “fraudulent” events are actually disputes or merchant issues.

Common mistake:

  • Train on “anything disputed within 30 days” as fraud.
  • Ignore the funnel: what fraction of true fraud is ever labeled?

Impact:

  • Models learn “customer irritation” or “high-support merchants,” not fraud.
  • You under-estimate emerging fraud patterns that don’t yet generate chargebacks.

Mitigations:

  • Explicitly model label delay and use temporal cross-validation.
  • Supplement with weaker signals (network intel, device flags) for early warning.
  • Maintain a golden set of hand-labeled fraud for regular rebaselining.

3. One-shot thresholds without business simulation

Typical story:

  • Risk team rolls out a new model with aggressive thresholds.
  • Fraud drops 40%. Celebration.
  • Two months later, revenue is down 15% and CAC is up because legit users are blocked.

Root cause:

  • No end-to-end model of:
    • approval → conversion → LTV,
    • vs. fraud loss, operational costs (manual review), regulator risk.

Mitigation:

  • Run backtests with realistic behavioral assumptions.
  • Attach dollars to:
    • false positives (lost good users, friction),
    • false negatives (loss, chargebacks, regulatory risk).
  • Encode this into policy engine experiments with caps:
    • “New policy cannot reduce approval rate by more than 50 bps in any cohort.”

4. Monolithic, opaque “AI layer”

Some teams centralize all risk/AML/fraud logic into a single “AI brain.”

Problems:

  • Hard to reason about interactions between use cases (card-present vs card-not-present vs onboarding).
  • Hard to test incremental changes.
  • Hard to explain decisions to regulators or partners.

Better pattern:

  • Narrow, composable services:
    • device risk service,
    • payment fraud service,
    • identity/KYC risk service.
  • Shared feature platform, not shared “AI brain.”
  • Explicit contracts and decision schemas between services.

5. Underestimating compliance and explainability requirements

Common mistake:

  • Assume tree-based models + SHAP values will satisfy AML regulators.
  • Assume “we log explanations” is enough.

Real constraints:

  • Regulators want to see:
    • documented model development process,
    • challenge models,
    • limitations and known biases,
    • change management and approval process,
    • ability to override model decisions in defined scenarios.

Mitigations:

  • In high-stakes areas (AML, KYC, credit underwriting), prefer:
    • simpler models with strong governance,
    • well-documented rules plus ML, not just ML.
  • Build an “explanation service” that:
    • translates feature-level output into human-readable rationales,
    • is itself versioned and auditable.

Practical playbook (what to do in the next 7 days)

This assumes you already have some fraud/risk/AML logic, even if it’s all rules.

Day 1–2: Map reality

  • Draw the end-to-end flow for one critical decision:
    • e.g., “Card-not-present payment authorization in EU.”
  • For that flow, document:
    • Data sources used (events, KYC, device, external).
    • Features computed and where.
    • Decision models/rules and owners.
    • Latency and availability requirements.
  • Identify:
    • where you have no labels,
    • where your labels are delayed,
    • where you cannot reconstruct a past decision.

Outcome: a single-page architecture that everyone agrees reflects reality.

Day 3–4: Instrument and baseline

  • Add explicit metrics for that decision:
    • Approval rate by cohort (device, region, product).
    • Chargeback/fraud rate (where applicable).
    • Manual review rate and SLA.
  • Add technical metrics:
    • P95/P99 latency for feature fetch and model scoring.
    • Feature completeness (fields present vs null).
  • Build a confusion-matrix view at operational thresholds:
    • False positive rate (good users blocked/stepped up).
    • False negative rate (fraud passed).

Outcome: you know how the current system actually performs in production, in numbers.

Day 5: Identify 1–2 high-leverage feature gaps

From the map and metrics, pick one narrow improvement:

Examples:

  • Add simple velocity features:
    • “# of cards per device in last 24h”
    • “# of accounts per IP in last 7 days”
  • Add device graph features:
    • “is this device connected to any known-bad account?”
  • For AML:
    • “jurisdiction risk score” and “business category risk score” as explicit features.

Implement these as:

  • Stream-processing jobs or periodically refreshed aggregates.
  • Features exposed via a well-defined interface with monitoring.

Day 6: Wire those features into a shadow model or rules

Low-risk changes:

  • Add new features as inputs to a shadow model, evaluated offline:
    • No effect on live decisions yet.
  • Or add them as manual-review triggers instead of auto-decline.
  • Run backtests on historical data to estimate:
    • incremental fraud catch,
    • incremental good-user friction.

Outcome: a quantified estimate of value with low production risk.

Day 7: Decide on the next 30 days

Based on what you saw:

  • If your data plumbing is weak:
    • Invest first in event schemas, feature computation, and logs.
  • If your infra is fine but labels and metrics are poor:
    • Invest in labeling workflows, analyst tools, and monitoring.
  • If you’re strong on both:
    • Start migrating from rules-only to rules + models with a clear rollout plan:
      • 1% traffic → 10% → 50% with guardrails on approval and loss.

Write down:

  • A 1-page doc of “What we’ll change in the next 30 days and how we’ll measure it.”
  • Ownership: who owns each metric (fraud rate, approval rate, latency).

Bottom line

In fintech, ML is an infrastructure problem, not an algorithm contest.

The teams that win on fraud, AML/KYC, and risk don’t:

  • chase the latest model architecture, or
  • buy a black-box AI platform and call it done.

They:

  • Treat events and features as first-class, versioned infrastructure.
  • Separate signal generation (models) from risk appetite (policy).
  • Invest in label quality, monitoring, and governance early.
  • Make ML teams responsible for production decisions, not offline metrics.

If your “AI for fintech” strategy doesn’t start with boring questions about schemas, latency, labels, and regulators, it’s an experiment—not infrastructure.

Build for the latter.

Similar Posts