Your ML System Is Not “Done” at Launch: A Pragmatic Guide to Evaluation, Drift, and Cost

Bygaz

A dimly lit operations room with large wall displays showing interconnected graphs, data pipelines, and model performance charts, a few engineers silhouetted at desks monitoring systems, cool blue and teal tones, cinematic wide-angle composition conveying complex machine learning infrastructure in motion

Why this matters this week

A pattern keeps repeating across teams that ship real systems:

  1. Models go live with “good enough” offline metrics.
  2. Product impact is unclear after the initial launch.
  3. Six months later, infra spend has doubled, alert noise is up, and nobody trusts the metrics.

This isn’t about frontier models or research. It’s about applied machine learning where you:

  • Spend real money on GPUs/CPUs and storage.
  • Touch customer data and have to worry about security and governance.
  • Are accountable for uptime, SLAs, and regression risk.

What’s changed recently is not that “AI is getting better.” It’s that:

  • Data distributions are shifting faster due to product changes and user behavior.
  • LLMs and embedding models have made inference costs and latency much more variable.
  • Execs now expect ML to be measurable like any other service, not a special R&D artifact.

If your evaluation and monitoring story is weak, you’ll either overfit to vanity metrics or throttle innovation because every change is scary.

This post is about a boring but critical question:
How do you run ML in production with sane evaluation, drift detection, reliable feature pipelines, and a clear cost/performance trade-off?

What’s actually changed (not the press release)

The underlying idea of monitoring and evaluation hasn’t changed in years. What has changed is the environment and failure surface.

1. More models are online, not batch-only

  • Recs, search ranking, fraud, and chat/assistants are now online systems with tight latency budgets.
  • This turns what used to be “train once a week and pray” into a continuous decision engine that must handle:
    • Spikes in traffic.
    • Real-time features.
    • Backpressure and partial outages.

2. The feedback loop is messier

  • With generative models and personalization, you get:
    • Delayed signals (did the generated response reduce tickets over 30 days?).
    • Biased signals (only some users rate, power users dominate metrics).
  • Classic A/B testing and offline accuracy are still necessary, but no longer sufficient on their own.

3. Infra cost is now a first-class constraint

  • Moving from small tabular models to large embedding-based or LLM-powered systems:
    • Inference can dominate your infra bill.
    • Feature stores, vector DBs, and streaming add persistent cost.
  • Engineers are being asked: “Is this extra 3 points of recall worth 2× the cost and 40ms more latency?”
    Most teams don’t have the numbers to answer.

4. Governance and security guardrails are real

  • Data residency, PII handling, access control on features, and auditability:
    • These were “nice to have” around reports.
    • They’re now required by security/legal the moment you touch customer data in online ML.

The net: you can’t treat ML as a black box microservice anymore. You need observability and operational discipline at the same level as your core APIs.

How it works (simple mental model)

Here’s a minimal mental model to align evaluation, monitoring, drift, and cost for an ML system in production.

Think in four loops, each with its own signals and time horizons:

Loop 1: Request-level health (milliseconds to minutes)

Goal: “Is this service functioning at all?”

  • Metrics:
    • QPS, p50/p95/p99 latency, error rate, timeouts.
    • Upstream/downstream dependency errors (feature store, DB, vector index).
  • Tools:
    • Standard service monitoring (Prometheus-style metrics, logs, traces).
  • Decision type:
    • SRE-style: rollback, failover, circuit breakers.

This is not “ML monitoring” in a fancy sense. If this loop is broken, nothing else matters.

Loop 2: Prediction quality proxies (minutes to days)

Goal: “Are outputs plausible right now?”

  • When you have labels quickly (fraud, spam detection):
    • Log (input, features, prediction, label, timestamp, model version).
    • Compute short-horizon metrics: precision/recall, AUC, calibration, etc.
  • When you don’t have labels quickly (LLM responses, long-term outcomes):
    • Use proxies:
      • Engagement metrics (click-through, dwell time).
      • Human rating samples (e.g., 1–5 usefulness).
      • Heuristic checks (toxicity scores, length, presence of disallowed patterns).

This loop catches “model went dumb” issues sooner than waiting for end-to-end business KPIs.

Loop 3: Distribution drift (hours to weeks)

Goal: “Is the world I’m predicting on still similar to training?”

You don’t need perfect statistical purity; you need early warning.

  • Monitor distributions:
    • Input features (e.g., user geography, device, query length).
    • Embeddings or feature projections (simple PCA/histograms).
    • Prediction distributions (e.g., score histograms over time).
  • Basic methods:
    • Population stats (mean, variance) per feature vs. baseline.
    • Simple divergence scores (PSI, KL approximation, Kolmogorov–Smirnov tests).
  • Interpretation:
    • Drift is a signal, not a verdict.
    • If important features or outputs drift suddenly:
      • Check for upstream data bugs or business changes.
      • Consider retraining or adapting model.

Loop 4: Business impact and cost (weeks to months)

Goal: “Is this model actually worth running?”

Tie ML metrics to:

  • Product metrics:
    • Conversion, revenue, churn, task completion, time saved.
  • Cost metrics:
    • Infra cost per 1K requests.
    • Storage/compute cost of feature pipelines.
    • Annotation/human review cost if applicable.

Track by model version and configuration:

  • “Model v3.2, using 512-dim embeddings and expensive reranker” vs
  • “Model v2.8, cheaper embeddings, no rerank.”

Only at this loop do you answer:
– “Do we keep, scale, or decommission this setup?”
– “Is the performance bump worth ongoing cost and complexity?”

Where teams get burned (failure modes + anti-patterns)

1. Treating offline metrics as the truth

Pattern: A team ships a model that’s +4% AUC offline. In prod:

  • CTR is flat or down.
  • Latency is up.
  • Tail users see worse results.

Why?

  • Offline data didn’t match online traffic.
  • Training labels don’t reflect current incentives or UI.
  • New model changed user behavior, invalidating historical assumptions.

Anti-pattern: Declaring success based on test-set metrics alone.

Mitigation:

  • Always run an A/B or shadow deployment.
  • Require a minimum detectable effect on a business metric before promoting.
  • Keep a regression guardrail: “Do not degrade key KPI by more than X%.”

2. “Fire and forget” deployment with no labels

Pattern: A text classification or LLM assistant is deployed. Nobody collects labeled feedback.

Consequences:

  • No way to know if quality is drifting.
  • Can’t justify upgrades or retrains.
  • Bugs in data or prompt silently accumulate.

Mitigation:

  • Decide how labels/proxies will be collected before launch:
    • Targeted annotation on a sample of predictions.
    • Lightweight user rating prompt for a fraction of sessions.
    • Heuristic labels (e.g., downstream explicit negative actions).
  • Budget for evaluation infra as part of the project, not “later.”

3. Feature pipeline drift and silent breakage

Real-world example:

  • A team computed a “30-day spend” feature using windowed aggregations.
  • A data engineer “optimized” the job to 7 days for cost reasons.
  • Fraud model behavior changed overnight; fraud losses spiked.
  • No explicit monitoring existed tying feature distribution changes to alerts.

Mitigation:

  • Treat feature pipelines like APIs:
    • Version them.
    • Document semantics.
    • Alert on schema and distribution changes.
  • Log both:
    • Raw inputs.
    • Final feature vectors with metadata (feature version, transform version).

4. Cost explosion from naïve “best model wins” logic

Example pattern:

  • Search team deploys two-stage ranking:
    • First-stage candidate generator using embeddings.
    • Second-stage reranker using a heavyweight model.
  • Initially limited to certain query classes; later quietly enabled broadly.
  • Infra bill doubles; SREs see latency creep at p95/p99.

Problem:

  • No per-query or per-feature cost accounting.
  • No clear policy on when to fall back to cheaper paths.

Mitigation:

  • Track cost and latency per request type and model path.
  • Implement a routing policy:
    • Cheap model for low-value or simple cases.
    • Expensive path only when expected benefit justifies it (e.g., queries above certain revenue thresholds or complexity).

Practical playbook (what to do in the next 7 days)

You don’t need a full “ML platform” to start. In one week, you can harden what you already have.

Day 1–2: Instrument the basics

  1. Add model-aware logging:

    • For each prediction, log:
      • Request ID, timestamp.
      • Model version / config.
      • Key input attributes (not raw PII when avoidable).
      • Prediction score / output.
    • For generative outputs:
      • Log response length, basic toxicity/PII checks, and flags.

  2. Connect infra metrics to model versions:

    • Tag metrics (latency, QPS, error rates) with model version or route.

Day 3: Define evaluation surfaces

  1. Pick 1–2 primary metrics that map to business value.

    • Example:
      • Recs: click-through and downstream add-to-cart.
      • Fraud: chargeback rate and false positive rate on high-value users.
      • LLM support bot: ticket deflection rate and CSAT for sampled cases.

  2. Define tolerance bands and SLO-like targets:

    • “False positive rate must not exceed X% on segment Y.”
    • “p95 latency under 300ms for 99.5% of requests.”

Day 4: Add lightweight drift checks

  1. Select ~10 most important features and outputs.

    • For each, compute:
      • Daily/weekly mean, std, and histogram.
    • Compare against a 30-day rolling baseline.

  2. Set simple alerts:

    • “If distribution shift for feature F exceeds threshold T, create an incident ticket.”
    • Start conservative and adjust; the goal is signal, not statistical perfection.

Day 5: Sample and label

  1. Stand up a small evaluation set from production traffic:
    • Random sample of predictions logged earlier.
    • Have humans (internal or vendors) label:
      –

Similar Posts