Your Models Are Lying to You in Production (You Just Don’t See It Yet)

---

Why this matters this week

Most teams are now “shipping ML,” but what’s actually in production often looks like this:

• Offline AUC: 0.93
• Production monitoring: 2 Prometheus counters and vibes
• Retraining: “whenever numbers look weird”
• Cost: “we’ll optimize later, infra is cheap”

Then reality hits:

• Infra bill quietly 3–5x’s because feature pipelines and embeddings are chatty and unbounded.
• Model performance silently degrades for months due to data drift.
• Incidents get misdiagnosed as “infra flakiness” when the real cause is bad features or stale models.
• Security/compliance flags pop up because some “temporary” logging of features contained PII and was never removed.

This week’s focus: applied machine learning in production — specifically evaluation, monitoring, drift, feature pipelines, and cost/perf trade-offs.

Not “fancy MLOps stack” discussion. Just: how to know if your model is doing what you think, at a cost you’re willing to pay, with failure modes you can detect.

---

What’s actually changed (not the press release)

What’s changed in the last 12–18 months for applied ML in production:

1. Feature pipelines got way more complex, way faster.

   • Widespread use of embeddings, vector stores, and cross-service feature aggregation.
   • “Just add another feature” now often means “add another RPC, transformation, and storage pattern.”

2. Feedback loops got longer and noisier.

   • More ML use cases where ground truth is delayed or ambiguous:
     • LLM-based assistants (no binary label, user satisfaction is fuzzy).
     • Long-horizon outcomes: churn, LTV, fraud discovered weeks later.
   • Standard “compute ROC once we get labels” is often too slow to catch drift.

3. Infra is cheap until it isn’t.

   • It’s easy to add:
     • Per-request feature recomputation.
     • High-cardinality metrics.
     • Full-fidelity logging of inputs/outputs.
   • At scale, you end up paying heavily for:
     • Network between services.
     • Storage & scans on logs/features.
     • GPU/CPU for over-complex models that don’t move the business metric.

4. Regulators and security teams woke up.

   • Feature logs and model outputs are now recognized as sensitive data.
   • You need real answers to:
     • Where is this data stored?
     • How long do we keep it?
     • Who can query it?
   • “We’ll clean it later” is no longer an acceptable data retention strategy.

5. Stakeholders expect A/B tests, not decks.

   • Execs are less impressed by “state-of-the-art” metrics.
   • They want: “Show me an experiment where this changed revenue, latency, or risk.”

Taken together, this means: offline metrics alone are basically a vanity exercise unless they’re backed by careful production evaluation and monitoring.

---

How it works (simple mental model)

Use this mental model for production ML systems:

Model = (Data → Features → Inference → Decision) + (Feedback → Update)

You need to monitor and control each arrow, not just the box labeled “model.”

1. Data → Features

Questions to ask:

• Where do your features come from? (raw sources, services, user input)
• What assumptions are baked in? (schemas, units, null handling, time windows)
• How often do they change without ML owners being notified?

Monitoring patterns:

• Schema and distribution checks at ingestion:
  • Validate types, ranges, and categorical values.
  • Track basic stats (mean, std, quantiles, top-k categories).
• Freshness checks:
  • Record “last updated” timestamps for feature tables.
  • Alert if windows are stale beyond X minutes/hours.

2. Features → Inference

This is the actual prediction step:

• With classical ML: a model server + feature fetch.
• With LLMs: prompt construction + model call + optional tools.

Monitoring patterns:

• Latency & error rates per model and per request type.
• Input validity checks:
  • Unexpected nulls, extreme values, malformed JSON, overly long texts.
• Shadow validation:
  • Run a simpler baseline model in parallel on a subset of traffic to see divergence.

3. Inference → Decision

Predictions don’t matter; decisions do:

• Thresholding, ranking, fallbacks, and rules wrap the model.
• Sometimes business logic dominates the outcome more than the model.

Monitoring patterns:

• Decision-level metrics:
  • What percent of traffic falls back to a rule?
  • How often do we override the model due to guardrails?
• End-to-end metrics:
  • CTR, conversion, fraud detection rate, time to resolution, etc.

4. Feedback → Update

This closes the loop:

• Ground truth labels (when available).
• Proxy metrics or human ratings (when labels are unclear).
• Retraining triggers, online learning, or manual model refreshes.

Monitoring patterns:

• Label arrival delay:
  • Distribution of time between prediction and label.
• Performance over event time, not only ingestion time:
  • Evaluate the model on data as of when it was predicted, not when labels arrived.

If any of these four segments isn’t monitored, you’ll misdiagnose outages and drift.

---

Where teams get burned (failure modes + anti-patterns)

A few anonymized but common patterns.

Failure mode #1: “Offline hero, production ghost”

• Team ships a model with great offline evaluation metrics.
• In production, business metrics don’t move, but no one can tell why.

Typical root causes:

• Training and serving features are not actually aligned (different joins, time windows).
• The population in production is different (geo, device, language skew).
• Decision layer (rules, thresholds) nullifies model improvements.

Anti-patterns:

• Treating the model as the only thing that changed.
• No production counterfactual logging (what would happen with the old model under current data).

Failure mode #2: Silent drift and “invisible incidents”

• Input data distribution changes gradually:
  • New product categories.
  • Different traffic mix (organic vs paid).
  • Seasonality or marketing campaigns.
• Error doesn’t spike; it just slowly increases.

This typically shows up as:

• Engineers firefighting “infra issues” because some downstream service is overloaded by unexpected shapes of traffic.
• Product owners losing trust: “the model feels off lately.”

Anti-patterns:

• Only monitoring point metrics (weekly AUC) rather than distribution shifts.
• No explicit canary deployments or gradual rollouts for model changes.

Failure mode #3: Cost explodes from feature creep

Example pattern:

• Initial model uses 10 simple features. Works fine.
• Over 18 months:
  • Add 30 more features, several requiring cross-service joins.
  • Introduce embeddings that require per-request encoding and vector store lookups.
• At scale:
  • P99 latency crosses SLO.
  • Cloud bill spikes (network + storage + vector operations).

Anti-patterns:

• No per-feature cost attribution (CPU, memory, IO).
• No regular feature importance + ablation runs to prune low-value features.

Failure mode #4: “Log everything” becomes a security incident

• To debug model issues, team logs:
  • Raw requests (including user-generated content and IDs).
  • Full feature vectors.
  • Model outputs.
• Months later, governance/legal asks:
  • Where is user X’s data?
  • How long do you keep it?
  • Is any of this used for training?

Anti-patterns:

• No data retention policy for logs/features.
• No classification of sensitive fields within ML data pipelines.

---

Practical playbook (what to do in the next 7 days)

Assuming you already have at least one model in production.

Day 1–2: Establish a minimal production eval spec

Define, in writing, for one key model:

1. Primary business metric (e.g., incremental revenue per 1k requests, fraud prevented, support resolution time).
2. Technical metrics:
   • For classification: calibration, precision/recall in key segments.
   • For regression: error by decile of prediction.
3. Latency & availability SLOs:
   • e.g., P95 < 150 ms, error rate < 0.1%.
4. Label/feedback source and delay:
   • Where labels come from.
   • Expected time-to-label distribution.

If you can’t describe these in 1 page, you don’t yet “own” the model in production.

Day 2–3: Add bare-minimum drift monitoring

Pick 5–10 highest-impact input features and:

• Track per-day:
  • Mean, std, min, max.
  • Histogram or quantiles.
  • Top-k categories with counts.
• Compute a simple population shift metric:
  • Even a basic KL divergence or PSI (Population Stability Index) is enough to flag “this looks different.”

For outputs:

• Track:
  • Distribution of scores.
  • Decision rates (approve/deny, recommend/not recommend).

Alert on sustained shifts, not one-off spikes.

Day 3–4: Implement a simple shadow or A/B path

For a single model:

• Route 1–5% of traffic to:
  • A baseline model, or
  • The previous version of the model.
• Log both predictions + decisions (without leaking PII).
• Compare:
  • Error (when labels arrive).
  • Business metric delta (if you can compute quickly enough).

You don’t need a huge experiment framework; a simple flag + extra column in logs often suffices initially.

Day 4–5: Instrument feature freshness and errors

For your feature pipeline:

• Add timestamps:
  • When the raw event happened.
  • When the feature was last computed.
• Track:
  • Percentage of requests with stale features beyond threshold.
  • Percentage of requests with missing or defaulted features.

Create dashboards and at least one alert for “feature freshness SLO violated.”

Day 5–6: Quick cost/perf sanity check

Collect:

• QPS and P95/P99 latency per model.
• Per-request CPU/GPU usage (or at least relative compute).
• Calls per request to:
  • External services (including vector DBs).
  • Data stores.

Ask:

• Which features require the most external calls or encoding time?
• If we removed N most expensive features, how much performance would we lose?

If you