Stop Treating AI Governance as a Deck Problem

Why this matters this week

In the last few weeks, three things have converged in a way that matters if you’re responsible for production systems:

• Vendors quietly updating terms around data retention and training use for customer inputs.
• Internal security teams starting to ask pointed questions like:
  “Show me exactly which models can see PHI and how long it’s retained.”
• Early SOC2 / ISO27001 audits explicitly calling out AI usage as in-scope, not “just another SaaS.”

If you’re using LLMs in prod—whether via API, self-hosted, or a vendor platform—you now have a governance surface area you didn’t sign up for:

• Where does your data actually go?
• How long is it stored?
• Which models see what?
• Can you prove that to an auditor (or regulator) in six months?

This is no longer a “we’ll clean it up later” problem. The longer you run without explicit AI governance, the more un-auditable behavior you accumulate: prompt logs in random S3 buckets, embeddings with sensitive data, unknown vendors in the path.

The rest of this post is about mechanisms, not “AI principles”: data retention, model risk, auditability, SOC2/ISO alignment, and policy-as-code you can actually implement.

---

What’s actually changed (not the press release)

The environment shifted in a few concrete ways:

1. AI is now explicitly in scope for audits

Real audits in the last 6–9 months are asking:

• “List all third parties that process customer data, including AI model providers.”
• “Show your data retention policy as it applies to model inputs/outputs.”
• “Show logging and access controls for AI systems in production.”
• “Describe how you prevent sensitive data from being used to train external models.”

If your answers are “we use a managed LLM and we trust them,” you’re going to have a bad time.

2. Vendor defaults aren’t aligned with your risk

Many AI vendors:

• Retain logs for debugging or abuse detection.
• Use customer-provided data to improve models unless you explicitly opt out.
• Store or replicate data across regions you don’t expect.

The marketing page might say “enterprise-grade” and “no training,” while the actual data processing addendum has carve‑outs. Your security team will (correctly) treat the contract, not the blog post, as truth.

3. Model usage has outpaced governance

Common pattern across teams:

• 3–6 different internal “pilot” projects using various APIs.
• Shadow usage of public AI tools by support, sales, and operations.
• No central inventory of:
  • Which models are used.
  • What data classes they see (PII, PHI, source code, financials).
  • Where logs and embeddings are stored.

Governance is happening by tribal knowledge and Slack messages. That works until you get:

• A data subject access request (DSAR).
• A security questionnaire from a large customer.
• A breach or incident involving model logs.

4. Regulators are moving from “watching” to “asking”

You don’t need a dedicated AI law to feel this; existing frameworks (GDPR, CCPA, HIPAA, GLBA, etc.) already apply:

• “Explainability” → Ability to trace automated decisions back to data flows.
• “Data minimization & purpose limitation” → Don’t shove everything into a model because it’s convenient.
• “Right to erasure” → Can you remove user data from prompt logs and vector stores?

You don’t have to predict every future rule, but you do need traceable, enforceable controls now.

---

How it works (simple mental model)

Use this mental model for AI privacy & governance: Three layers of control.

1. Data Surface
   What data can touch a model at all?

   • Inputs: prompts, documents, events, database exports.
   • Outputs: completions, embeddings, intermediate artifacts (e.g., chain-of-thought logs if you store them).
   • Metadata: user IDs, IPs, session tokens, customer IDs.

   Questions:

   • Which data classifications are allowed (public, internal, confidential, restricted)?
   • Where is each class allowed to go (internal models vs. external APIs)?

2. Model Boundary
   What happens inside and around the model?

   • Model type:
     • External API (e.g., hosted LLM).
     • Self-hosted open-weight model.
     • Vendor “AI platform” with its own plugins and tools.
   • Core concerns:
     • Is data used for training or fine-tuning?
     • Are prompts/completions logged? By whom?
     • Do plugins/tools execute arbitrary calls (e.g., to internal APIs)?

   Questions:

   • Can you configure per‑model policies (retention, redaction, allowed tools)?
   • Can you show which model handled which request six months ago?

3. Control Plane
   How do you enforce and audit behavior?

   Mechanically, this looks like:

   • Central policy: “No production PII to external models,” “Logs retained 30 days,” etc.
   • Policy-as-code enforcement:
     • At ingestion (redaction, blocking).
     • At routing (only certain models per data class).
     • At storage (retention policies, encryption, access control).
   • Observability:
     • Structured logging of model calls.
     • Per‑tenant or per‑user attribution.
     • Policy decisions logged (allow/deny/mutate with reasons).

In other words: treat your AI stack as a regulated data pipeline, not a smart function.

---

Where teams get burned (failure modes + anti-patterns)

These are patterns from real deployments (details anonymized).

Failure mode 1: “We’ll centralize it later”

Pattern:

• Multiple teams ship AI features independently.
• Each uses a different LLM vendor or library.
• Logging is local to each team’s stack (e.g., app logs with prompts in plain text).

Impact:

• You can’t answer: “Show me all the places where PHI might be in a prompt log.”
• Users of one feature get stronger guarantees than others, and you don’t even know which.
• Incident response is guesswork; data subject rights are almost impossible to honor.

Anti-pattern indicators:

• No single YAML/JSON config or code module that defines model usage policies.
• Security reviews are per‑project, not platform‑level.
• “Governance” lives in Confluence pages, not in CI/CD or code.

Failure mode 2: Retention by accident

Pattern:

• You send prompts to an external LLM.
• You also stream them through an observability or logging pipeline.
• Logs end up in:
  • A third-party log SaaS.
  • An S3 bucket with weak access controls.
  • A BI warehouse where analysts have broad access.

Impact:

• Whatever protections you negotiated with your LLM vendor are bypassed by your own logging.
• Sensitive completions (e.g., summarised support tickets) sit in logs indefinitely.
• You can’t confidently say where sensitive data is stored or for how long.

Anti-pattern indicators:

• No classification tags on prompts/completions in logs.
• No retention policy attached to AI-related streams.
• Developers copy-paste prompts into logs for debugging without redaction.

Failure mode 3: “PII is banned” as the only control

Pattern:

• Leadership issues a blanket “no PII in AI tools” directive.
• Implementation is a slide deck and maybe a training session.
• Engineers still need to solve real problems that inherently involve PII (support triage, risk scoring, KYC, etc.).

Impact:

• Shadow usage: people paste data into whatever tool gets their job done.
• Security believes risk is low; reality is the opposite.
• The org avoids building governed, auditable flows and ends up worse off.

Anti-pattern indicators:

• Policies exist only as documents, not as code.
• No safe, governed alternative for legitimate PII use cases.
• Security relies heavily on “awareness” instead of guardrails.

Failure mode 4: Unbounded vector stores

Pattern:

• Team stands up a vector database.
• They dump mixed data into it: docs, tickets, chat logs, code.
• No per-field redaction or classification; just “make it searchable.”

Impact:

• The vector store becomes an unbounded, poorly governed data lake:
  • Sensitive content from one tenant leaks to another via retrieval.
  • You cannot fulfill deletion or retention requirements (embeddings are opaque).
• It’s hard to even enumerate which source systems are represented.

Anti-pattern indicators:

• No mapping from embedding IDs back to original records with metadata.
• No retention or deletion pipeline—data only accumulates.
• Retrieval tests show cross-tenant bleed, and nobody owns fixing it.

---

Practical playbook (what to do in the next 7 days)

You won’t build a perfect AI governance program in a week, but you can establish a backbone.

Day 1–2: Inventory and classification

1. Create a quick AI use inventory (don’t over-engineer):

   • List every product or internal tool using:
     • External AI APIs.
     • Self-hosted models.
     • AI “platforms” used by business teams.
   • For each, capture:
     • Data sources (CRM, ticketing, logs, DBs, uploads).
     • Data classes handled (public, internal, PII, PHI, financial, source code).
     • Providers and regions used.

2. Mark the top 3 risky flows:

   • Any use of:
     • Customer PII/PHI.
     • Production incident data.
     • Legal, HR, or financial documents.

These become your initial governance targets.

Day 3: Define baseline policies (plain language first)

Write a 1–2 page AI Data Policy that you can turn into code. Cover:

• Data classes:
  • What counts as “restricted” vs “confidential” vs “internal.”
• Allowed destinations:
  • E.g., “Restricted data must stay on self-hosted models or vendor X with DPA Y.”
• Retention:
  • Prompt/completion logs for each data class (e.g., 30/90/365 days).
• Training restrictions:
  • “No customer data may be used for third-party training; fine-tuning must be opt-in per customer.”
• Access controls:
  • Who can see raw prompts/completions? (e.g., SREs vs. all engineers vs. no humans.)

Make this the canonical source for the next step; don’t aim for perfection, aim for something you’re willing to enforce.

Day 4–5: Translate into policy-as-code

Pick one enforcement point first. Common options:

• Gateway / proxy for model calls:
  Wrap all outbound model calls through a service (or shared