Your LLM Is a Data Leak Until Proven Otherwise

Why this matters this week

Three things have converged in the past few weeks that make “we’ll figure out AI governance later” an increasingly bad bet:

• Regulators are connecting AI systems to existing obligations, not waiting for bespoke “AI laws.”

  • SOC2 / ISO auditors are starting to ask concrete questions about LLM data flows, retention, and who can change prompts/policies.
  • Several regulators have clarified: if your model touches personal data, your existing privacy/security duties already apply.

• Vendors have quietly changed defaults on data retention, logging, and “model improvement” flags. Some are more conservative now, others less so, and very few customers have noticed or updated internal policies.

• Internal usage has crossed a threshold at many orgs:

  • From “a couple of pilot bots” to “10+ teams hitting LLMs in production flows.”
  • At that point, ad-hoc Slack rules and a wiki page don’t cut it. You need policy-as-code and real auditability, or you’ll be retrofitting under audit pressure.

If you ship production systems, the new question isn’t “Should we use AI?” It’s:

Can we explain, prove, and reproduce what our AI systems did with user data over the last 90 days?

For many teams, the honest answer today is “no.”

What’s actually changed (not the press release)

Ignoring the vendor hype, here’s what’s materially different for privacy and governance around AI right now:

1. LLM usage has moved into sensitive transaction paths

   • Not just “assistant for docs”; now:
     • Drafting customer emails with live account data.
     • Summarising support tickets with PII/PHI.
     • Flagging risky payments / content moderation.
   • That means: retention, lawful basis, and access control for model I/O are now core risk vectors, not side concerns.

2. Centralised logging is colliding with data minimisation

   • Standard practice: “log everything, forever, S3 is cheap.”
   • With LLMs, logs now routinely contain:
     • Raw prompts with secrets, credentials, IDs.
     • Generated content that may include inferred attributes.
   • SOC2 / ISO27001 expectations around log retention vs. privacy policies are starting to conflict: you can’t both “keep logs for 7 years” and “not retain personal data longer than needed” without segmentation and redaction.

3. Model risk is now more than “hallucinations”

   • Risk categories that actually matter for governance:
     • Data risk: training on, or fine-tuning with, regulated data.
     • Policy risk: models routing around your access control / authorisation boundaries.
     • Explainability & reproducibility risk: inability to show why a decision was made and under which policy/version.
   • Regulators and auditors don’t care what you call it; they will ask: “Show me where this decision came from and who could change the configuration.”

4. Policy is moving from slides to code

   • A noticeable pattern in the last quarter:
     • Mature teams encode rules like “no PII to external models” into gateways and CI checks.
     • Immature teams have Confluence pages plus a training session.
   • When something goes wrong, the only thing that counts is what the system enforced or logged, not what the slide deck said.

5. Audit scopes now explicitly include AI components

   • In recent SOC2 / ISO27001 work:
     • AI services are being treated as in-scope systems handling production data.
     • Expect requests for:
       • Data flow diagrams including LLMs.
       • Retention and deletion procedures for prompts, outputs, and fine-tuning data.
       • Change management for prompt templates and model configs.

How it works (simple mental model)

Use this as a mental model: treat your LLM stack like a new microservice tier with strict data contracts and policy guards.

Concretely, think in four layers:

1. Data classification layer (what is allowed where)

   • Define 3–4 data sensitivity levels:
     • Public
     • Internal
     • Confidential
     • Restricted (PII/PHI/secrets/regulated)
   • For each level, define:
     • Which models can see it (internal only vs. external SaaS).
     • How long it may be retained.
     • Whether it can be used for training/fine-tuning.
   • Outcome: a simple matrix you can enforce mechanically.

2. Policy-as-code layer (what must never happen)

   • Express constraints as code, not advice:
     • “Restricted data must not be sent to external LLM endpoints.”
     • “All LLM API calls must include a tenant ID and user ID.”
     • “Model configs and system prompts must be versioned and go through code review.”
   • Implement via:
     • API gateways / service mesh policies.
     • Linting / static checks on prompt templates and orchestration code.
     • CI/CD checks on LLM-related config changes.

3. Governance & logging layer (what actually happened)

   • Every LLM interaction creates governance events:
     • Who (user/service principal).
     • What (input data class, model, policy version).
     • Where (region, vendor).
     • When (timestamp).
   • Logs must be:
     • Queryable by tenant/user/time window.
     • Linked to incident response playbooks.
     • Retained per your data retention policies (not “forever by default”).

4. Lifecycle & risk layer (how it evolves)

   • New models / prompts are a change like new code:
     • Risk assessment: data access, purpose, potential impact.
     • Approval workflow: security, privacy, and data owners sign off.
     • Re-certification: periodic review of usage, drift, and access.
   • For fine-tuned or self-hosted models:
     • Track training datasets, data sources, and lineage.
     • Maintain a bill of materials (which data, which scripts, which base model).

If you treat LLMs as “magic APIs”, you get unbounded risk. If you treat them as a new microservice tier with strict contracts, you can fit them into existing SOC2/ISO structures with less drama.

Where teams get burned (failure modes + anti-patterns)

Patterns showing up repeatedly:

1. “Shadow fine-tunes” with forbidden data

   • Example: a team exports a CSV of support tickets (with emails + account IDs) to “improve the help bot.”
   • Anti-patterns:
     • No documented legal basis or DPA update.
     • CSV sits in personal cloud storage.
     • Fine-tune data is never deleted; model is shared widely internally.
   • Fallout: privacy/compliance scramble when a customer asks for deletion or regulator asks for processing records.

2. Logs as the real data lake

   • Example: central logging captures full prompts and responses, including secrets and PII, because “we might need it for debugging.”
   • Anti-patterns:
     • No field-level redaction.
     • No separate retention policies for LLM logs vs. regular logs.
   • Fallout: you technically have a large, poorly controlled PII repository; data subject access requests (DSARs) become near-impossible to fulfil.

3. Prompt changes bypassing change control

   • Example: product manager edits the system prompt in a config UI to “improve tone,” accidentally weakening safety guards.
   • Anti-patterns:
     • Prompts treated as content, not code.
     • No versioning, no approvals, no rollback plan.
   • Fallout: model starts exposing sensitive internal details or making previously blocked recommendations; no one can say exactly when the behaviour changed.

4. Vendor assumptions out of date

   • Example: security review done 12 months ago assumes:
     • “No prompts are used for training.”
     • “Data stored for 30 days.”
   • Vendor silently changes defaults or product tiers; retention is longer, or some features use data for improvement unless you opt out.
   • Fallout: you’re out of alignment with your own privacy notice and processor agreements without realising it.

5. No tenant isolation in orchestration

   • Example: a central “AI middleware” service proxies all LLM calls, but:
     • Tenant ID is optional.
     • Cache is global.
   • Fallout: responses may leak patterns or fragments from one tenant’s data to another, even if models are stateless, because your caching/store isn’t.

Practical playbook (what to do in the next 7 days)

You won’t “solve AI governance” in a week, but you can get from hand-wavy to defensible. Focus on actions that produce artefacts you can show to auditors and leadership.

1. Map your AI data flows (2–4 hours)

• Inventory where LLMs touch production-ish data:
  • Internal tools (Copilot-like, docs Q&A).
  • Customer-facing features (assistants, summarisation, scoring).
  • Back-office workflows (support triage, analysis).
• For each, capture:
  • Data sources (DB tables, logs, external APIs).
  • LLM provider / model / region.
  • Storage locations for prompts, outputs, intermediate artefacts.
• Outcome: a single diagram/Doc that can be reused in SOC2/ISO documentation.

2. Define a minimal classification & policy matrix (2 hours)

• Establish at least:
  • Restricted: PII/PHI, secrets, financial account details.
  • Confidential: internal docs, runbooks, non-public strategy.
  • Public/Internal: marketing, public docs.
• For each class, answer:
  • Can it go to external SaaS models?
  • Retention for:
    • Prompts/outputs.
    • Training/fine-tune data.
  • Who approves new use-cases?
• Write this in plain language first; then translate to policy-as-code later.

3. Lock down obvious vendor settings (1–3 hours)

For each LLM vendor you use:

• Review:
  • Data retention defaults.
  • “Use data to improve models” toggles.
  • Region controls.
• Align with your policy matrix:
  • For Restricted data: require no-train, minimal retention, region constraints where feasible.
• Document the current settings (screenshots are fine) and store them with your security/compliance artefacts.

4. Introduce some policy-as-code (1–2 days)

Pick one or two high-value controls and implement them in code:

• On the egress path (API gateway, custom middleware):
  • Mandatory metadata: tenant ID, user ID, data classification tag.
  • Block rule: if classification == Restricted and provider == external, reject the call.
• On the config path (prompts, model configs):