Policy-as-Code or Policy-as-PDF? Getting Real About AI Governance

Why this matters this week

Several things are converging right now:

• Legal/compliance is waking up to the fact that “we put a wrapper around GPT” doesn’t answer questions about data retention, cross-border data transfer, or model risk.
• SOC 2 and ISO 27001 auditors are starting to ask concrete questions about AI use, not just “do you have a policy?”
• Cost reviews are revealing silent sprawl: shadow AI tools, unmanaged PII flowing into prompts, and unbounded log retention in vector stores and observability systems.

If you’re running production systems, the core problem is simple:

You’ve probably added at least one powerful non-deterministic subsystem (LLMs, embeddings, RAG, fine-tunes) without giving it the same rigor you apply to databases, queues, and source control.

This week’s topic: how to treat privacy and AI governance as an engineering problem, not an aspirational slide deck. Specifically:

• Data retention you can actually prove, not just promise.
• Model risk you can articulate, not hand-wave.
• Auditability that doesn’t rely on Slack archaeology.
• Policy-as-code that devs don’t bypass on day two.

What’s actually changed (not the press release)

What has changed in the last 6–12 months, at a “you’ll feel this in your architecture” level:

1. Model vendors added more knobs

   • Tenant-level data retention settings.
   • Region pinning and data residency options.
   • Finer-grained logging controls (request/response log toggles).
   • “No training on your data” flags that actually hook into their pipelines.
     Trade-off: more configuration surface area to get wrong.

2. Infra teams started treating LLM traffic as a first-class data flow

   • Proxies and gateways specifically for LLM calls.
   • Central prompt/response logging (often over-logging by default).
   • Token-based cost and usage monitoring.
     Trade-off: more observability → more sensitive data in logs.

3. Compliance frameworks began to include AI sections in practice, if not in name

   • SOC 2 and ISO auditors are now asking:
     • How do you control which SaaS AI tools employees use?
     • How do you prevent PII from leaving your boundary?
     • How do you review/approve models and prompts for risky behavior?
   • They’re not standardized yet, but the questions are becoming repeatable.

4. Regulators are starting to care about “automated decisioning” again, but now with LLMs in the loop

   • Anything touching credit, employment, healthcare, or kids: expect scrutiny.
   • “We had a human in the loop” is no longer a get-out-of-jail-free card if the human just clicks “approve” on AI output.

What hasn’t changed:

• No widely accepted, concrete standard for AI-specific SOC 2/ISO controls.
• No turnkey “AI data governance” platform that isn’t mostly marketing.
• No model vendor giving you full transparency into their internal retention and training processes.

You still have to design and implement your own governance mechanisms.

How it works (simple mental model)

Use this mental model for AI privacy & governance:

Every LLM integration is a new untrusted data sink + a new decision engine in your architecture. Govern both.

Think in four layers:

1. Data classification & routing

   • Label: Which data must never leave your trust boundary? (e.g., raw card numbers, PHI, minors’ data)
   • Route: Based on labels, decide:
     • Not allowed to LLM at all.
     • Only to self-hosted / VPC model.
     • OK to send to external vendor with DPA in place.
   • Mechanism: middleware that inspects requests (metadata, not full content, where possible) and enforces routing and blocking.

2. Retention & redaction contracts

   • Define per system:
     • How long to keep prompts, responses, embeddings, logs.
     • Where they are stored (region, provider).
     • What fields are redacted/hashed at each boundary.
   • Mechanism: config-backed, versioned policies that apply to:
     • App logs
     • Vector DBs
     • LLM vendor logs (where controllable)
     • Analytics tooling

3. Model risk tiers

   • Classify use cases, not models:
     • Tier 0: Non-prod, experiments. No customer data.
     • Tier 1: Low-risk internal assist (e.g., code gen, summarizing public docs).
     • Tier 2: Customer-facing but advisory (draft emails, search).
     • Tier 3: Customer-impacting decisions (eligibility, pricing, approvals).
   • For each tier, define:
     • Required human oversight.
     • Required logging and explanations.
     • Required approval path before deployment.

4. Policy-as-code enforcement points
   Implement policies where engineers can’t “forget” them:

   • Infrastructure:
     • Terraform modules that spin up LLM gateways with mandatory logging/PII filtering.
   • Application:
     • Shared SDK that wraps all LLM calls, with policy hooks.
   • CI/CD:
     • Checks in pipelines that block promotion of:
       • LLM calls to disallowed vendors.
       • Missing or misclassified risk tier tags for new AI features.

This keeps you out of the “15-page AI policy in Confluence nobody reads” trap. The PDF is for auditors; the code is what enforces it.

Where teams get burned (failure modes + anti-patterns)

Some recurring failure modes from real-world patterns:

1. Logging everything “for debugging”

Pattern:
– Team builds an AI feature; to debug, they log:
– Full user prompts
– Full model responses
– User IDs, sometimes email and org
– Six months later, you have a shadow data lake of sensitive info with:
– No retention policy
– No access controls beyond “prod logs” permissions

Fallout:
– Incident response becomes a nightmare: unknown copies, unbounded retention.
– Auditors find: “You say 30-day retention, but these logs go back 18 months.”

Mitigation:
– Never log raw content by default.
– Explicitly define:
– A redacted log (for analytics).
– A short-lived detailed log (for debugging) with strict TTL and access.

2. Vector store = forever store

Pattern:
– Company builds RAG on top of a vector DB.
– Ingested content includes:
– Customer documents
– Internal tickets (potentially with PII)
– Vectors never get deleted when:
– Customer churns
– Data subject requests erasure
– Source content is updated or removed

Fallout:
– Actual retention is “until someone manually runs a cleanup job,” not what your policy says.
– “Right to be forgotten” cannot be honored in practice.

Mitigation:
– Make vector embeddings derivative, not primary:
– Keep a canonical record with explicit lifecycle.
– Link each embedding to that record via ID, and delete on source deletion.
– Build delete-by-customer and delete-by-source routines as first-class operations.

3. Shadow AI tools in the business

Pattern:
– Sales and support teams adopt third-party AI tools on their own.
– They paste:
– Customer contracts
– Support transcripts
– Internal roadmap docs

Fallout:
– No DPA, no data residency controls.
– Unknown third-party retention and training behavior.
– Security review discovers 10+ active vendors, none in your vendor inventory.

Mitigation:
– Provide a blessed alternative:
– An internal AI assistant with clear data handling.
– Simple request process for new tools with a fast SLA.
– Block outbound access to known risky tools at the network/proxy layer where possible.

4. No versioning of prompts or guardrails

Pattern:
– Prompts live in app code or config, updated frequently.
– No migration/versioning tied to model outputs.
– Investigating an incident: you cannot reconstruct what the model was told at the time.

Fallout:
– Impossible to perform root-cause on bad model decisions.
– Weak story to regulators: “we think the prompt was roughly…”

Mitigation:
– Treat prompts and safety rules like code:
– Version them.
– Store IDs in logs alongside requests and responses.
– Require PR review for changes to prompts in high-risk flows.

Practical playbook (what to do in the next 7 days)

This is a seven-day, low-theory, high-impact checklist you can realistically execute.

Day 1–2: Inventory and classify

1. Inventory all AI usage paths

   • Ask each team: “Where do you call an LLM or embeddings API?”
   • Look for:
     • Direct vendor calls (OpenAI, Anthropic, etc.).
     • Indirect usage via SDKs, analytics tooling, or SaaS features.
   • Capture:
     • Endpoint called
     • Data types sent (free text, logs, PII, code, etc.)
     • Whether it is customer-facing or internal.

2. Assign model risk tiers (0–3) per use case

   • Be explicit about:
     • Which ones are making or influencing customer-impacting decisions.
     • Which ones touch PII, PHI, or regulated data.

Day 3: Retention & logging reality check

1. For each AI path, answer:

   • How long are:
     • Prompts and responses kept?
     • Derivative data (embeddings, features) kept?
     • Logs in your logging platform kept?
   • Where are they stored? (region, provider)
   • Who can access them?

2. Implement immediate guardrails where trivial:

   • Turn off vendor-side logging where possible for sensitive flows.
   • Set or tighten log retention on your logging platform and vector DBs.
   • Add a basic PII scrubber for logs that captures:
     • Emails, phone numbers, card-like patterns, SSNs (if applicable).

Day 4–5: Introduce policy-as-code at the edges

1. Create a common LLM client or gateway (if you don’t have one)

   • Even a thin internal SDK is better than every service calling vendors directly.
   • SDK responsibilities:
     • Inject standard headers/tags (service, environment, risk tier).
     • Route requests based on data classification (e.g., internal vs. external model).
     • Redact or hash known sensitive fields.

2. Encode minimal policies as code:

   • Example policies:
     • “Tier 3 uses only internal/VPC models.”
     • “Requests with contains_pii=true cannot call vendor X.”
     • “All Tier 2+ requests must log prompt