Your LLM Stack Is Leaking: A Pragmatic Guide to Privacy & Governance

Bygaz

A dimly lit modern data center with glowing server racks connected by bright, branching data streams, overlaid by translucent gridlines and lock icons, viewed from a high angle, cinematic wide shot, cool blue tones with sharp contrast

Why this matters this week

The last 6–8 weeks have had a quiet but important shift: legal, security, and audit teams have moved from “What is this AI thing?” to “Show me your data map and retention policies.”

Three drivers:

  • New internal pressure:
    • SOC 2 and ISO 27001 auditors are starting to ask explicit questions about AI systems.
    • Data protection officers are noticing LLM logs with PII and no retention policy.
  • Vendor behavior changes:
    • Foundation model providers are tightening or clarifying their privacy and data retention terms, but also shipping more “observability” features that create new log surfaces.
  • Scale effects finally hitting:
    • “Experiment” LLM endpoints are now in production paths.
    • Shadow usage (prompting prod data into chat assistants) is getting discovered in access logs.

If you run production systems, the risk is not “AI will become sentient,” it’s:

  • Silent data exfiltration into third-party systems via prompts and tool calls
  • Inability to answer basic questions from auditors:
    • “Where can user data land in your AI stack?”
    • “How long is it retained, and how can you prove that?”
  • Model behavior you cannot reconstruct or justify when an incident hits

This post is a concrete playbook for privacy, data retention, and AI governance that aligns with SOC 2 / ISO without turning your team into compliance clerks.

What’s actually changed (not the press release)

Ignoring the marketing around “trustworthy AI,” here’s what has materially changed for engineering teams.

1. The data surface area has exploded

Previously:

  • You knew where PII lived: app DBs, data warehouse, logs, backups.

Now, you have:

  • Prompt logs (often shipped to third-party observability tools)
  • Model provider logs (requests, responses, tool traces)
  • Vector databases holding embeddings of semi-structured user data
  • Fine-tuning datasets stored separately from core infra
  • Feature stores or intermediate caches used in RAG pipelines

Each of these can contain:

  • Direct PII (names, emails, phone numbers)
  • Highly identifying quasi-identifiers (search queries, support transcripts)
  • Sensitive derived attributes (inferred intent, risk scores, internal decisions)

2. Model vendors are making stronger (but narrower) commitments

Most major providers now say some variant of:

  • “We don’t train on your data by default”
  • “You can request log deletion or set retention”
  • “We support data residency in region X”

The fine print:

  • Telemetry, safety logs, and abuse logs are often excluded from the strictest retention guarantees.
  • Some providers differentiate between “API” and “hosted app” (e.g., chat UI) in their training/retention policies.
  • Retention defaults can be provider- and even model-specific, and change over time.

You can’t rely on a single slide from a sales deck. You need policy-as-code controls in your own stack.

3. Regulators and auditors have updated their question sets

  • SOC 2 / ISO 27001 controls around:
    • Data minimization
    • Data retention & deletion
    • Third-party risk management
    • Change management and approval

…are now being applied explicitly to:

  • Prompt pipelines
  • LLM orchestration layers
  • Evaluation datasets
  • Internal chatbots integrated with production systems

You’re not being asked “Do you use AI?” but “Where does customer data touch AI systems, and what is your control framework?”

How it works (simple mental model)

Use this as a sanity-check model for privacy & governance in LLM systems:

1. Three “planes” of an LLM system

  1. Control plane – how you configure and govern:
    • Policy-as-code (who can call what, with which data)
    • RBAC for prompts, tools, and datasets
    • Audit logs of configuration changes
  2. Data plane – what flows at runtime:
    • Inputs (prompts, tool parameters)
    • Intermediate artifacts (embeddings, retrieved docs)
    • Outputs (model responses, tool results)
  3. Evidence plane – what you keep to prove behavior:
    • Redacted logs
    • Evaluation runs & metrics
    • Incident records and remediation steps

Privacy and governability come from clean separation and explicit contracts between these planes.

2. Four policies you must make explicit

You need clear, documented stances (even if v1 is simple) on:

  1. Data retention

    • How long do you keep:
      • Raw prompts and responses?
      • Embeddings / vector store content?
      • Fine-tuning datasets?
    • Where are these configured (code vs config vs vendor UI)?

  2. Data classification

    • What classes of data are allowed into:
      • External model APIs?
      • Internal/self-hosted models only?
    • How is this enforced (e.g., classifiers, regex, schema-level controls)?

  3. Model risk tiers

    • Tier 0: Non-sensitive, no user data (e.g., doc rewriting with test content)
    • Tier 1: Low-sensitivity user content (support tickets)
    • Tier 2: PII/financial/health data
    • For each tier: which models, which vendors, which regions, which logging policies

  4. Access & oversight

    • Who can:
      • See raw prompts and responses?
      • View embeddings / RAG source documents?
      • Change model routing or config in prod?
    • How are changes reviewed (PRs, change advisory, approvals)?

3. Governance as “policy at boundaries”

Instead of abstract “AI ethics,” focus on four boundaries:

  • User → App: what is allowed into prompts?
  • App → Model: which fields enter the model, under which data class?
  • Model → Tools/Data: what tools can the model call on which datasets?
  • Model → Logs: what is logged, how is it redacted, and how long is it kept?

If you define and enforce policies at these four edges, you get 80% of practical AI governance.

Where teams get burned (failure modes + anti-patterns)

Failure mode 1: “Observability” becomes an uncontrolled data lake

Pattern:

  • Team adds an LLM observability platform or “just log everything to our existing tool.”
  • Full prompts and responses with PII, secrets, and internal docs are shipped and stored indefinitely.
  • Months later, an audit or security review discovers a second, poorly protected copy of sensitive data.

Anti-pattern flags:

  • JSON blobs of full chat history in logs “for debugging”
  • No structured separation of metadata (which you can keep) vs content (which you should trim/redact)
  • No retention config on the observability side

Failure mode 2: Vector stores treated as “not really data”

Pattern:

  • RAG system ingests customer contracts, support chats, or financial records into a vector database.
  • Engineers assume “it’s embeddings, so it’s safe.”
  • But:
    • The raw documents are stored alongside or accessible via IDs.
    • The embeddings themselves may be linkable to individuals via IDs or metadata.

Anti-pattern flags:

  • No DSR (data subject request) story for “delete or export my data” across embeddings and raw docs
  • Same retention as primary DB but with much weaker access controls
  • Backups and snapshots of vector DBs handled as an afterthought

Failure mode 3: Policy in slides, not in code

Pattern:

  • Security and legal draft AI usage policies.
  • Engineering nods, merges a wiki page, and continues shipping.
  • Business pressure leads to one-off exceptions and “temporary” bypasses.

Anti-pattern flags:

  • No automated checks enforcing:
    • Which endpoints can be called from which services
    • Which data fields can cross the model boundary
  • Manual approvals only; nothing in CI/CD or runtime

Failure mode 4: Model selection without data path thinking

Pattern:

  • Teams pick a model (external or self-hosted) purely on cost/quality benchmarks.
  • They ignore:
    • Data residency guarantees
    • Tenant isolation
    • Logging defaults
  • Later, they discover sensitive workloads went through a general multi-tenant SaaS endpoint with long retention.

Anti-pattern flags:

  • Same model/vendor for both Tier 0 and Tier 2 workloads
  • No environment-level separation (e.g., dev and prod using the same external project with shared logs)

Real example patterns

  1. Customer support bot

    • Multi-region SaaS, LLM-powered summarization of tickets.
    • Governance gap: PII-laden support transcripts logged in vendor observability tool with 18-month retention.
    • Impact: Painful remediation effort to delete historic data across multiple vendors and regions.

  2. Internal “ask the docs” bot

    • Jira, Confluence, and Drive ingested into vector store; accessible company-wide.
    • Governance gap: HR and legal documents accidentally indexed and exposed via semantic search.
    • Impact: Emergency ACL refactor and a rewrite to add document-level authorization checks.

  3. Risk scoring assistant

    • LLM used to suggest manual review decisions on high-risk transactions.
    • Governance gap: Model prompt and outputs not logged in a way that supports dispute resolution.
    • Impact: When a customer contested a decision, the team couldn’t reconstruct why the model suggested it.

Practical playbook (what to do in the next 7 days)

This assumes you already have at least one LLM system touching real data.

Day 1–2: Map the data flows (narrow but deep)

Pick one high-risk LLM use case (support bot, internal QA, risk scoring) and:

  • Draw a diagram that includes:
    • Sources: where data comes from (DBs, warehouses, APIs)
    • Destinations: LLM APIs, vector stores, observability tools, fine-tuning buckets
    • Paths: which services talk to which vendors in which regions
  • For each hop, annotate:
    • Data classes (PII, financial, health, internal-only, public)
    • Retention today (guess is fine; you’ll refine)
    • Access controls (who can see it, including vendors)

Deliverable: a 1-page diagram and table. This is your baseline for auditors and for fixing things.

Day 3: Decide and document your tiering & retention

Create v1 of two simple documents (even if just in a repo README):

  1. Model risk tiering

    • Tier 0 / 1 / 2 definitions (by data sensitivity)
    • Allowed models/vendors per tier
    • Required controls per tier (logging, redaction, approvals)

  2. Data retention matrix
    For the selected use case, fill out:

    • L

Similar Posts