Your RPA Bots Are Rotting: A Pragmatic Guide to AI Automation That Survives Reality

Bygaz

Why this matters right now

Most companies already learned the hard way that:

  • Classic RPA breaks every time the UI team sneezes.
  • Hand-built workflow automations explode in complexity as edge cases accumulate.
  • “Let’s just put a human in the loop” quietly becomes “we built a parallel manual process.”

AI automation (agents, copilots, workflow orchestration) is the first thing that can plausibly bend this curve for real businesses:

  • It can operate on messy inputs (emails, PDFs, logs, docs) without a static schema.
  • It can generalize across small variations instead of needing new rules for every case.
  • It can explain its own steps (to a degree), which makes observability and control possible.

The risk is obvious: you’re swapping brittle RPA for opaque “agents” that can be wrong, expensive, and hard to debug.

If you’re responsible for production systems and real SLAs, the question isn’t “should we use AI agents?” The question is:

Where does AI automation belong in the stack, and how do we keep it from turning into an un-auditable Rube Goldberg machine?

This post is about mechanisms, not slogans: how these systems actually work, where teams are getting burned, and what you can do in the next week that’s concretely useful.

What’s actually changed (not the press release)

Three real shifts matter for automation in businesses. Everything else is marketing.

1. Models can now reliably interpret and route messy work

In 2019, a typical RPA bot needed:

  • fixed UI selectors,
  • fixed screen layouts,
  • unchanging process steps.

Now large language models can:

  • Parse unstructured emails and map them to canonical intents.
  • Extract entities from arbitrary invoices, contracts, and logs.
  • Decide which of several tools / APIs to call based on instructions and context.

That doesn’t mean they’re perfect; it means they’re good enough to front-load routing and normalization, which used to be the brittle part of automation.

2. Tooling has made “agent as orchestrator” practical

You no longer have to hand-roll:

  • “If this, call that microservice, then transform JSON, then update that system.”

LLM-based agents can be the decision layer that picks which tools to use and in what order, while your existing APIs do the real work.

Key change: the LLM is not “doing the work,” it’s planning and calling tools.

This is a big shift from RPA:

  • RPA tried to automate the same UI humans use.
  • AI agents can use internal APIs, queues, and services, i.e., the same interfaces your backend uses.

3. Observability and control patterns are emerging

Early agents were black boxes. Now:

  • You can log each “thought” and tool call as structured events.
  • You can run shadow mode (agent suggests, human executes).
  • You can enforce guardrails (hard constraints, policy checks, approval steps).

This moves AI automation out of “experiment” territory and into something that can be audited, rate-limited, and governed like any other critical system.

How it works (simple mental model)

You can model most practical AI automation stacks as a 4-layer system:

  1. Interface layer – where work arrives
  2. Brain layer – LLMs doing planning, understanding, and decisions
  3. Muscle layer – deterministic tools, APIs, existing services
  4. Governance layer – policies, limits, auditing

1. Interface layer

Sources of work:

  • Email inboxes
  • Support chat / ticketing systems
  • Internal dashboards
  • Webhooks from external systems

Job: normalize inputs and hand them to the “brain” with enough context:

  • raw_input: the original text/email/document
  • metadata: user, channel, timestamps
  • prior_state: related tickets, customer history, etc.

2. Brain layer (agents, copilots, planners)

This is where LLMs live. They do three main things:

  • Interpretation
    “What is this?” → classify intent, extract entities, determine required outcome.

  • Planning
    Break work into steps:
    validate request → lookup account → compute refund → log decision → send email

  • Decision-making
    Which tool(s) to call? With what arguments? In what sequence? When to escalate?

In code, think:

pseudo
loop:
observe(state, new_input)
decide(next_action)
if next_action == "call_tool":
tool_result = tools.call(...)
state.update(tool_result)
elif next_action == "ask_human":
enqueue_for_approval(...)
break
elif next_action == "finish":
return final_result

This is what people market as “agents.”

3. Muscle layer (tools, services, RPA remnants)

These are deterministic capabilities:

  • Internal microservices
  • SaaS APIs (CRM, ERP, ticketing, billing)
  • Vector search (for knowledge retrieval)
  • In some cases, legacy RPA scripts behind an API

Key property: they return predictable outputs and have known side effects.

The LLM should never be allowed to “improvise” side effects; it should only compose these deterministic pieces.

4. Governance layer

Wrap everything with:

  • Policies – what the system may not do (issue refunds > $X, modify PII, etc.).
  • Rate limits – per user, per tool, per tenant.
  • Approvals – insert humans for specific high-risk transitions.
  • Logging – every decision, tool call, and final action is traceable.

This is where AI automation starts to look like any other production service you’d be comfortable owning.

Where teams get burned (failure modes + anti-patterns)

Patterns from real deployments in e-commerce, SaaS, and logistics.

Failure mode 1: “We replaced humans” before we understood the work

Teams:

  • Wire an LLM agent directly into production workflows.
  • Let it take actions in core systems (refunds, config changes, account updates).
  • Figure out edge cases after customers complain.

Symptoms:

  • Quiet data corruption.
  • Inconsistent decisions that erode trust.
  • Surprise costs due to long reasoning chains and retries.

Anti-pattern: agent as autonomous operator from day one.

Better: agent as co-pilot / recommender with forced review for a while.

Failure mode 2: Over-delegation to the LLM

Teams:

  • Ask the model to both plan and execute complex workflows and do domain-specific logic.
  • Embed rules in prompt text instead of code.
  • Rely on “it got it right in the demo” as validation.

Example: A logistics company had an agent that:

  • Parsed free-form delivery change requests.
  • Recomputed optimal delivery dates and fees inside the LLM.
  • Updated orders directly via an API.

Under unusual holiday constraints, the model invented pricing rules that sounded plausible but were wrong, leading to under-billing.

Anti-pattern: LLM as business logic engine.

Better:

  • LLM handles interpretation + tool selection.
  • Tools/microservices encode business rules and calculations.

Failure mode 3: RPA mindset transplanted to AI

Teams:

  • Model their AI automation as “screen-clicking bots” with LLMs just scraping text.
  • Leave critical logic in the UI layer: XPaths, CSS selectors, brittle DOM assumptions.
  • Never build internal APIs, just automate UIs.

Result:

  • AI inherits all the fragility of RPA plus new failure modes (hallucination, context loss).
  • When the UI changes, the agent fails in non-obvious ways.

Anti-pattern: agent puppeteering UIs as the primary approach.

Better:

  • Use automation projects as an excuse to build minimal internal APIs to key operations.
  • Let the agent call those APIs, not the UI.

Failure mode 4: No explicit reliability target

Teams:

  • Hand-wave about “AI will occasionally be wrong.”
  • Don’t define acceptable failure modes or measurable error rates.
  • Don’t distinguish “we got the answer wrong” from “we made an unsafe change.”

Example: SaaS support automation:

  • Agent auto-responds to billing inquiries.
  • No separation between “uncertain answer” and “confident answer.”
  • Customer gets an incorrect promise in writing; legal risk ensues.

Anti-pattern: heuristic quality (“seems good enough”).

Better:

  • Define explicit SLOs:
    e.g., “<1% incorrect high-impact actions, <5% incorrect low-impact responses.”
  • Route low-confidence cases to humans.

Practical playbook (what to do in the next 7 days)

You don’t need a platform migration to start. You do need discipline.

Day 1–2: Pick one narrow, high-friction workflow

Criteria:

  • Text-heavy, repetitive, semi-structured.
  • Today handled by humans or brittle scripts.
  • Low to moderate blast radius if wrong (not “wire money,” more “draft response”).

Examples:

  1. B2B SaaS support triage
    Classify tickets, extract key entities, attach relevant docs, suggest initial response.

  2. Invoice intake
    Route invoices to the right cost center, extract line items and PO references.

  3. Internal access requests
    Parse request, check policy, draft approval/denial rationale.

Document:

  • Current steps a human takes.
  • Inputs, tools, systems touched.
  • Where judgement is required vs. pure mechanics.

Day 3–4: Implement “LLM as router and scribe,” not autonomous agent

Goal: Put an LLM in front of the workflow without letting it take final actions.

Concrete steps:

  1. Wrap tools behind APIs

    • Query CRM, billing, permissions, or ticketing as separate functions.
    • Avoid giving the model direct SQL or admin access.

  2. Prompt the model to:

    • Interpret the request (intent + entities).
    • Decide which tools to call and in what order.
    • Produce:
      • a proposed plan,
      • tool arguments,
      • a human-readable summary of what it did and why.

  3. Keep execution human-driven:

    • The system calls tools.
    • The model only proposes.
    • A human reviews and clicks “apply.”

You now have:

  • AI automation of cognitive overhead (reading, routing, summarizing).
  • Humans still own final actions.

Day 5: Add basic governance and observability

Even for this small workflow:

  • Log everything:
    • Raw input, model outputs, tool calls, and human decisions.
  • Add a simple policy layer:
    • If a proposed action violates static rules (e.g., “refund > $200”), require extra approval.
  • Measure:
    • How often humans accept vs. edit vs. reject the agent’s plan.
    • Where the agent gets confused (missing data, ambiguous requests).

This is your initial feedback loop.

Day 6–7: Decide where to safely automate the “last click”

Based on real data from a few days:

  • Identify cases where humans nearly always accept the agent’s proposal.
  • Define strict constraints for full automation:
    • e.g., “If intent = password reset AND user is active AND no prior security flags → auto-approve.”

Then:

  • Allow the system to auto-execute only in those constrained scenarios.
  • Leave everything else in assist mode.

You’ve now:

  • Replaced a brittle rules engine or RPA script with a hybrid AI workflow.
  • Got actual metrics on precision/recall of your AI automation.
  • Scoped risk to a narrow, well-understood band.

Bottom line

AI automation in real businesses is not about “autonomous agents replacing humans.” It’s about shifting:

  • From: brittle, UI-bound RPA and hand-coded workflows that explode with complexity.
  • To: LLMs handling interpretation and orchestration, calling deterministic tools under policy.

If you’re a CTO or tech lead, the relevant questions are:

  • Where can we insert AI as a router / planner in front of existing systems?
  • What APIs do we need so that automation touches services, not UIs?
  • How do we instrument agents like any other production service?

You don’t need a grand strategy deck to start. You need:

  • One narrowly scoped workflow.
  • A clear separation of concerns: LLM for understanding, services for doing.
  • Guardrails that treat AI as fallible from day one.

Done right, AI agents, copilots, and workflow orchestration don’t replace your RPA overnight. They gradually squeeze it into the corners—the truly legacy bits that can’t be cleanly surfaced as services.

And if you can’t explain, in concrete terms, what your agent is allowed to do, what tools it can call, and how you’d detect if it went off the rails—you’re not automating. You’re just rolling the dice in production.

Similar Posts