When an AI agent goes off-script

An AI agent was supposed to correct a delivery address in a CRM. Instead, due to an unclear rule, it overwrote the billing record, triggered an automatic notification, and created three follow-up tasks for sales. The model didn’t “fail.” The process had overly broad permissions, no approval layer, and no exception handling. That’s why there’s no honest yes-or-no answer to the question “How safe are AI agents?” They’re only safe within clearly defined tasks, data, and intervention rights.

For companies, this isn’t a theoretical risk. An agent can read emails, extract data, modify CRM fields, reconcile documents, or prepare bookings via APIs. Each of these capabilities saves manual work. But each also expands the potential damage when context, permissions, or control are missing. Safety doesn’t come from a better prompt alone—it comes from an operationally managed workflow.

Why AI agents fail in processes

An AI agent typically combines three things: a language model for classification, rules for decisions, and tools like CRM, email, databases, or ticketing systems. The language model formulates or categorizes. The real impact only happens when the agent is allowed to use a tool.

That’s the key difference between a chatbot and an operational agent. A chatbot can give a wrong answer. An agent, based on a misclassification, can modify a record, forward a request, or flag a payment for review. The more systems are connected, the more critical the question becomes: What is the agent explicitly allowed to do, and what is it explicitly not allowed to do?

A common mistake is granting an agent the same access as an experienced employee. That’s convenient but not pragmatic. An employee usually recognizes when an email with unusual wording or an invoice with a different bank account doesn’t fit the standard process. An agent needs explicit checks for that. Without them, it often treats an exception like a standard case.

The inputs themselves are also a risk. An email, an uploaded PDF, or a support ticket can contain instructions meant to derail the agent. Example: A supplier sends an invoice and writes in the document that internal review steps should be ignored. The agent must treat document content as data, not as instructions. This separation must be implemented technically and procedurally.

How safe are AI agents for real tasks?

Safety depends less on the “AI” label and more on four questions: What data does the agent see? What decisions does it make? Which systems can it modify? Who reviews unclear cases? This defines a clear safety framework.

An agent that sorts incoming requests by topic and drafts a response carries limited risk. An agent that confirms terminations, changes delivery addresses, or evaluates KYC, KYB, or AML documents operates in a different risk class. A good accuracy rate in a demo isn’t enough there. You need to measure how often the agent is correct, what types of errors occur, and how many cases go to exception handling.

Take document processing for invoices. An agent can extract invoice number, amount, supplier, and IBAN. The process isn’t secure just because all four values are extracted. It becomes safer through checks: Does the supplier exist in the master register? Does the IBAN match a previously confirmed bank account? Is the amount within the agreed tolerance? Do order number and goods receipt match? Only when these checks pass can the next step proceed.

In regulated areas, additional requirements apply. For KYC, KYB, or AML, an agent can pre-sort information, check documents for completeness, and justify anomalies. But the final decision on a risk-relevant case should remain with a responsible person—as long as the rule base, traceability, and approvals require it. For eIDAS-relevant documents, you must also cleanly separate what the agent extracts, what a cryptographic proof verifies, and what is legally binding.

The honest answer is: AI agents are reliable when the scope is limited, decisions are traceable, and fallback to humans is built in. They’re unreliable when given an open-ended task, broad access, and no oversight.

The safe path starts with a narrow task

Don’t start with “The agent handles all customer inquiries.” Start with a task that has a defined outcome, a data source, and a permitted action. Example: The agent reads inquiries from an inbox, detects missing information, and creates a response draft. It doesn’t send anything on its own.

This allows measurable quality checks. Take 100 to 300 historical cases, remove or protect personal data, and run the planned workflow against them. Then evaluate not just whether the response sounds good. Check if the correct category was chosen, if data was transferred correctly, and if the agent stopped when uncertain.

Misclassifications are particularly revealing. If the agent confuses two similar request types, a more precise rule or an additional data field often helps. If it doesn’t recognize a missing order number, a mandatory field check can route the case to exception handling. A test is only useful if every error leads to a concrete change in the workflow.

Rights by task, not convenience

An agent doesn’t need an admin account just because it works in a system. It needs exactly the rights its step requires. For CRM enrichment, that might mean read access to contact data and write access to two clearly named fields. For an email agent, it might mean: read messages, create drafts, but don’t send messages.

This limitation reduces damage even if an input is manipulative or the model misclassifies a situation. Additionally, especially critical actions should be technically blocked or tied to approval. These include changes to bank details, contract approvals, deletion processes, payment instructions, and external communications with legal effect.

Where APIs are connected, separate credentials for development, test, and production are essential. A test agent must not modify real customer data. A production agent may only work via documented interfaces. This sounds obvious but is regularly skipped in fast pilot projects.

Human-in-the-loop is a rule, not an alibi

Human-in-the-loop doesn’t mean someone glances at a dashboard occasionally. It means the workflow defines upfront under which conditions a human decides. These could include a low confidence score, contradictory data, a new business partner, an unusual amount, or an unreadable document.

The handoff must be usable for the reviewer. They need the original source, extracted values, applied rules, and the reason for escalation. If a compliance team sees only a result without justification for a KYB case, the process isn’t auditable. If they can trace which data was missing and which rule was triggered, they can decide and document the case properly.

The goal isn’t to artificially automate as many cases as possible. The goal is to handle standard cases quickly and in a controlled way, while making exceptions visible. That’s operationally better than a system that silently makes wrong decisions.

Operations determine safety after go-live

An agent isn’t “done” after deployment. Data formats change, CRM fields get renamed, suppliers use new invoice layouts, and interfaces return different error codes. Without monitoring, you often only notice these changes when a team suddenly handles an unusual number of manual corrections.

Ongoing operations therefore require defined metrics: processing time per case, share of automated standard cases, share of exceptions, error rate by error class, and time to correction. Supplement these with technical signals like API errors, failed jobs, and availability. For critical workflows, you need defined response times and clear responsibilities for uptime and SLAs.

Reconciliation is just as important. If an agent transfers data between inbox, CRM, and accounting, you must regularly check whether the number and status of cases match. If the inbox received 500 invoices, the processing system successfully completed 498, and two went to exception handling, the state is clear. Without this reconciliation, cases can disappear unnoticed or be processed twice.

Changes to the agent must be versioned and tested. New prompts, additional tools, or modified approval rules aren’t minor text adjustments—they’re changes to a business process. Document what was changed, why it was changed, and what the test results were. This creates an auditable history and prevents surprises after an update.

Safety means: someone operates the process

Many companies buy a prototype and treat it like finished software afterward. With AI agents, that rarely works long-term. They need an owner for the process, a technical owner for integration errors, and a subject-matter owner for rules and exceptions. These roles can be internal or managed as a service. What matters is that they’re assigned.

That’s why CINDR.LA designs such systems not from the model but from the process: permitted actions, data flows, approvals, monitoring, and exception handling are defined before broad deployment. It’s not spectacular, but clear, honest, and measurable.

If you’re introducing an agent, don’t first ask how autonomous it should appear. Ask which concrete step it reliably saves time on today, when it must stop, and who will notice the next morning if something’s wrong. That’s how a demo becomes a reliably operated workflow—pragmatic, controllable, and without surprises.