An AI agent is production-ready when it can carry out a useful task within verifiable limits, with its own identity, least-privilege permissions, reproducible evaluation, complete traces, human oversight and a safe kill switch. Working in a demo doesn't prove it's reliable: production requires controlling what it can do, with what data, at what cost, and how each action gets reversed.
From pilot to production: what really changes
A prototype usually tests a happy path: it receives a request, looks up information, calls a tool and returns a convincing answer. Production brings ambiguous or malicious inputs, poisoned documents, downed services, inherited permissions, partial states, concurrency, model changes, and people who treat the response as a valid decision.
The decisive difference isn't the model — it's the system around the model. An agent combines probabilistic reasoning, memory, information retrieval, tools and the ability to act. Each component introduces its own failure modes as well as interaction failures. That's why design should start from the autonomy that's necessary, not the autonomy that's technically possible.
It's worth distinguishing four levels:
| Level | Capability | Dominant risk | Recommended control |
|---|---|---|---|
| Assistant | Suggests text or a decision | Error or inappropriate content | Human review before use |
| Copilot | Queries systems and prepares actions | Incorrect data or excessive access | Sources, permissions and confirmation |
| Supervised agent | Executes bounded actions | Misuse of tools | Threshold-based approval and rollback |
| Autonomous agent | Plans and chains actions | Cumulative or hard-to-detect harm | Sandboxed environment, strict limits and automatic kill switch |
Autonomy should only increase when there is evidence that it improves the process without pushing residual risk above the accepted threshold.
The minimum architecture of a reliable agent
Its own identity and permissions
The agent shouldn't operate under an administrator's account or inherit all of their privileges. It needs a distinct technical identity, rotatable credentials, and permissions scoped by tool, resource, operation and environment. Reading an order, drafting a document and approving a payment are distinct capabilities.
Permissions should also cap volume and frequency: maximum amount, number of records, authorized recipients or time windows. Wherever possible, read and write operations are separated, and an independent confirmation is required for irreversible actions.
Orchestrator with an explicit policy
The model can propose a plan, but a deterministic component must decide whether each action is allowed. The policy validates schema, identity, context, tool, arguments and limits. An instruction embedded in an email or document must never expand permissions.
Every tool call should include an execution identifier and idempotency key. That way, a retry after a timeout doesn't create two invoices, two tickets or two transfers. Compound operations need a compensation strategy: if the third step fails, the system must know what to roll back and what to leave pending for review.
Memory and knowledge with provenance
Conversation memory must not turn into an ungoverned database. Ephemeral context, approved memory and corporate sources need to be kept separate. Every retrieved fragment needs a provenance, date, owner, classification and set of permissions. The agent must not store an unvalidated output from the model itself as fact.
In RAG systems, authorization is applied before content is retrieved and checked again before it's shown. Filtering after the model has already read the document doesn't prevent exposure.
Sandboxed execution environment
If the agent generates code, processes attachments or browses the web, these tasks must run in a constrained environment, with no unnecessary secrets, restricted network access, quotas and teardown afterward. The output is treated as untrusted input. Generated code is never executed directly in production.
Evaluate before letting it act
Evaluation must represent the real work and its consequences. A useful test suite includes normal cases, edge cases, incomplete data, contradictory instructions, attacks, tool failures and version changes. It must be kept separate from the examples used to tune prompts.
Metrics are grouped into layers:
- Outcome: task resolved, field accuracy, coverage and decision quality.
- Process: correct tools, proper sequencing, number of steps, retries and loops.
- Security: prohibited actions, information leakage, resistance to injection and privilege escalation.
- Oversight: correct alerts, escalations, human overrides and time to detect a failure.
- Operations: latency, availability, cost per task and consumption in percentiles, not just averages.
- Impact: errors that reach people, complaints, corrections and potential harm.
A single overall score hides risk. Acceptance criteria must include hard gates: for example, zero unauthorized critical actions and complete traceability. A 95% success rate is worthless if the remaining 5% can make incorrect payments.
Before deployment, regression tests run for every change to the model, prompt, tool, policy, index or source. The agent is then activated first in shadow mode — without executing actions —, then with a limited population, and finally with a gradual rollout.
Threats specific to agents
The OWASP Top 10 for Agentic Applications 2026 identifies risks that belong in the threat model. These include goal hijacking, tool misuse, identity and privilege abuse, supply chain, unexpected code execution, and memory or context poisoning.
The most important controls are:
- Treat messages, web pages, documents and tool outputs as data, not as authoritative instructions.
- Allow only registered tools and approved versions.
- Validate arguments with schemas and deterministic business rules.
- Keep secrets out of the prompt and release them only at the moment of the authorized call.
- Sign or verify artifacts and log changes to tools, connectors and models.
- Limit plan depth, number of iterations, cost and time.
- Test for direct and indirect injection attacks, identity confusion and contaminated content.
- Implement a kill switch that blocks new actions without depending on the model itself.
Human oversight that is genuinely human
Adding an "approve" button doesn't create a safeguard if the person doesn't understand the action or is flooded with hundreds of alerts. Review should focus on decisions that matter: payments, contract changes, sensitive communications, data deletion, privilege grants, or decisions about people.
The approval screen should show the goal, sources, proposed action, tool, affected fields, uncertainty, consequences and alternatives. The reviewer needs time, competence and the authority to reject. Approvals are logged with identity and reason.
A matrix can be used:
| Impact | Reversibility | Confidence | Decision |
|---|---|---|---|
| Low | Easy | High | Automatic execution with sampling |
| Medium | Easy | Medium/high | Execution with alert or after-the-fact review |
| High | Partial | Any | Mandatory prior approval |
| Critical | Hard | Any | Out of scope or dual approval |
The model's confidence never substitutes for impact and reversibility.
Observability, incidents and continuous improvement
Every run needs a correlated trace: agent and model version, policies applied, sources retrieved, tools called, arguments — with sensitive data protected —, results, human decisions, cost and latency. Logs must not indiscriminately store prompts containing personal data.
Useful alerts detect repeated actions, a rise in rejections, sudden cost swings, unusual calls, denied access, loss of sources, deviation from rules, and degradation of the evaluation suite. There must be clear owners and response times.
An incident requires the ability to stop, isolate, reconstruct and fix. The minimum procedure includes preserving evidence, revoking credentials, identifying the actions taken, rolling back where possible, assessing the data and people affected, and deciding whether a regulatory notification is required.
Cost and performance control
Cost per token isn't the cost of the process. You need to add retrieval, tools, retries, human review, infrastructure, observability and incidents. The right economic unit is usually cost per valid task, not cost per call.
To reduce it without degrading quality:
- Use deterministic rules for simple validations.
- Select models based on the difficulty of each step.
- Limit context and retrieve only relevant documents.
- Cache only results with an expiry policy.
- Cut off loops and plans that show no progress.
- Reserve more expensive models for decisions that justify it.
A practical 90-day plan
Days 1-30: scope and evidence
- Choose a bounded, frequent and reversible process.
- Map out data, systems, people and consequences.
- Define permitted actions and explicitly prohibited ones.
- Create 50-100 initial test cases, including attacks and failures.
- Assign an owner, technical lead, security and legal review.
Days 31-60: controls and testing
- Implement its own identity, least-privilege permissions and deterministic policies.
- Add traces, idempotency, limits and a kill switch.
- Run functional, security and privacy tests.
- Operate in shadow mode and compare against the current process.
Days 61-90: limited rollout
- Activate with restricted users and operations.
- Review samples and every high-impact case.
- Measure outcome, risk, cost and adoption.
- Expand only the capabilities that pass the acceptance gates.
Go-live checklist
- Use case, owner and limits documented.
- Technical identity and least-privilege permissions per tool.
- Data, memory and sources with authorization and provenance.
- Actions validated by deterministic policies.
- Idempotency, timeout, retries and compensation tested.
- Functional, adversarial and regression evaluation passed.
- Human oversight tied to impact and reversibility.
- Traces, metrics, alerts and incident runbook in place.
- Budget and consumption limits configured.
- Gradual rollout and kill switch verified.
Frequently asked questions
What accuracy rate does an agent need?
There's no universal figure. It depends on the potential harm. For critical actions, certain classes of error must have zero tolerance and mandatory human approval.
Is it enough to add guardrails to the prompt?
No. The prompt is a weak layer. The decisive controls sit outside the model: identity, permissions, schemas, rules, isolation, limits and auditing.
Can it operate under a person's account?
It's not advisable. It prevents applying least privilege and clearly attributing actions. It should use a well-defined technical identity.
When should it stay a copilot?
When the task has high impact, limited reversibility, low frequency, or there isn't enough evidence to automate the decision.
How do you avoid a costly loop?
With limits on steps, time and budget; detection of stalled progress; idempotency; and escalation to a person.
Do you need to re-evaluate when changing models?
Yes. Also when changing prompts, tools, permissions, data, the index or rules. Any of these can change the system's behavior.
Primary and technical sources consulted
- NIST AI Risk Management Framework.
- NIST AI RMF: Generative AI Profile.
- NIST AI Agent Standards Initiative, February 2026.
- OWASP Top 10 for Agentic Applications 2026.
- AEPD (Spanish DPA): Agentic artificial intelligence and data protection.
Summum IA supports the move from prototype to production through architecture, evaluation, LLMOps and governance. The goal isn't to grant maximum autonomy, but to reach the minimum autonomy that delivers value in a controlled way.