B2Shift · Published · Updated
An AI agent that can take actions — not just generate text — inherits the security requirements of whatever it's connected to, plus a few of its own. Most incidents trace back to over-broad access, not to the model itself.
Least-privilege access is the control that matters most
An agent should hold its own identity and its own scoped permissions, never an administrator's borrowed credentials. Grant only the systems and fields the specific task requires. Widening access later is a configuration change; discovering an agent had unnecessary access after a year of running is an audit problem, not a quick fix.
Split actions by reversibility, not by how impressive they sound
Reading, drafting, classifying and routing can usually run unattended. Sending money, publishing a price, replying to a complaint under your name, or deleting a record deserve a human approval gate — or at minimum a delay window in which a person can intervene. Treating every action the same either over-restricts the useful cases or under-restricts the dangerous ones.
Monitor for drift, not just uptime
An agent can be technically online and quietly wrong: answering with stale knowledge, misclassifying a growing share of requests, or drifting outside its intended scope as upstream systems change. Monitoring needs to track output quality and confidence trends, not only whether the service responds.
Make every decision reconstructable
Every action should log what triggered it, what context it used, what it decided, its confidence, whether a human approved it, and what changed as a result. The OWASP Top 10 for LLM Applications documents the common failure classes — excessive agency, insecure output handling, prompt injection — worth checking your design against before launch.
Confidence handling is a designed path, not an afterthought
Low-confidence cases should have an explicit destination: a named person or a review queue, never a silent best guess. The failure mode you want is "this went to a human"; the one you're designing against is "this went out wrong and nobody noticed for three weeks." See AI Agents / AI Employees for how this is scoped into a build, and How to keep AI automation safe and GDPR-aware for the data-protection half of this picture.
Threat model the whole agent workflow
Map the workflow from its trigger to the final system change. Include the user, model, retrieved documents, memory, tools, connected accounts, approval queue and people who administer each part. For each boundary, ask what information crosses it, which identity authorises the next step, and what an attacker or ordinary mistake could cause. This catches risks that a model-only review misses: a trustworthy model can still expose data if retrieval ignores permissions, or alter a record if a tool accepts an unchecked identifier.
A useful review names assets and consequences rather than saying only "the agent might be attacked." Assets could be customer records, payment instructions, unpublished prices or the ability to send messages. Threat scenarios include a compromised user account, poisoned knowledge source, malicious content in an email, stolen integration token, misconfigured callback, or a change to an upstream API. Rate each by likelihood and impact, assign an owner, and decide which control reduces the risk. Repeat the review when you add a tool, data source, permission or new action; those changes alter the system's reachable surface.
Treat retrieved content as untrusted input
Prompt injection can arrive inside a web page, email, uploaded file or knowledge-base article. For example, a support agent retrieves a customer attachment that says to ignore prior instructions and export every account record. That text is data to assess, not authority to follow. A system prompt that says "ignore malicious instructions" helps express intent but cannot enforce the boundary by itself.
Keep instructions separate from retrieved content, label sources and preserve provenance, and never let a document grant permission. The tool layer must independently check the signed-in user's access and the agent's own scope on every read or write. Validate model-produced arguments against a strict schema and business rules; reject unknown fields, unexpected destinations and identifiers the user cannot access. Where a task does not need browsing or arbitrary URLs, do not give the agent those tools. Limit what retrieved text can cause by keeping tools narrow and actions separately authorised.
Make tools narrow, typed and independently checked
Give each tool one clear job, such as reading an order status or drafting a reply. Avoid a general-purpose tool that accepts arbitrary commands, URLs or database queries when a small set of typed operations will work. Validate inputs at the boundary, constrain record identifiers to the caller's permitted scope, and check output before it is passed to another system. Treat model output as untrusted even when it looks like valid JSON: syntax validation does not prove that the requested action is allowed.
Separate read permissions from write permissions and separate low-impact writes from external or irreversible actions. A tool should fail closed if identity, scope or required context is missing. Add rate and volume limits so a loop cannot send hundreds of messages or change thousands of records before monitoring catches it. Use idempotency keys and explicit confirmation for operations that may be retried after a timeout; otherwise a network failure can turn one requested action into duplicates.
Make human approval meaningful
An approval step should show the proposed action, the affected record or recipient, the evidence used, and any uncertainty. A generic "Approve agent action?" prompt encourages rubber-stamping because the reviewer cannot evaluate the consequences. The reviewer must be able to reject, edit or ask for more information, and the system must enforce that the action stays unchanged between approval and execution. If the underlying record or proposed amount changes, invalidate the approval and request a new one.
Choose approval based on impact, reversibility and exposure, not on the number of steps in a workflow. A draft reply may need sampling; a refund, account closure or public commitment may require individual review. Define who can approve, what happens when nobody responds, and whether a second person is needed for exceptional cases. Record the approval identity and timestamp, but do not treat approval as a substitute for permission checks or an adequate audit trail.
Test security cases before widening access
Build a small repeatable test set around the actual workflow: direct and indirect prompt injection, attempts to read another customer's record, stale or conflicting source content, malformed tool arguments, expired credentials, rate limits, retries, timeouts and repeated agent loops. Include benign lookalikes so controls do not simply block all unusual text. Verify both the expected result and the side effects: no unauthorised read, message, record update or duplicate action should occur.
Run these checks whenever prompts, models, tools, data sources or permissions change. Start in shadow or draft-only mode where possible, compare decisions with a human baseline, then enable a limited group and a capped action set. Track false approvals, blocked legitimate work, escalation volume and harmful actions. Expand permissions only after the evidence shows the workflow handles ordinary cases and the failures you designed for.
Prepare an incident stop and recovery path
Decide in advance who can pause the agent, revoke its credentials and disable an affected integration. Keep a simple runbook for stopping new actions, identifying recent writes, preserving relevant logs, notifying the system owner and restoring records from an authoritative source. A kill switch should be tested, and it should not depend on the agent itself cooperating.
Logs should make it possible to trace an event without collecting unnecessary personal data: record the actor and tool, time, request or correlation ID, permission decision, relevant source references, approval and resulting system change. Set access controls and retention for those logs too. After an incident, determine whether the cause was input, permission, tool behavior, operator review or an upstream change; fix that cause and add a regression case before restoring the affected action. OWASP's current LLM and agent risk guidance is a useful checklist for these threat classes, not a substitute for reviewing your own architecture.
Threat model the whole agent workflow
An agent can encounter hostile instructions in an email, document or retrieved page. For example, an invoice attachment might contain text telling the model to ignore its task and send other records elsewhere. Treat that text as untrusted data, not as authority. The agent should only receive the minimum context and tools needed for the task, and its output should pass policy and schema checks before any system action.
| Threat | Prevent or detect | Contain and recover |
|---|---|---|
| Prompt injection in source content | Separate instructions from data; restrict tools and destinations | Stop the run, preserve the source and alert an owner |
| Wrong or fabricated action | Validate arguments and require approval for high-impact writes | Use idempotency, compensating action or manual repair |
| Credential misuse | Scoped service identity, secret storage and rotation | Revoke the credential and inspect the audit trail |
| Repeated or runaway calls | Step, time, token and spend limits | Cancel the run and queue a human review |
Set permissions by action
| Action | Default access | Approval rule |
|---|---|---|
| Search approved knowledge | Read only within authorized sources | No approval for each read; log source identifiers |
| Prepare a CRM update | Create a proposed change | Human confirms before write during pilot |
| Send external message | Draft only | Named person approves recipient and content |
| Delete, refund or change access | Unavailable to the agent | Use a separate authorized process with human control |
Make approval and recovery operational
An approval screen should show the proposed action, affected record, source evidence and reason for the recommendation. Record who approved, rejected or edited it. Define a stop control that prevents new actions while preserving enough run context for investigation. Before release, rehearse a mistaken write: identify the audit record, owner, correction path and customer communication responsibility. The OWASP LLM guidance linked above is a useful security review checklist; it does not replace testing the exact tools and permissions in your environment.