B2Shift · Published · Updated

An AI agent that can take actions — not just generate text — inherits the security requirements of whatever it's connected to, plus a few of its own. Most incidents trace back to over-broad access, not to the model itself.

Least-privilege access is the control that matters most

An agent should hold its own identity and its own scoped permissions, never an administrator's borrowed credentials. Grant only the systems and fields the specific task requires. Widening access later is a configuration change; discovering an agent had unnecessary access after a year of running is an audit problem, not a quick fix.

Split actions by reversibility, not by how impressive they sound

Reading, drafting, classifying and routing can usually run unattended. Sending money, publishing a price, replying to a complaint under your name, or deleting a record deserve a human approval gate — or at minimum a delay window in which a person can intervene. Treating every action the same either over-restricts the useful cases or under-restricts the dangerous ones.

Monitor for drift, not just uptime

An agent can be technically online and quietly wrong: answering with stale knowledge, misclassifying a growing share of requests, or drifting outside its intended scope as upstream systems change. Monitoring needs to track output quality and confidence trends, not only whether the service responds.

Make every decision reconstructable

Every action should log what triggered it, what context it used, what it decided, its confidence, whether a human approved it, and what changed as a result. The OWASP Top 10 for LLM Applications documents the common failure classes — excessive agency, insecure output handling, prompt injection — worth checking your design against before launch.

Confidence handling is a designed path, not an afterthought

Low-confidence cases should have an explicit destination: a named person or a review queue, never a silent best guess. The failure mode you want is "this went to a human"; the one you're designing against is "this went out wrong and nobody noticed for three weeks." See AI Agents / AI Employees for how this is scoped into a build, and How to keep AI automation safe and GDPR-aware for the data-protection half of this picture.

Threat model the whole agent workflow

Map the workflow from its trigger to the final system change. Include the user, model, retrieved documents, memory, tools, connected accounts, approval queue and people who administer each part. For each boundary, ask what information crosses it, which identity authorises the next step, and what an attacker or ordinary mistake could cause. This catches risks that a model-only review misses: a trustworthy model can still expose data if retrieval ignores permissions, or alter a record if a tool accepts an unchecked identifier.

A useful review names assets and consequences rather than saying only "the agent might be attacked." Assets could be customer records, payment instructions, unpublished prices or the ability to send messages. Threat scenarios include a compromised user account, poisoned knowledge source, malicious content in an email, stolen integration token, misconfigured callback, or a change to an upstream API. Rate each by likelihood and impact, assign an owner, and decide which control reduces the risk. Repeat the review when you add a tool, data source, permission or new action; those changes alter the system's reachable surface.

Treat retrieved content as untrusted input

Prompt injection can arrive inside a web page, email, uploaded file or knowledge-base article. For example, a support agent retrieves a customer attachment that says to ignore prior instructions and export every account record. That text is data to assess, not authority to follow. A system prompt that says "ignore malicious instructions" helps express intent but cannot enforce the boundary by itself.

Keep instructions separate from retrieved content, label sources and preserve provenance, and never let a document grant permission. The tool layer must independently check the signed-in user's access and the agent's own scope on every read or write. Validate model-produced arguments against a strict schema and business rules; reject unknown fields, unexpected destinations and identifiers the user cannot access. Where a task does not need browsing or arbitrary URLs, do not give the agent those tools. Limit what retrieved text can cause by keeping tools narrow and actions separately authorised.

Make tools narrow, typed and independently checked

Give each tool one clear job, such as reading an order status or drafting a reply. Avoid a general-purpose tool that accepts arbitrary commands, URLs or database queries when a small set of typed operations will work. Validate inputs at the boundary, constrain record identifiers to the caller's permitted scope, and check output before it is passed to another system. Treat model output as untrusted even when it looks like valid JSON: syntax validation does not prove that the requested action is allowed.

Separate read permissions from write permissions and separate low-impact writes from external or irreversible actions. A tool should fail closed if identity, scope or required context is missing. Add rate and volume limits so a loop cannot send hundreds of messages or change thousands of records before monitoring catches it. Use idempotency keys and explicit confirmation for operations that may be retried after a timeout; otherwise a network failure can turn one requested action into duplicates.

Make human approval meaningful

An approval step should show the proposed action, the affected record or recipient, the evidence used, and any uncertainty. A generic "Approve agent action?" prompt encourages rubber-stamping because the reviewer cannot evaluate the consequences. The reviewer must be able to reject, edit or ask for more information, and the system must enforce that the action stays unchanged between approval and execution. If the underlying record or proposed amount changes, invalidate the approval and request a new one.

Choose approval based on impact, reversibility and exposure, not on the number of steps in a workflow. A draft reply may need sampling; a refund, account closure or public commitment may require individual review. Define who can approve, what happens when nobody responds, and whether a second person is needed for exceptional cases. Record the approval identity and timestamp, but do not treat approval as a substitute for permission checks or an adequate audit trail.

Test security cases before widening access

Build a small repeatable test set around the actual workflow: direct and indirect prompt injection, attempts to read another customer's record, stale or conflicting source content, malformed tool arguments, expired credentials, rate limits, retries, timeouts and repeated agent loops. Include benign lookalikes so controls do not simply block all unusual text. Verify both the expected result and the side effects: no unauthorised read, message, record update or duplicate action should occur.

Run these checks whenever prompts, models, tools, data sources or permissions change. Start in shadow or draft-only mode where possible, compare decisions with a human baseline, then enable a limited group and a capped action set. Track false approvals, blocked legitimate work, escalation volume and harmful actions. Expand permissions only after the evidence shows the workflow handles ordinary cases and the failures you designed for.

Prepare an incident stop and recovery path

Decide in advance who can pause the agent, revoke its credentials and disable an affected integration. Keep a simple runbook for stopping new actions, identifying recent writes, preserving relevant logs, notifying the system owner and restoring records from an authoritative source. A kill switch should be tested, and it should not depend on the agent itself cooperating.

Logs should make it possible to trace an event without collecting unnecessary personal data: record the actor and tool, time, request or correlation ID, permission decision, relevant source references, approval and resulting system change. Set access controls and retention for those logs too. After an incident, determine whether the cause was input, permission, tool behavior, operator review or an upstream change; fix that cause and add a regression case before restoring the affected action. OWASP's current LLM and agent risk guidance is a useful checklist for these threat classes, not a substitute for reviewing your own architecture.

Threat model the whole agent workflow

An agent can encounter hostile instructions in an email, document or retrieved page. For example, an invoice attachment might contain text telling the model to ignore its task and send other records elsewhere. Treat that text as untrusted data, not as authority. The agent should only receive the minimum context and tools needed for the task, and its output should pass policy and schema checks before any system action.

Threats, controls and recovery paths for a tool-using agent
ThreatPrevent or detectContain and recover
Prompt injection in source contentSeparate instructions from data; restrict tools and destinationsStop the run, preserve the source and alert an owner
Wrong or fabricated actionValidate arguments and require approval for high-impact writesUse idempotency, compensating action or manual repair
Credential misuseScoped service identity, secret storage and rotationRevoke the credential and inspect the audit trail
Repeated or runaway callsStep, time, token and spend limitsCancel the run and queue a human review

Set permissions by action

Example permission boundary for an agent
ActionDefault accessApproval rule
Search approved knowledgeRead only within authorized sourcesNo approval for each read; log source identifiers
Prepare a CRM updateCreate a proposed changeHuman confirms before write during pilot
Send external messageDraft onlyNamed person approves recipient and content
Delete, refund or change accessUnavailable to the agentUse a separate authorized process with human control

Make approval and recovery operational

An approval screen should show the proposed action, affected record, source evidence and reason for the recommendation. Record who approved, rejected or edited it. Define a stop control that prevents new actions while preserving enough run context for investigation. Before release, rehearse a mistaken write: identify the audit record, owner, correction path and customer communication responsibility. The OWASP LLM guidance linked above is a useful security review checklist; it does not replace testing the exact tools and permissions in your environment.

Sources

OWASP — LLM06:2025 Excessive Agency

OWASP — Top 10 for Agentic Applications 2026

OWASP — Top 10 for Large Language Model Applications

European Data Protection Board — Opinion 28/2024 on AI models

Discuss your workflow