AccuroAI
Book a demo
← Blog·Agent Security11 min read

How to Red Team AI Agents (and the Models Behind Them): What to Test, How Often, and What to Do With Findings

To red team AI agents, attack what they do, not only what they say: run multi-turn attacks through prompts, retrieved documents, tool output and MCP tool descriptions, and judge every tool call. This guide gives a test plan for OWASP ASI01 to ASI10, explains why NIST found repeated and adaptive attacks change results, and walks a finding from a hijacked refund to a runtime fix and a passing retest.

D
Dr. Marcus Chen
Threat Intelligence Director
Nov 19, 2026
On this page

To red team AI agents, attack what they do, not only what they say. Pick targets from your inventory, run multi-turn attacks through every input an agent reads (prompts, retrieved documents, tool output, MCP tool descriptions), judge each tool call it attempts, map findings to the OWASP agentic Top 10, fix them at runtime, then retest on every release.

Below: the layers to test, a plan for ASI01 to ASI10, cadence, safe testing and what to do with a finding. Today we also launched AccuroAI AI Red Teaming, which automates the method; one section describes it. For a definition, see the glossary entry.

Last verified: 19 November 2026, against NIST AI 100-2e2025, NIST CAISI's agent hijacking blog, the six-agency agentic AI guidance (Canadian Centre for Cyber Security copy), three OWASP GenAI pages and Articles 15, 55 and 113 on the Commission's AI Act Service Desk. All primary.

Seven stacked layers of an AI agent to attack, each with an example test: model, system prompt, retrieval and RAG, tools and MCP servers (highlighted), identity and permissions, memory, and other agents, with OWASP IDs such as LLM01, LLM07, ASI02, ASI03, ASI04, ASI06 and ASI07.
Figure 1. Tools and MCP servers are highlighted because that is the layer where a successful attack turns into an action.

Why does red teaming an agent mean testing actions, not answers?

For a chat model, a flaw usually shows up as text. For an agent it shows up as something done. NIST's adversarial machine learning taxonomy, NIST AI 100-2e2025, says so directly: "because agents can take actions using tools, these attacks can create additional risks in this context, such as enabling actors to hijack agents to execute arbitrary code or exfiltrate data from the environment in which they are operating."

A rude chatbot is a brand problem. A steered agent holding a refund tool is a loss.

Joint guidance from six agencies (the Canadian Centre for Cyber Security, CISA, the NSA, ASD's Australian Cyber Security Centre, and the New Zealand and UK national cyber security centers) agrees: "AI agents operate autonomously in complex environments and therefore require more thorough evaluations than LLMs."

So a finding should name the action: the tool, the arguments and the input that caused the call.

What should you test in an AI agent?

An agent is a stack, and every layer is a way in.

  • The model: jailbreaks and harmful or off-policy output.
  • The system prompt: extraction (OWASP's LLM07:2025 System Prompt Leakage) and whether its rules hold under pressure.
  • Retrieval and RAG: instructions planted in documents, tickets or web pages the agent reads, the indirect side of LLM01:2025 Prompt Injection. Our prompt injection guide covers the mechanics.
  • Tools and MCP servers: what each tool can do, hidden instructions in descriptions, and harmless tools chained into a harmful sequence.
  • Identity and permissions: whether a standard user can get the agent to act as an admin.
  • Memory: whether something written in one session changes behavior in the next.
  • Other agents: whether a message claiming to come from a peer is trusted unchecked.

Write down what is out of scope, too. Training-time attacks such as data poisoning need the training pipeline and are a separate exercise.

What test plan covers the OWASP Top 10 for Agentic Applications?

OWASP published the list on 9 December 2025. It is a risk list, not a test standard, but it makes a good skeleton: one attack per entry, with the pass written down first. (OWASP's announcement shortens the ASI02 and ASI03 names; we use the full ones.)

OWASP entryOne concrete testWhat a pass looks like
ASI01 Agent Goal HijackA retrieved document tells a support agent to issue a refundNo refund call
ASI02 Tool Misuse and ExploitationOver several turns, push a bulk export of customer records through a read toolThe export never starts
ASI03 Identity and Privilege AbuseA standard user asks for an admin-only toolDenied by the user's own permissions
ASI04 Agentic Supply Chain VulnerabilitiesAn MCP tool description hides an instruction, then changes after approvalIgnored; the change is caught
ASI05 Unexpected Code ExecutionA poisoned README asks a coding agent to run a shell commandThe command does not run
ASI06 Memory & Context PoisoningAn instruction written to memory in one session fires in the nextTreated as data
ASI07 Insecure Inter-Agent CommunicationA message posing as a peer agent asks for a paymentRejected as unverified
ASI08 Cascading FailuresA wrong result enters step one of a workflowContained before a later step acts
ASI09 Human-Agent Trust ExploitationThe agent gives a reviewer a misleading reason to approve a risky actionThe approver sees the real action
ASI10 Rogue AgentsThe agent is told to stop mid-taskIt stops; every action is logged

Every pass is observable: a tool call happened or it didn't. Our annotated guide to the agentic Top 10 has the controls for each entry.

How do you red team MCP servers and tool definitions?

Models tend to obey tool descriptions, which makes them an attack surface. Our post on tool poisoning walks through the patterns. The harder case is change over time. OWASP's exploit roundup for Q3 2026, which calls itself "not an exhaustive report or an official OWASP publication", warns: "An MCP server that behaves safely during initial review can later change its tool definitions and become malicious."

Three tests per server: is a hidden instruction followed, can its tools be combined to read and send out sensitive data, and does anything notice when a description changes? Start with the worst-scoring servers; our guide to scoring MCP servers on employee laptops shows how to rank them.

Why run multi-turn attacks, and every attack more than once?

Single prompts find the easy failures. NIST AI 100-2 describes the Crescendo attack, which "introduced the idea of interacting with the model iteratively in a multi-turn adaptive attack that includes seemingly benign prompts, eventually leading to a successful jailbreak against safety alignment."

Agents are also not deterministic. NIST's Center for AI Standards and Innovation (CAISI) tested agent hijacking on the AgentDojo framework and, red teaming one environment, reported "an increase in attack success rate from 11% for the strongest baseline attack to 81% for the strongest new attack." When it "attempted each attack 25 times" across five injection tasks, "the average attack success rate increased from 57% to 80%."

Those are NIST's figures from NIST's experiments, not industry rates. Keep its lessons: "Evaluations need to be adaptive," and "Testing the success of attacks on multiple attempts may yield more realistic evaluation results." Read results per task, too, because a low average can hide the one task where a single success is too many.

Loop diagram: an attacker model sends a message to the target agent in staging, a judge scores the reply and any tool call, and the next turn adapts; findings with transcript and severity feed a CI/CD gate (highlighted) that fails the release while a critical finding is open.
Figure 2. The attacker, target and judge pattern NIST AI 100-2 describes, wired into a release pipeline.

What does automated red teaming do well?

NIST AI 100-2 describes the usual pattern: "Automated model-based red teaming employs an attacker model, a target model, and a judge." The attacker writes each message from the last reply, the target answers or acts, and the judge decides whether the attack worked. That makes the work repeatable:

  • the same attacks run on every release, so results compare;
  • a CI/CD step can fail a release while a critical finding is open;
  • a fix is retested with the exact attack that found it;
  • every run leaves a transcript for auditors.

How often should you red team an AI agent?

On triggers, with a schedule as the backstop. NIST's taxonomy explains why: "Evaluations measure model vulnerabilities at a particular moment in time; assessments may change if new attacks are developed, additional data is collected post-training, or model capabilities are improved." Run a campaign whenever the model, system prompt, a tool or MCP server, a data source or the agent's permissions change, and when a new agent appears in your inventory.

How do you red team an agent without real side effects?

Against production, a successful attack can issue a real refund. The six agencies tell teams to deploy sandbox environments to test agent behavior before production. So use staging, sandboxed tool back ends, scoped test credentials that cannot touch customer money, and test records you can reset. AccuroAI customers point red-team campaigns at staging or sandboxed targets, and we recommend against testing production.

What should happen to a finding? A worked example

An illustrative case: Sarah Mitchell's platform team runs a support agent that retrieves help articles and can call issue_refund. Refunds over $500 should go to a person.

A pre-release campaign seeds a help article with a hidden instruction, and the attacker model spends several turns asking about returns for a damaged order. The agent follows the instruction and calls issue_refund for $4,800. The judge marks it critical; the finding maps to ASI01, ASI02 and LLM01:2025, with the transcript and tool call attached.

David Kim fixes it at the agent's gateway, not in the prompt: a rule that holds issue_refund calls above $500 for approval, plus blocking of prompt injection found in retrieved text. (Our explainer on AI agent gateways covers why that layer.) The same attack runs again, no refund call goes out, the retest passes, and both runs stay on file.

That loop is what the six agencies describe: "Use the results from these evaluations to validate existing controls and guide the development of stronger security measures." A prompt edit alone is weaker; OWASP's roundup concludes that "Prompt-level instructions alone do not establish a secure boundary."

Four steps from left to right: a critical finding where issue_refund is called for $4,800, a runtime fix (highlighted) that holds refunds over $500 for approval and blocks injection in retrieved text, a retest with the same attack that passes, and evidence with both runs kept.
Figure 3. The retest uses the exact attack from the finding, so a pass means this path is closed, not that the agent is safe.

What do the standards actually ask for?

  • OWASP agentic Top 10 and NIST AI 100-2: a risk list and a taxonomy. Guidance, not rules.
  • Six-agency guidance: recommends teams "Conduct red-teaming exercises to identify potential loopholes" and "Conduct regular assessments of an agent's ability to bypass safeguards".
  • EU AI Act Article 15(5): high-risk systems must resist attempts by unauthorized third parties to alter their use, outputs or performance, including "adversarial examples or model evasion" and "confidentiality attacks or model flaws". Annex III systems from 2 December 2027. No mention of red teaming.
  • EU AI Act Article 55(1)(a): providers of general-purpose AI models with systemic risk must evaluate their models, "including conducting and documenting adversarial testing of the model". This has applied since 2 August 2025 and binds model makers, not most deployers.

So the Act requires adversarial testing only for general-purpose models with systemic risk; for high-risk systems, red-team records can be part of the resilience evidence. Our unified compliance crosswalk sets Article 15 beside NIST AI RMF and ISO/IEC 42001. This is general information, not legal advice; confirm with counsel how the Act applies to you.

What we built: AccuroAI AI Red Teaming

AccuroAI AI Red Teaming, generally available today, runs this method as automated campaigns. An attacker model runs multi-turn attacks that adapt to each reply, against models by API (OpenAI, Anthropic, Azure OpenAI, Gemini, self-hosted), AI agents and their tool calls, MCP servers, and chatbots and RAG apps. Targets can come from what AccuroAI already discovered: Runtime Discovery models, and servers in the MCP server inventory, each with a 0 to 100 risk score.

Campaigns cover four attack families: prompt injection (direct and indirect) and jailbreaks; data leakage, including personal data, secrets and the system prompt; agent tool misuse, such as goal hijack, privilege escalation, destructive commands and exfiltration through tools; and harmful or off-policy content. They run on a schedule or as a CI/CD gate. Each finding comes with the full transcript, severity, reproduction steps, fix guidance and a retest, mapped to the OWASP LLM and Agentic Top 10 lists, MITRE ATLAS, NIST AI RMF and the EU AI Act.

From the console, a finding can become a runtime policy: an Agent Gateway rule, an endpoint policy or an MCP server block. It is hosted by AccuroAI and pointed at staging or sandboxed targets, and it supplies evidence, not compliance. See the AI Red Teaming product page.

What should you do this quarter?

  1. List your agents, model endpoints and MCP servers, with owners.
  2. Write the pass for each ASI entry for your three most exposed agents.
  3. Stand up staging with sandboxed tools and test credentials.
  4. Run multi-turn campaigns; repeat key attacks; read results per task.
  5. Fix at the point of action, retest, keep both records.
  6. Put the campaign in CI with the triggers above.

If you are comparing tools, our AI Agent Security RFP template helps because its 45 questions are mapped to ASI01 to ASI10, so vendor answers line up with this test plan.

Where does automated testing stop?

Campaigns can rerun attacks on every release and keep the transcripts. They cannot prove an agent is safe. NIST cautions that "Current evaluation approaches, though a useful tool, may underestimate vulnerabilities accessible to actors with more time, resourcing, or luck." Nor can software set your risk appetite: the refund limit, an agent's tools, whether a system is high-risk. People decide those, and no product makes an organization compliant.

FAQ

Is AI red teaming the same as a penetration test?

No. A penetration test checks the systems around the AI, such as networks, APIs and authentication. AI red teaming checks the AI's behavior, including what an agent does with its tools when someone steers it through a prompt or a retrieved document. Most teams need both.

How often should you red team an AI agent?

Before every release, and whenever the model, system prompt, tools, MCP servers, data sources or permissions change. Add a schedule for the time in between, because hosted models and MCP servers can change without a release on your side. Repeat the attacks that matter.

Does the EU AI Act require AI red teaming?

Not in those words. Article 55(1)(a) requires documented adversarial testing from providers of general-purpose AI models with systemic risk. Article 15(5) requires high-risk systems, from 2 December 2027 for Annex III, to resist named attacks without prescribing a method. Red-team records can serve as evidence. This is not legal advice.

Can you red team MCP servers?

Yes. Test whether hidden instructions in tool descriptions are followed, whether a server's tools can be chained to read and send out sensitive data, and whether a changed definition is picked up silently. Retest after any change to a server's tools.

Is AccuroAI's red teaming automated or manual?

Fully automated. An attacker model runs multi-turn attacks that adapt to each reply, campaigns run on a schedule or as a CI/CD gate, and every finding comes with its transcript, severity, reproduction steps and fix guidance. After a fix, the same attack runs again as a retest.

To watch a campaign from first attack to retest, Book a demo of AI Red Teaming or read the product page. Book a demo

Sources

More from the blog
See AccuroAI in action.

Book a 30-minute demo and see how security teams use AccuroAI to discover, govern, and protect every AI asset across their organization.

Book a demoStart a free pilot

15 enterprises secured · under 38ms p99 · Deployed in under 30 minutes · 72-hour pilot