AI Red Teaming vs Penetration Testing: What Each Finds and Why You Need Both
A penetration test checks the systems around your AI, such as networks, APIs, authentication and infrastructure; AI red teaming checks the AI's behavior, including what an agent does with its tools under hostile input. They find different flaws, so most teams need both. This guide quotes the NIST and CISA definitions, compares the two side by side, shows what each one misses, and sets out cadence, procurement questions and what the EU AI Act actually says.
J
James Okafor
Field CISO
Nov 20, 2026
On this page
A penetration test checks the systems around your AI: networks, APIs, authentication and infrastructure. AI red teaming checks the AI's behavior, including what an agent does with its tools when someone feeds it hostile input. The two find different flaws, and a clean result from one says little about the other. Most teams need both.
That is the short answer for a CISO asked whether this year's pen test covered the new support agent.
It almost certainly covered the web front end, the login flow and the servers. Whether anyone tried to talk the agent into misusing its tools depends on scope, not on the testers' skill.
Last verified: 20 November 2026, against the NIST CSRC penetration testing glossary, NIST AI 100-2e2025 and NIST AI 600-1 (text extracted from the PDFs), CISA's AI red teaming post, the six-agency agentic AI guidance (Canadian Centre for Cyber Security copy), two OWASP GenAI pages and AI Act Articles 15, 55 and 113 on the Commission's AI Act Service Desk. All primary.
Figure 1. Agent actions are highlighted because that is where a steered model turns into a refund, a deleted record or a sent file.
How do NIST and CISA define penetration testing and AI red teaming?
Vendors use these words loosely, so start with NIST. Its CSRC glossary quotes NIST SP 800-115: "Security testing in which evaluators mimic real-world attacks in an attempt to identify ways to circumvent the security features of an application, system, or network." A shorter definition, credited to NIST SP 800-12 Rev. 1 and SP 800-53 Rev. 5: "A test methodology in which assessors, typically working under specific constraints, attempt to circumvent or defeat the security features of a system."
Notice the object. Security features.
NIST's adversarial machine learning taxonomy, NIST AI 100-2e2025 (March 2025), defines the other discipline: "red teaming in the AI context, means a structured testing effort, often adopting adversarial methods, to find flaws and vulnerabilities in an AI system, including unforeseen or undesirable system behaviors or potential risks associated with the misuse of the system." The Generative AI Profile, NIST AI 600-1 (July 2024), has: "AI Red-teaming: A structured testing exercise used to probe an AI system to find flaws and vulnerabilities such as inaccurate, harmful, or discriminatory outputs, often in a controlled environment and in collaboration with system developers."
Here the object is behavior.
CISA connects the two. Its post "AI Red Teaming: Applying Software TEVV for AI Evaluations" (26 November 2024) says: "AI red teaming is the third-party safety and security evaluation of AI systems; AI red teaming is a subset of AI Testing, Evaluation, Verification and Validation (TEVV)." It argues that AI TEVV "must be treated under software TEVV from a strategic and operational perspective," and says CISA's own post-deployment testing support "includes varied forms of testing, such as penetration testing, vulnerability scanning, and configuration testing." The lesson: AI red teaming belongs in the same discipline as the pen test, with the same scoping, reporting and retesting.
What is the difference between AI red teaming and penetration testing?
Quoted cells use the source's words; the rest is our summary of common practice, and your contracts may differ.
Dimension
Penetration test
AI red teaming
Scope
Networks, hosts, web apps and APIs, authentication, cloud configuration
Model, system prompt, retrieved documents, tools, MCP servers, agent permissions
What it targets
"the security features of a system" (NIST SP 800-12 Rev. 1)
"unforeseen or undesirable system behaviors" and misuse (NIST AI 100-2e2025)
NIST AI 100-2e2025, NIST AI 600-1, CISA, six-agency guidance; EU AI Act Article 55(1)(a) (as adversarial testing)
Two differences matter most. The first is determinism. A missing authorization check fails on every request; a model may refuse an attack on Monday and comply on Tuesday, so one clean pass proves less. The second is where you test. NIST notes that pen testing "often involves issuing real attacks on real systems and data." Against an agent holding a refund tool, that means a real refund, so test agents in staging with sandboxed tools.
What does each test miss?
Two illustrative cases.
The pen test passes; the agent still pays out
Rachel Torres runs security for a retailer whose support agent reads help-center articles and can call an issue_refund tool. The annual pen test comes back clean: sessions expire, the API rejects calls without a valid key. Then an AI red team campaign plants a hidden instruction in one help article and spends several turns asking about a damaged order. The agent follows the planted text and calls issue_refund for $4,800.
No security feature failed. The agent did what its permissions allowed; someone steered its judgment. The finding maps to OWASP's ASI01 Agent Goal Hijack, ASI02 Tool Misuse and Exploitation and LLM01:2025 Prompt Injection. Our guide to red teaming AI agents follows a finding like this through to a runtime fix and a passing retest.
The AI red team passes; the system around it does not
Michael Brooks's platform team red teams an internal agent before release. Every attack fails. Meanwhile the agent's model API key sits in a configuration file in a public repository, and the gateway in front of the agent accepts expired tokens. Either one lets an outsider bypass the agent and act with the company's access.
The campaign never looked there; it talked to the agent through the front door. An exposed key or an authentication bypass is pen test work.
Figure 2. Neither result is wrong. Each test answered the question it was scoped to ask.
Scopes can overlap, which is fine if the statement of work says so. The failure is assuming coverage nobody wrote down.
Is "AI penetration testing" the same as AI red teaming?
Sometimes. "AI penetration testing" and "LLM penetration testing" are used for at least three engagements: a conventional pen test of the app that hosts a model, prompt-level attacks on a chatbot, or a full adversarial exercise against an agent and its tools. The label tells you little, so ask for the scope in writing:
Which layers are in: infrastructure, APIs, the model, retrieval, tools, MCP servers?
Are documents, web pages and tool output attacked, or only direct prompts?
Single messages, or multi-turn conversations?
Is a tool call judged as a finding, or only the text?
Does the report include transcripts, and is the same attack rerun after the fix?
How do you run both in one program?
Give each test a lane and share one backlog.
Scope split. The pen test owns infrastructure, APIs, authentication, keys and cloud configuration, including the gateway around every AI app. AI red teaming owns behavior: model, prompt, retrieval, tools, MCP servers and permissions in use.
Cadence. Pen test annually, plus after a major change. AI red teaming on every release and every model, prompt, tool or data change, with scheduled runs between.
Safe targets. Point AI red teaming at staging or sandboxed tools, as the six-agency guidance recommends for testing agent behavior before production.
That guidance, from the Canadian Centre for Cyber Security, CISA, the NSA, ASD's Australian Cyber Security Centre and the New Zealand and UK national cyber security centers, says it in one line: "Conduct regular security assessments, including penetration testing and red team exercises," aimed at agentic behavior. And then: "Use the results from these evaluations to validate existing controls and guide the development of stronger security measures."
Figure 3. The cadences differ because AI behavior can change with a prompt edit, while infrastructure changes less often.
Procurement is where the split gets enforced. Ask the scope questions above, and ask for a sample report. Our AI Agent Security RFP template helps because its 45 scored questions map to OWASP ASI01 to ASI10, so answers line up with testable risks; the walkthrough shows which red-team evidence to request.
Does the EU AI Act or another standard require either?
Neither term appears where most people expect.
Article 15(5) requires high-risk AI systems to be resilient against attempts by unauthorized third parties to alter their use, outputs or performance by exploiting system vulnerabilities. One sentence covers general cybersecurity: "The technical solutions aiming to ensure the cybersecurity of high-risk AI systems shall be appropriate to the relevant circumstances and the risks." The next lists AI-specific attacks that measures should address where appropriate, including data and model poisoning and "inputs designed to cause the AI model to make a mistake (adversarial examples or model evasion), confidentiality attacks or model flaws." No testing method is named. In our reading, pen test results speak to the first sentence and AI red team results to part of the second; poisoning needs another kind of test. Under Article 113, as amended by the Digital Omnibus on AI, this applies from 2 December 2027 for Annex III systems and 2 August 2028 for Annex I.
Article 55(1)(a) is where the Act names adversarial testing. Providers of general-purpose AI models with systemic risk must evaluate models "including conducting and documenting adversarial testing of the model with a view to identifying and mitigating systemic risks." Chapter V, which contains it, has applied since 2 August 2025. It binds the makers of the largest models, not most companies building on them.
In the US, NIST AI 600-1 recommends AI red-teaming in several actions, and it profiles the AI RMF, which NIST says "is intended for voluntary use." Our unified compliance crosswalk sets Article 15 beside NIST AI RMF and ISO/IEC 42001.
This section is general information, not legal advice; ask counsel how the AI Act applies to your systems.
Where does AccuroAI fit?
In the AI-behavior column only. AccuroAI AI Red Teaming is fully automated: an attacker model holds multi-turn conversations with the target and changes tactics after every reply. Targets include models by API (OpenAI, Anthropic, Azure OpenAI, Gemini, self-hosted), agents and their tool calls, MCP servers, and chatbots and RAG apps, and can be picked from what AccuroAI already discovered. Campaigns cover prompt injection and jailbreaks, data leakage, agent tool misuse, and harmful or off-policy content, on a schedule or as a CI/CD release gate.
Every finding carries its transcript, severity, reproduction steps, fix guidance and a retest, mapped to the OWASP LLM and Agentic Top 10, MITRE ATLAS, NIST AI RMF and the EU AI Act. From the console, a finding can become an Agent Gateway rule, an endpoint policy or an MCP server block. It is hosted by AccuroAI, and customers point it at staging or sandboxed targets.
What it is not: AccuroAI does not do penetration testing of networks or infrastructure, and it does not offer human or manual testing. The pen test column of your program stays where it is. Details are on the AI Red Teaming product page.
What should you do this quarter?
Mark which AI layers your last pen test actually covered.
List your AI apps, agents, model endpoints and MCP servers, each with an owner.
For the three most exposed agents, write down what failure looks like.
Stand up staging with sandboxed tools and test credentials, and run a first campaign there.
Put both tests on one findings backlog, with one severity scale and one retest rule.
Automation is good at repetition: the same attacks on every release, the exact attack rerun after a fix, every transcript kept.
No test proves an AI system safe. NIST is plain: "Current evaluation approaches, though a useful tool, may underestimate vulnerabilities accessible to actors with more time, resourcing, or luck." And "Evaluations measure model vulnerabilities at a particular moment in time," which argues for testing on change, not once a year. Nor can software set your risk appetite, such as which tools an agent holds or what refund limit is acceptable. People decide that, and a test result is evidence, not compliance.
FAQ
Is AI red teaming a type of penetration test?
Not in NIST's terms. NIST defines penetration testing as attempts to circumvent a system's security features; its AI red teaming definitions target flawed behavior and misuse. CISA places both under software testing, evaluation, verification and validation. They are siblings with different scopes, not substitutes.
Does a pen test cover prompt injection?
Only if the scope says so. A conventional engagement tests authentication, access control and infrastructure, and may never send an adversarial prompt or plant an instruction in a document the agent retrieves. Check the statement of work for indirect injection, multi-turn attacks and judged tool calls.
How often should you do each?
Our view: the penetration test annually and after a major change to the environment. AI red teaming before every release and whenever the model, system prompt, tools, MCP servers, data sources or permissions change, with scheduled runs between, because a hosted model can change without a release on your side.
Can automated tools do AI red teaming?
Yes. NIST AI 100-2e2025 describes automated red teaming with an attacker model, a target and a judge. NIST AI 600-1 lists human and AI red-teaming beside expert and general-public teams, noting different approaches may suit different harms. Automation suits tests that repeat on every change. AccuroAI's product is fully automated.
Which one does the EU AI Act require?
Neither by name. Article 15(5) requires high-risk systems to resist named attacks, from 2 December 2027 for Annex III, without prescribing a method. Article 55(1)(a) requires documented adversarial testing only from providers of general-purpose models with systemic risk. Test records can serve as evidence. This is not legal advice.