An AI agent gateway is a control point that sits between your AI agents and the systems they act on, inspecting each tool call, argument, and response against policy before the action reaches a database, an API, or an external service. It is not a recognized category. Four major vendors currently ship products under that name, and they do four materially different things.
That last point is the reason this article exists. If you are evaluating one, the word on the box tells you almost nothing about what you are buying, and one of the differences decides whether the product can see half your agent traffic at all.
What is the difference between an AI gateway and an agent gateway?
An AI gateway, in the way most vendors use the term, fronts inference. It sits between your applications and the model providers, and it is built for the concerns that come with paying per token: caching, retries, model fallback, rate limits, spend analytics. Cloudflare's AI Gateway is the clean example of this shape. Its documentation is organized entirely around inference traffic, and contains no references to MCP, tool calls, or function calling.
An agent gateway is positioned one layer further out. The interesting traffic is not the prompt going to the model, it is the tool call coming back out of it: the database query, the file write, the outbound HTTP request, the email send. That is where an agent stops being a text generator and starts being something that changes state in your estate.
The distinction matters because the two sit on different paths. A gateway in front of the model never sees what happens after the model decides to call a tool. If your concern is cost and latency, the inference path is the right place to stand. If your concern is what the agent does, it is the wrong one.
Four vendors, four definitions
Here is the divergence, in each vendor's own framing. This is the clearest evidence available that the term has not settled into a category.
| Product | What it actually is | Speaks MCP? |
|---|---|---|
| agentgateway (Linux Foundation) | Data-plane proxy for agent traffic, supporting stdio, HTTP and SSE transports | Yes, MCP and A2A |
| AWS Bedrock AgentCore Gateway | Protocol converter and credential broker; turns APIs and Lambda functions into MCP-compatible tools | Yes, it produces MCP tools |
| Docker MCP Gateway | Container supervisor that launches MCP servers in isolated containers with restricted privileges | Yes, it runs the servers |
| Cloudflare AI Gateway | Inference observability and caching proxy | No MCP references in its documentation |
Last verified: September 2026.
Read that table twice. A proxy, a protocol converter, a container supervisor, and an analytics layer are being sold under one phrase. AWS describes its Gateway as a "fully managed AI gateway that provides a single, secure entry point for agentic traffic," with capabilities it names Translation and Secure Credential Exchange. Docker describes its MCP Gateway as a service that "runs MCP servers in isolated Docker containers with restricted privileges, network access, and resource usage." Those are not competing implementations of the same idea. They are different products that happen to share a noun.
No standards body has defined the term. There is no NIST, ISO, IETF, or CISA definition of an AI gateway, an agent gateway, or an MCP gateway. The closest standards-adjacent artifact is NIST SP 800-207A from September 2023, which names API gateways, sidecar proxies, and identity infrastructures such as SPIFFE as zero-trust enforcement platforms, and never uses the phrase at all.
Is an agent gateway the same as an API gateway?
No, and the reason is worth being precise about, because on an architecture diagram they occupy the same box.
An API gateway routes. It authenticates a caller, applies a rate limit, maps a path to a backend, and forwards. It is indifferent to payload semantics by design, because that indifference is what lets one gateway serve a thousand endpoints. An agent gateway has to read the payload, because the payload is the decision. Blocking a tools/call requires knowing that the call is database.query, that its argument is a SELECT against a table holding credentials, and that the sequence it belongs to started with content pulled from an untrusted source.
| API gateway | AI/LLM gateway | Agent gateway | |
|---|---|---|---|
| Sits between | Clients and services | Applications and model providers | Agents and the tools they call |
| Primary unit | HTTP request | Prompt and completion | Tool call and its arguments |
| Reads the body? | Rarely, by design | Yes, for tokens and caching | Yes, the arguments are the policy object |
| Optimizes for | Throughput and routing | Cost, latency, provider failover | Whether the action should happen |
| Sees a destructive DELETE? | As bytes | Not at all | As a policy decision |
Last verified: September 2026.
What does an agent gateway actually inspect?
Six things, and each is a mechanical capability rather than a marketing claim.
The tool surface. It sees every tools/list response and every tools/call, so it can allowlist tool names, pin schemas, and notice when a server quietly changes a tool's description between calls. That last pattern has a name in the research community, the rug pull, and it is invisible to anything that does not keep a record of what the tool looked like yesterday.
Arguments, as structured data. This is the part people underestimate. Tool arguments arrive as JSON, not prose, which means policy on literal values is enforceable without asking a model to interpret anything. Table names, file paths, recipient addresses, and raw SQL are all inspectable strings. When General Analysis demonstrated an exfiltration path through a Supabase MCP server in July 2025, the malicious step was a query reading an integration tokens table. A checkpoint reading arguments sees that query. A checkpoint reading routes does not.
The shape of a sequence. A gateway sees order. Untrusted read, then sensitive read, then outbound write is a shape, and the shape is blockable even when no individual step is. Invariant Labs published a GitHub MCP flow in May 2025 where an attacker files an issue in a public repository containing an injected instruction; when the victim later asks their agent to triage issues, the agent uses its own token to read private repositories and writes the contents into a pull request the attacker can read. Their own conclusion was that this "is not a flaw in the GitHub MCP server code itself, but rather a fundamental architectural issue that must be addressed at the agent system level." Blocking the shape stops that flow without ever detecting the injection that started it.
Credential boundaries. The MCP specification is blunt about the anti-pattern here. Token passthrough, where a server accepts a token from a client and forwards it downstream without validating that the token was issued for it, is called out directly: "MCP servers MUST NOT accept any tokens that were not explicitly issued for the MCP server." The spec also names the consequence, which is the one auditors care about. "The downstream Resource Server's logs may show requests that appear to come from a different source with a different identity, rather than the MCP server that is actually forwarding the tokens." A gateway that terminates the inbound token, checks its audience, and mints a fresh short-lived credential scoped to the one downstream call is implementing what the spec asks for.
Keeping the model away from secrets. In September 2026 the IETF's WIMSE working group adopted a draft on AI agent identity, draft-ietf-wimse-aims-00, co-authored by people at AWS, Zscaler, Ping Identity, OpenAI, and Okta. It carries a normative rule that a large number of current deployments violate: "The Large Language Model MUST NOT have access to an agent's credentials or to credentials that may be needed to access tools and services." Injecting credentials at the checkpoint rather than handing them to the agent is the conformant way to satisfy that.
Hostile encodings. Stripping Unicode Tags, the U+E0000 to U+E007F block that renders invisibly while tokenizers still read it, is cheap and deterministic. Johann Rehberger documented this smuggling technique in 2024. It costs nothing to normalize it out of arguments and outbound URLs, and it closes a class of attack that no amount of model alignment addresses.
What an agent gateway structurally cannot do
This is the half that should decide your evaluation, and it is the half no vendor page covers.
It cannot see a local stdio MCP server. Everything above assumes the traffic crosses a wire. A large share of real MCP deployments does not.
MCP defines two standard transports. Streamable HTTP puts each message in an HTTP POST, and the specification deliberately mirrors metadata into HTTP headers "so that intermediaries can route and inspect requests without parsing the body." Gateways are anticipated on that path. The stdio transport works differently: the client launches the MCP server as a subprocess and the two communicate over the subprocess's standard input and output. The specification is explicit that "all request metadata for the stdio transport is carried inline in the JSON-RPC message body," and that "there is no header layer."
No socket. No packet. No TLS session. A network-positioned device, whether that is a proxy, a service mesh sidecar, a CASB, or a SASE egress point, observes precisely nothing of that conversation.
It gets sharper. MCP's authorization framework applies only to HTTP transports, and tells stdio implementations to stay out of it: "Implementations using an STDIO transport SHOULD NOT follow this specification, and instead retrieve credentials from the environment." So on the stdio path there is no token, no audience, and no scope for anything to validate. And the specification's own security guidance recommends stdio as a hardening measure, advising that servers intended to run locally should "use the stdio transport to limit access to just the MCP client."
Put those three facts together. The specification recommends, as a security control, the exact transport that a network checkpoint cannot observe, on which its authorization framework does not apply. The only way to cover stdio is to stop being a network device and become the process launcher, which is what Docker's MCP Gateway is, and why agentgateway lists stdio among its supported transports. That is a supervisor, not an inline checkpoint.
If you take one thing from this article, take this: "inline" and "covers local stdio MCP servers" describe two different architectures, and most buyers assume they are one product. Ask the vendor which one they are, and ask them to show you a local stdio server's tool calls in their console.
It cannot authorize intent, because intent never crosses the wire. Every tool call the gateway sees is syntactically valid and inside the agent's granted scope. It sees read_file; it does not see why. A prompt-injected call and a legitimate call are byte-identical at the checkpoint. This is not an implementation gap that a better product will close. The WIMSE draft concedes the point on behalf of the whole industry, stating that translating an agent's natural-language mission into authorization requirements "is out of scope of this specification."
It cannot enforce beyond where it stands. NIST said this about enforcement points generally, in 2020, and it remains the cleanest statement of the limit: "The PDP/PEP cannot apply additional policies beyond its location in the flow of traffic." A checkpoint governs the path it is on. Agents have several.
It breaks transport-layer identity, by design. The same WIMSE draft is direct about what an intermediary costs you. "In architectures involving intermediaries, such as proxies, API gateways, service meshes, load balancers, or protocol translators, TLS sessions are often terminated and re-established, breaking the end-to-end continuity of transport-layer identity." If your identity story depends on mutual TLS end to end, inserting a gateway is the thing that ends it, and you need application-layer identity to carry the context instead.
Do you need an agent gateway if you already have an MCP proxy?
Probably, because those words describe a position rather than a capability, and the gap between them is where most of the risk lives.
An MCP proxy that routes and authenticates is doing the API gateway job on a new protocol. The questions that separate it from a control point are concrete. Does it read tool arguments, or only route by tool name? Does it retain a record of tool schemas so a silent change is detectable? Does it evaluate sequences across a session, or each call in isolation? Does it terminate and re-mint credentials, or pass the caller's token through, which the specification names as an anti-pattern?
There is a protocol detail that makes the sequence question harder than it sounds. MCP as of the 2026-07-28 revision is a stateless protocol: "A server processes each request independently; no state should be inferred from previous requests, even those on the same connection or stream." The specification goes further, warning that "an open connection, such as a STDIO process, is not a conversation or session." If your checkpoint infers session identity from a connection, the protocol has told you that inference is unsound. Correlating a sequence has to be done on explicit identifiers the client passes, which is a design decision, not a default. If you are inventorying what your agents connect to before you decide where to stand, governing every MCP server your agents touch is the prerequisite, and our MCP field guide covers the inventory step in detail.
How much latency does an agent gateway add?
The honest answer is that it depends on what the checkpoint does, and the range across products is wide enough that a single number is meaningless without knowing which work is in the path.
Three things sit on the critical path. Argument inspection is cheap, because it is string and schema work on structured JSON. Credential minting adds a round trip to your identity provider unless it is cached. Content classification, which is where redaction and sensitive-data detection happen, is the variable one, and it is the one worth measuring, because it is the part that reads every argument value rather than just its shape.
The number to hold a vendor to is a p99 under load, not an average in a demo, and it should be measured with classification enabled rather than in a routing-only configuration. For reference, AccuroAI inspects every agent action inline at under 38ms at p99, with classification in the path. Ask for the equivalent figure and the conditions it was measured under. An average latency quoted without a percentile is a number chosen to look good.
Latency also is not the only budget. A checkpoint that terminates TLS is a checkpoint that can fail, and an inline control that fails closed stops your agents while an inline control that fails open stops protecting them. Which one it does under load, and whether you get to choose, belongs on the same page of the evaluation as the millisecond figure.
When does an enterprise actually need one?
Three conditions, and you probably need one when two of them are true.
The first is write access. An agent that only reads is a disclosure problem, which is serious but recoverable. An agent that can delete, transfer, send, or deploy is an integrity problem, and integrity problems are not recoverable by apology. The point at which agents got write scopes is the point at which a checkpoint started earning its keep.
The second is untrusted input reaching a privileged path. If any content an agent processes comes from outside your control, a public repository issue, an inbound email, a web page, a customer-submitted document, then the injection surface is live. Every published agent exfiltration flow of the last eighteen months has this shape.
The third is scale past the point where review is real. Ten agents can be read by a person. Four hundred cannot, and the second number arrives faster than teams expect because agents are created by developers rather than procured by IT. If you want a number for your own estate before deciding, our agent risk calculator works from five inputs.
What a gateway does not fix is worth stating plainly too. It does not fix an agent holding a standing credential with a broad scope, because it is not in the identity path. It does not fix an agent whose granted permissions exceed the user who invoked it, which is an authorization design question that the policy layer the gateway enforces has to answer. And it does nothing at all about the agents nobody registered, because you cannot put a checkpoint in front of traffic you have not found. That is why discovery comes first, and why the MCP servers your security team cannot see are the ones that matter most.
The short version
An agent gateway is the data plane of an AI control plane: the place a decision gets enforced, not the place it gets made. That distinction is the one to hold on to while you evaluate, and our guide to the AI control plane covers the control side.
Ask three questions of any product wearing the name. Which transports can you actually observe, including local stdio. Do you inspect arguments and sequences, or route and authenticate. What happens to identity and to availability when you terminate the connection. The answers vary enormously between products that describe themselves identically, and the variance is not in the marketing, it is in the architecture.
A checkpoint is worth having. It is not worth mistaking for coverage.