How to Evaluate an AI Agent Gateway for Enterprise Use

A gateway is a centralized control point sitting between AI agents and every model, tool, and system they call. Authentication, authorization, policy enforcement, observability: these either exist at that control point or they don't. There is no partial credit, and there is no retrofitting them later once your agent fleet is already running in production.

An agent calls a model to reason, then calls tools to act: GitHub, Stripe, internal databases, cloud configuration APIs. Often it spawns sub-agents that repeat the same pattern. Every hop consumes tokens. Every hop touches a system with its own permissions. Without a gateway, credentials scatter across codebases, enforcement fragments across teams, and cost tracking becomes genuinely impossible at fleet scale. You don't discover this gradually; you discover it all at once when something breaks and nobody can reconstruct what happened.

Anthropic released the Model Context Protocol in November 2024. By December 2025 it had 97 million monthly SDK downloads, backing from every major AI lab, and governance under the Linux Foundation. It moved from emerging standard to enterprise default in under two years. That acceleration answers one question: governing MCP traffic is not optional. The only open question is how.

A single AI request can traverse LLMs, MCP servers, agents, APIs, and event systems before producing a response. Most enterprises attempt to govern this through multiple proxies, which fragments visibility in ways that make incident response genuinely painful. A unified gateway collapses that into one runtime with shared auth, authorization, and observability.

The criteria that follow reflect what this infrastructure layer actually requires to function in production. Not aspirational targets. Requirements.

The Distinction Between Runtime Enforcement and Monitoring-Only That Most Gateway Comparisons Miss

Most vendor comparisons skip this entirely, which is how organizations end up with something they believe is a security layer and is actually a log aggregator.

Runtime enforcement means the gateway intercepts an agent's action before execution and blocks it if it violates policy. Monitoring-only means the same event is recorded after it has already occurred. Useful for postmortems, genuinely insufficient as a safety mechanism. These aren't two points on a maturity spectrum; they are architecturally different systems serving different purposes, and conflating them in an evaluation produces a real gap in production.

Many vendors describe their products as providing "security" or "governance" without specifying which model they implement. The architectural tell is straightforward: an inline gateway or proxy can block; an out-of-band log analysis system cannot. When a vendor is evasive on this point, the evasion is itself the answer.

Require a practical demonstration in any proof-of-concept. Ask the vendor to show real-time blocking of an unauthorized action, not a description of alerting capabilities. The demo either shows interception or it doesn't. Alerting you that an agent just wrote to a production database is not the same as stopping it from doing so.

Security Architecture Criteria: Credential Management, Identity Propagation, and Attack Surface Control

Credential sprawl is not theoretical. GitGuardian found 24,008 unique secrets exposed in MCP configuration files in 2025 alone, with Google API keys and PostgreSQL connection strings among the most common exposures. This is what happens when every agent carries its own credentials: the cumulative exposure surface scales with every new agent you deploy, and at some point it becomes unauditable.

What to require here is specific. Secrets vault integration or centralized credential management so agents never hold raw API keys. Credential rotation handled at the gateway, not distributed across individual codebases. In any regulated environment, these are table stakes.

Identity propagation is the second architectural requirement, and it's the one that tends to get glossed over in evaluations until it becomes a compliance problem. The gateway should validate incoming identity via JWT, OAuth 2.0 with PKCE, or OIDC, then propagate it downstream to MCP servers so that requests execute on behalf of authenticated users rather than shared service accounts. The practical consequence is direct: if a user cannot delete a repository, neither can the agent acting on their behalf. The MCP specification introduced OAuth 2.1 support in its March 2025 revision, with refinements in June 2025. Implementation quality varies materially between gateways; verify conformance against the specification, not just claimed support.

Access control models should be evaluated against your actual deployment profile. Role-based access control is sufficient for predefined agent roles with stable, predictable requirements. Attribute-based access control enables context-aware decisions across dimensions like time, location, and data sensitivity, which matters for dynamic agent fleets. Policy-based access control uses centralized policy decision points that govern agent fleets consistently across all deployments, and it's the model best suited to enterprise scale. These are not equivalent, and choosing the wrong one for your deployment profile creates rework.

The MCP-specific attack surface requires explicit attention. Security researchers identified in April 2025 that prompt injection, tool permission combinations that enable data exfiltration, and lookalike tools that silently replace trusted ones are active vulnerabilities in ungoverned MCP deployments. CVE-2025-6514 in the mcp-remote npm package, a command-injection vulnerability patched in v0.1.16, illustrates why every MCP server must be registered, versioned, and governed. An unregistered server is an identity the organization cannot account for.

The evaluation question is direct: can the gateway enforce that only approved, registered MCP servers are reachable? If not, the security perimeter has holes that no amount of monitoring will close.

Access Control Criteria: Making Sure Agent Permissions Reflect Deliberate Organizational Decisions

The core question is not whether access control exists. It is whether agent permissions resulted from deliberate policy decisions or accumulated by default as teams provisioned tools. In most ungoverned deployments it's the latter, and nobody on the team can tell you exactly when or why a particular agent got access to a particular system. That ambiguity is fine until an agent does something it shouldn't, and then it's a serious problem.

Minimum viable access control for production requires three things. Per-agent permission scopes: agents should have access only to the tools and data their role requires, not blanket access inherited from a service account. Tool-level granularity: access control should operate at the level of individual MCP server tools, not just server-level allow or deny. Dynamic scope adjustment: permissions should be modifiable without redeployment, because a gateway that requires a code change to update policy will cause teams to over-provision as a workaround. That workaround defeats the entire point of having access control.

The registry requirement follows directly from this. Every MCP server the organization uses should have a registry entry covering server identity, owner, approved scopes, and version. A server without a registry entry is an identity the organization cannot govern or revoke.

Shadow AI is the adjacent risk that access control discussions consistently underweight. Teams blocked from governed access to tools they actually need will find ungoverned paths; this is predictable behavior, not a failure of judgment. A gateway that makes access control frictionless to configure reduces shadow AI more reliably than restriction alone, because the governed path becomes the path of least resistance.

Ask the vendor to demonstrate provisioning a new agent with scoped tool access, then modifying that scope without a deployment. How much friction that involves is a direct proxy for how access policy will actually be maintained at scale. If Gartner's projection holds and 40% of enterprise applications embed agents by end of 2026, each carrying broad permissions, the cumulative blast radius of a single compromised agent identity becomes an enterprise-wide exposure.

Governance and Auditability Criteria: What a Usable Audit Trail Actually Requires

An ECI Research survey from its 2025 AI Builder Summit found that 44% of enterprise AI leaders have only moderate confidence that AI agents can act autonomously without human intervention. That hesitancy is rational. It reflects unresolved questions about accountability: when an agent creates a regulatory exposure, can the organization prove what actually occurred, and to whom?

A usable audit log captures which agent accessed what tool, with what input parameters, at what time, and what the outcome was. Logs that capture only success or failure are insufficient for security analysis or regulatory response. Tamper-resistance is a hard requirement; the platform should demonstrate how log integrity is maintained, because a log that can be altered after the fact is not a compliance artifact.

Policy-as-code is a governance prerequisite that evaluations underweight. Policies expressed in natural language documents cannot be executed by a policy engine. "Access should be approved by management" is not a machine-readable rule. Evaluate whether the gateway supports policy-as-code with version control, so governance changes are auditable in the same way code changes are. If your compliance team cannot diff a policy change, it is not really governed.

Human-in-the-loop controls require evaluation across specific dimensions. Multiple notification channels ensure that policy violations and escalations reach humans through the channels they actually monitor, not just one. Async approval mode means the agent does not hold an open socket waiting for a human response, which prevents HITL from becoming a latency bottleneck. High-risk actions in regulated environments should require multiple approvers. Full context capture on approval requests, including arguments, agent justification, and matched policy, is essential because approvers making decisions on minimal information will make worse decisions, consistently.

The practical verification: can the gateway prove and bound what an agent did, with its own identity, a full audit trail, and a defined scope of impact? If a vendor cannot demonstrate this in a proof-of-concept, the governance claim won't survive a real security review.

Observability Criteria: The Difference Between a Log and a Working Operations Layer

An audit log answers "what happened." Operational observability answers "what is happening now" and enables intervention before the incident report is written. These serve different functions. A gateway must provide both, and conflating them during evaluation leads to discovering the gap at the worst possible time.

A full observability layer captures, per agent run, every tool call, every model call, every policy decision, every approval, every error, with timestamps, costs, and linked artifacts enabling drill-down from fleet view to individual session. Fleet view answers what all agents have done this week, which tools are most called, where costs are concentrating, and which agents are generating policy violations. Session view answers why a specific run failed, what the exact tool call sequence was, and where the agent deviated from expected behavior. The fleet view without session depth is just a dashboard. Session depth without fleet context means you are always investigating the wrong incident.

Integration requirements create long-term friction when underspecified during evaluation. OpenTelemetry export is the de facto standard for trace data; a gateway that cannot emit OTEL-compatible traces will require bespoke integration with every monitoring tool the organization already runs. Compatibility with existing enterprise monitoring stacks, whether Datadog, Splunk, Grafana, or others, should be native.

Real-time versus batch observability is a distinction worth verifying explicitly. Observability data arriving with a multi-minute delay does not support intervention. Confirm whether the gateway's observability pipeline is real-time or near-real-time, and what the latency guarantee is under production load, not just under demo conditions.

Cost attribution is an observability output that procurement teams overlook during evaluation and then urgently need six months into deployment. Token consumption and tool-call costs should be attributable to specific agents, teams, and use cases. Without this, cost governance is impossible and ROI measurement becomes guesswork dressed up as analysis.

The demo question worth asking: can an operator go from a fleet-level anomaly to the specific session and tool call that caused it, in under five minutes? That end-to-end trace capability is what separates operational observability from log storage with a nice interface.

Integration Depth Criteria: Where Most Gateway Evaluations Underestimate Future Friction

The agent gateway segment is projected to grow at a 36.48% CAGR from 2026 through 2035, according to SNS Insider. That trajectory matters because a gateway purchased to solve today's LLM routing problem will lack the MCP and agent-to-agent protocol support the organization needs within 12 months. Evaluating only for present requirements is how you end up replacing infrastructure you just bought.

Protocol support requires explicit verification. MCP support should be validated against the March and June 2025 specification revisions, not against an earlier version that predates OAuth 2.1 support. Agent-to-Agent protocol support matters as multi-agent architectures become standard; inter-agent traffic needs the same governance as agent-to-tool traffic, and a gateway that handles only the latter creates a governance gap the moment multi-agent orchestration is deployed. Standard REST and webhook support is equally necessary because existing internal systems will not be MCP-native, and the gateway must bridge to them without requiring bespoke connectors for every integration.

Identity provider integration is where many evaluations miss a consequential detail. SSO compatibility with whatever IdP the enterprise already runs, whether Okta, Azure AD, or another provider, is a baseline requirement. An agent gateway that cannot consume existing identity infrastructure creates a parallel identity management problem, which is precisely the opposite of what a control plane should do.

The vendors active in this space include established infrastructure players extending their platforms into agent governance and purpose-built agent gateway providers. Among the purpose-built options, Pangea and Cequence have addressed specific security and API governance dimensions, while Tyk has extended its API management heritage toward agent traffic. Portia AI has built specifically for the agent gateway use case rather than adapting from an adjacent category, which affects architectural fit at the protocol level and the integration debt that accumulates over time.

One scenario integrates everything above cleanly and is worth running in any proof-of-concept: ask each vendor to walk through a new agent being provisioned, given scoped tool access, making a policy-violating call, being blocked in real time, and generating a tamper-resistant audit record, while that session is simultaneously visible in the operational observability layer and attributable to a cost center. Against your actual identity provider. Against your actual monitoring stack. A gateway that can complete that scenario in a proof-of-concept can handle production. One that can't complete it under controlled conditions will not somehow improve under load, with real data, when the stakes are real.