Enterprise MCP DeploymentLong read
Closing the Observability Gap Between What MCP Agents Request and What They Actually Retrieve
MCP logs that agents called tools, not what those tools returned.
· 10 min read

Why MCP standardized the call but not the answer
- Role: Opens the argument by establishing the foundational design choice that creates the gap — MCP governs invocation, not outcome — so every subsequent section has a structural cause to point back to.
- MCP gave AI agents a universal way to discover and call tools but defined no standard for reporting what those calls actually returned — no trace format, no eval interface, no convention for token usage, cost, or model identity on tool-call responses.
- The protocol's three core primitives, Tools, Resources, and Prompts — are all about capability and communication, not accountability; the spec is articulate about what agents can do and silent about what they did do
- This was a deliberate scope decision, not an oversight: interoperability came first, and observability was left to the ecosystem
- The consequence is that every observability tool invented its own schema, and none of them talk to each other — teams A, B, and C on the same product generate traces in three incompatible formats, making cross-team cost or behavior analysis impossible without custom adapters
- The parallel to pre-OpenTelemetry microservices is direct: before OTel converged the ecosystem, every APM vendor had its own trace format; MCP's observability ecosystem is at that same inflection point now
- Note: Keep this section tight — it sets the stage, it does not survey the market. Close on the design gap, not on adoption statistics.
What a successful HTTP 200 actually tells you about an agent
- Role: Sharpens the claim from the previous section into a precise, falsifiable problem — the metric enterprises rely on to confirm success is structurally blind to the failures that matter most — before the article shows why.
- A successful request status tells an enterprise almost nothing about what data an agent touched, leaked, or acted on — because agents fail in ways that look like success.
- Traditional APM instruments requests; agents produce reasoning chains — a single agent execution is a sequence of decisions, tool invocations, and intermediate results, not one call
- Four failure modes that return HTTP 200 and look healthy to every conventional monitoring tool:
- PII leaked in a response (the status was fine; the content was not)
- A hallucinated citation returned as a valid tool result
- A prompt-injection instruction followed silently
- The request-versus-retrieval divergence at the tool-description layer compounds this: in a survey of tool descriptions, the vast majority had at least one defect; because the agent selects tools from names, descriptions, and schemas alone, a tool can return a valid JSON result from the wrong operation entirely, and no current logging layer catches the mismatch
- Binary pass/fail monitoring is blind to all of it — and the gap is wide: 89% of software teams have production observability in place, yet AI agents remain largely unobserved — step-level tracing — recording each reasoning step, tool call, and model response as a nested span — is the minimum viable signal.
- Note: The cost case from the sources is useful here as a concrete anchor, but do not state the dollar figure — describe it in magnitude terms ("a monthly bill large enough to alarm any finance team") and move on. The point is invisibility, not the specific number.
The three-identity problem MCP's audit trail cannot solve on its own
- Role: Moves from the generic observability gap to its specific governance consequence — an audit record that names software and an account but never the human who decided — escalating the stakes for enterprises that believe their policies already cover them.
- Every agent action involves three identities — the agent, the account, and the person who caused it — and MCP's audit record captures at most two of them.
- The fragmented-log problem multiplies this: each MCP server writes only what happened in its own app, so the question an auditor asks first — what one agent did for one person across everything it accessed — has no single place where the answer exists
- Note: This section should feel like a structural indictment, not a compliance checklist. The approval-session story is the sharpest illustration — lead with it, then explain why it is a logging problem, not just an authentication problem.
How real incidents exposed the retrieval layer as the actual attack surface
- Role: Grounds the abstract gap in named, dated incidents — shifting the reader from "this could happen" to "this has happened, repeatedly, and the logging layer was part of why it went undetected."
- In each of the significant MCP security incidents documented in 2026, the logging that existed was not sufficient to detect what the agent actually retrieved or acted on — the gap was not theoretical.
- Pillar Security / Deadbugz campaign (August 2026): a malicious server stayed harmless for its first three tool calls and then instructed the agent to collect SSH keys and cloud credentials; because the safety check lived inside the agent's own software, an attacker who controlled the agent's infrastructure could disable it silently and nothing downstream noticed — the request log showed normal tool calls throughout
- Dolt MCP authentication bypass (August 2026, versions 0.3.1–0.3.6, fixed in 0.3.7): requests with invalid tokens got a 401 response but kept processing; an unauthenticated caller could run database tools with the server's credentials while the status line told the monitoring layer access was denied
- The pattern across all five: the request layer reported normal behavior; the retrieval layer — what was actually returned, acted on, or exfiltrated — left no trace that existing logging could surface
- Note: Present these as a pattern, not a list. After each case, say the one thing the monitoring layer missed. The pattern is the argument.
Why native MCP server logging cannot close this gap alone
- Role: Anticipates the obvious objection — "just enable logging on the MCP server" — and shows why the default logging architecture is structurally insufficient before introducing what instrumentation at the retrieval layer actually requires.
- The logging that MCP servers produce by default is session-scoped, ephemeral, and server-local — it cannot answer the cross-session, cross-server questions that governance and security actually require.
- There is no unified view across multiple MCP servers, no retention policy, and no mechanism to correlate events that span different services — security teams describe this as a visibility black hole
- The multi-server deployment makes it worse: logs split by server, each in its own format, so a cross-server audit requires collecting and merging four logs in four formats after the fact
- The semantic quality failures that matter most — wrong tool selected, PII in a response, hallucinated result accepted by the agent — are invisible to heuristic logging entirely; logging that a tool was called does not record whether the agent chose correctly or what the returned data contained
- Objection: "A proxy gateway captures everything passing through it" — answer: each gateway defines its own telemetry format, so data collected by one cannot be queried from another's dashboard; teams that deploy multiple gateways have traded tool-level fragmentation for gateway-level fragmentation
- Note: This section exists to close the escape hatch before the next section opens the real solution. Keep it brief — two or three paragraphs — and end on the point that the problem is architectural, not configurational.
What observability instrumented at the retrieval layer actually looks like
- Role: Pivots from diagnosis to solution — defining concretely what retrieval-layer observability means in practice, what it captures that request-layer logging cannot, and what the emerging standards and tools make possible today.
- Closing the gap requires instrumentation that records the content, cost, and reasoning context of what a tool returns — not just whether the call succeeded — and two converging developments make this buildable now.
- OpenTelemetry v1.39 (January 2026) introduced MCP semantic conventions — mcp.method.name, mcp.session.id, mcp.protocol.version, at Development (experimental) stability; these conventions have since moved to the dedicated semantic-conventions-genai repository — so the tool layer that was previously a black box in agent traces is now addressable; an agent calling ten MCP servers can be traced as cleanly as one calling a single local function
- As of v1.41, the spec defines agent, workflow, tool, and model spans plus streaming latency and token-usage metrics; two histogram metrics are the production floor: gen_ai.client.operation.duration and gen_ai.client.token.usage
- Critically, all gen_ai.* attributes carry Development stability badges — attribute names can change without a major version bump — and in June 2026 the GenAI conventions moved to a dedicated repository to iterate independently; as of August 2026, that repository has no official release, so the vocabulary exists but stability does not yet
- The opt-in tension: tool argument and message attributes — the most valuable signals for detecting PII leakage or prompt injection — are deliberately opt-in because they may carry sensitive data; more visibility creates its own governance burden, and teams must decide which attributes to collect and under what retention policy
- Google Cloud MCP Toolbox shows what retrieval-layer tracing looks like when properly instrumented: enabling telemetry during a tools/call produces a distributed trace mapping the entire path from client request to database, with W3C traceparent injected into the MCP request's params._meta field so server and client spans are fully correlated — client latency, server latency, and raw database processing time become distinct, queryable signals
- Step-level tracing — each reasoning step, tool call, and model response as a nested span — is the minimum viable signal; a health check that reports only "up" is close to useless for an agent in production
- Note: This is the most technical section. Write it for a technical decision-maker, not a developer — explain what each capability enables operationally, not how to implement it. The Google Cloud and ToolHive examples are the two concrete anchors; use them to show what "retrieval-layer instrumentation" means in practice.
What retrieval-layer visibility enables that request-layer logging never could
- Role: Shows the positive case — what becomes possible once the gap is closed — connecting observability instrumentation back to the governance and compliance obligations that make this a business imperative, not just an engineering preference.
- Real-time visibility into what an agent actually retrieved is not just a debugging tool — it is the precondition for regulatory traceability, cost accountability, and the kind of access governance that lets organizations say yes to production deployments.
- Cost accountability becomes possible: token usage per query, per tool, per agent, per session becomes a queryable signal rather than a disappearing number inside a tool's internal LLM call
- Security response changes character: instead of reading an audit log after an incident — which is a postmortem, not a safety net — real-time retrieval-layer telemetry surfaces anomalous tool selections, unexpected data returns, and prompt-injection patterns as they happen
- The EchoLeak case makes the contrast concrete: organizations with content-level inspection of agent outputs would have seen the outbound requests carrying stolen data as distinct from normal Copilot API calls; those without it had no signal at all
- Governed observability also reduces shadow AI: when teams have a governed path to deploy agents with observable tool access, they are less likely to route around controls; the observability layer is what makes "yes" to production politically possible for risk-conscious organizations
- A gateway with retrieval-layer instrumentation — one that enforces access policy, records what was returned, and surfaces that record in real time — is what turns an MCP deployment from an experiment into a production-grade system an enterprise can audit, defend, and extend
- Note: This is where the company's positioning applies directly and naturally — observability as the enabler of speed, not the brake on it. Weave it in as the operational conclusion, not as a product pitch. Close on the governance-as-competitive-advantage framing.
Where the standards work stands and what enterprises should do before it stabilizes
- Role: Lands the piece practically — giving the reader an honest picture of what is settled, what is still in motion, and what decisions they can make now without waiting for the spec to reach stable status.
- Because the OTel GenAI conventions are in Development and MCP's own observability story is still being written, enterprises face real decisions now about what to instrument, on what schema, and at what layer — and waiting is itself a choice with consequences.
- What is settled: OTel v1.39's MCP semantic conventions exist and are usable; the two mandatory histogram metrics (operation duration, token usage) are the floor any team can instrument today without betting on attribute stability
- What is not settled: nearly all gen_ai.* attributes carry Development badges; the dedicated GenAI conventions repository has no official release as of August 2026; attribute names can change without a major version bump
- The practical decision framework:
- Instrument the retrieval layer now, even imperfectly: a schema that captures tool name, arguments, returned content (opt-in), token cost, and session ID gives a team more governance than waiting for a stable spec
- Treat tool argument and message attributes as opt-in governance decisions, not technical defaults — which attributes to collect, under what retention policy, and with what access controls is a data governance question before it is an engineering one
- The organizations that instrument observability before incidents — rather than assembling postmortems from fragmented server logs after them — are the ones that can move fast in production without moving blind
- Note: End on a forward-looking but grounded note. Do not oversell stability that the sources do not confirm. The honest message is: the foundation exists, the vocabulary is there, the stability is not — instrument deliberately now and plan to migrate attribute names as the spec matures.
Sources
Filed underEnterprise MCP Deployment
More in Enterprise MCP Deployment
Testing MCP Server Integrations Before Production Deployment
Camille Osei-Bonsu
MCP Server Registry Design and Implementation
Priya Subramaniam
Centralizing MCP Access Policy Enforcement Without Fragmenting It Across Individual Servers
Tobias Wrenfield


