MCP Stateless Explained: Evals That Stick
MCP tool calls are up 98x in 2026 with a new stateless spec. What changed in July, how LangChain adapted, and a minimal eval loop for MCP servers.
On this page
MCP tool calls from ChatGPT users are up 98x across 2026, and the protocol just got its biggest rewrite since launch. On September 3 LangChain moved MCP into its main package on top of FastMCP to match the July stateless spec. If you ship tools for agents, here is what changed and the smallest eval loop that keeps servers reliable.
Takeaways
July spec: stateless core, cacheable tool lists, elicitation as retryable interrupts: reliability and scale fixes.
LangChain shift: langchain[mcp] with MCPAdapter, FastMCP transports, tool-name prefixing per server.
Eval rule: test what the model does with your server, not just protocol shape: tool choice, order, payloads, recovery.
Solo loop: 5 scenarios, assert tools called, judge one quality rubric, run on every change.
What did stateless actually fix?
The old pain: sessions pinned to connections, so a redeploy killed live runs and every agent run re-fetched the tool catalog. The new core removes pinning, nothing left to pin, plus two practical wins: servers declare how long tool lists stay fresh, and mid-call questions become ordinary retried requests instead of held-open connections.
before: connect -> pin session -> fetch tools every run -> hold connection for questions
after: connect -> use cached catalog per TTL -> elicitation as interrupt + retry -> redeploy safeLangChain in one snippet
MCPAdapter collapses the old multi-client setup into one class with list_tools(cache_mode="use"), namespaced tools like billing_search, and elicitation surfaced as LangGraph interrupts. Python today, TypeScript next.
How do you eval an MCP server?
Pick 5 real scenarios
One happy path, one missing-parameter elicitation, one wrong-tool trap, one error recovery, one latency check. If a scenario never happens in prod, it does not belong in the gate.
Assert behavior, not just output
Check tools.was_called, called_with, and sequence: did the agent call fetch with the right URL before answering? Content checks alone miss the most common MCP bug: right answer, wrong tool.
Measure cost per success
Record tokens, latency, and iterations per passing run. A server whose description trims one retry per call pays for its own evals within a week.
Run it in CI, inspect one trace by hand
Tools like mcp-eval give JSON reports and badges for regressions; local workbenches show every reasoning step and token of context. Automate the gate, eyeball the failure that matters.
What makes a tool description agent-proof?
One change, re-run, watch the metric, that is the whole loop. Tighten the description, fix a parameter schema, shrink the payload, then re-run the same 5 scenarios. The score moving from "fetch called 60% of runs" to "fetch called 100% with correct args" is worth more than any new tool.
What should you skip?
Should I support both MCP eras?
Yes, via negotiation: FastMCP tries the new protocol and falls back to the old handshake automatically. Your code does not branch; the client handles the era split.
Do I need a full eval platform on day one?
No. Start with decorator-style tests: drive the agent, assert tool calls, check one LLM-judged quality bar. Graduate to datasets, OpenTelemetry traces, and PR gates once the server has real users.
As of September 17, 2026: MCP won the tool protocol war, which moves the competition to reliability: cached catalogs, clean elicitation, and evals that prove the agent uses your server correctly. Tighten one description today and re-run the suite. Pair with permission-scoping patterns to keep those tools leashed.