MADBench - Measuring Agentic DBMS: Quantifying the Protocol Tax of MCP-Mediated Query Workloads

Abstract

Large language model agents increasingly access relational databases through the Model Context Protocol (MCP), a tool-calling standard in which the model discovers tool definitions from a middleware server, issues calls through it, and consumes query results serialized as text into its context window. Under this architecture, every database interaction pays costs that classical database clients do not: tool discovery, server lifecycle, result serialization, transport, and context-window consumption. Existing agent benchmarks report end-to-end task scores and therefore cannot attribute these costs to the layer that causes them.

This thesis presents MADBench, a benchmark that measures and decomposes this overhead, termed the Protocol Tax: end-to-end interaction time minus raw database execution time. OpenTelemetry instrumentation splits the tax into seven protocol phases (discovery, lifecycle, handover, serialization, parsing, transport, and teardown), with the unattributed residual held below 5% of wall-clock time on 98.7% of runs. Two measurement tracks separate protocol effects from model effects. Track A replays fixed TPC-H queries through three open-source MCP servers over PostgreSQL, MySQL, and DuckDB at scale factors 1 through 10, under paired warm-start and cold-start conditions, collecting 5,378 traces with no model in the loop. Track B runs a fixed LLM agent over 30 pre-registered natural-language questions on the same server-backend grid, yielding 1,080 attempts scored against gold SQL results.

Three findings emerge. First, Protocol Tax spans three orders of magnitude on identical queries, from 10⁻³ of database execution time for an in-process engine to 3× it for a subprocess server on PostgreSQL; the dominant cost is the server's process model, not its wire format. Second, protocol speed does not predict agent success: the server with the lowest Protocol Tax ranks last in agent completion, and DuckDB carries 112× less protocol overhead than the client-server backends yet finishes mid-pack. Third, completion is instead governed by result-payload compactness and schema-metadata quality; one server-backend pairing collapses from 61.1% to 17.8% completion between scale factors when verbose query results overflow the model's context window. The benchmark, pre-registered workload, and full trace corpus are released to support reproduction and extension.