AI Agent Optimisation

After proxying your AI agent's LLM calls through MockServer, you can export a structured optimisation brief — a pre-framed Markdown document covering both spend reduction and agent design. It tells you where inference money is wasted on redundant tokens, and where the architecture has reliability or design problems: tool calls that could bypass the model entirely, bloated tool surfaces that give the model more opportunities to choose wrongly, and retry patterns that repeat work unnecessarily.

MockServer analyses the captured traffic offline, computing nine deterministic optimisation signals along with token counts and estimated USD costs from provider pricing tables. The signals fall into two categories: cost and token efficiency (repeated system prompts, low cache-hit rates, output bloat, model overspend) and process, reliability and design (deterministic tool calls that could be replaced by a direct endpoint, oversized tool results, unused tool schemas, duplicate consecutive calls). MockServer never calls an LLM itself — every number in the report is deterministic and computed locally from your captured traffic. An in-product verdict (A–F grade and a "$X recoverable" headline) summarises the findings at a glance.

How It Works

1. Start MockServer as a proxy and capture LLM traffic — including real traffic from a headless OpenCode run
2. Export the optimisation brief — via the dashboard, MCP tool, or REST endpoint
3. Read the verdict — instant A–F grade and "$X recoverable" headline, no LLM required
4. Act on the detected opportunities — nine signal types covering cost, reliability, and design, each mapped to a concrete lever

1. Start MockServer as a Proxy and Capture Traffic

Any MockServer instance can act as an HTTPS proxy — no special mode is needed. See AI Traffic Inspection for the full setup, including how to trust the MockServer CA certificate and configure standard environment variables for Node.js, Python, and other tools.

The quick-start for local use:

docker run -d --rm -p 1080:1080 mockserver/mockserver

export HTTPS_PROXY=http://localhost:1080
export NODE_EXTRA_CA_CERTS=/path/to/mockserver-ca.pem   # Node.js tools
export SSL_CERT_FILE=/path/to/mockserver-ca.pem          # Python tools

# Now run your agent — its LLM calls are captured automatically
claude   # or: opencode, python my_agent.py, etc.

MockServer captures every LLM request/response pair (including SSE-streamed completions) in its event log, and these are the source the optimisation report is built from. Both proxied/forwarded traffic (to a real provider) and mocked LLM responses served by MockServer itself are analysed — so you can optimise against real captured runs or against mocked conversations (for example, the data created by npm run demo). LLM traffic is recognised by request shape (the provider's API path), so it works regardless of the upstream host.

Generate real traffic with OpenCode

The offline npm run demo dataset includes a crafted, deterministic agent run that fires every optimisation signal — great for a first look. To analyse your own agent's real spend instead, proxy a headless OpenCode run through MockServer and pointed at a real provider.

Heads-up: this generates real, non-deterministic LLM traffic — it needs network access and your own provider API key, and the captured run varies each time. It is not wired into npm run demo, which stays offline. Never commit or share API keys.

The repository ships a helper script, mockserver-ui/scripts/demo-opencode-proxy.sh, that starts MockServer as an HTTPS proxy with a machine-local CA, extracts that CA, and prints the exact environment to run OpenCode through it:

# Start the proxy and print the run instructions
./mockserver-ui/scripts/demo-opencode-proxy.sh

# …or start the proxy AND run a one-shot OpenCode prompt through it
./mockserver-ui/scripts/demo-opencode-proxy.sh "summarise the README and suggest one improvement"

The equivalent done by hand, so you can run OpenCode in your own shell:

# 1. Start MockServer as an HTTPS proxy with a unique, machine-local CA
#    (the default CA private key is public — never trust it for real traffic).
docker run -d --rm --name mockserver-proxy -p 1080:1080 mockserver/mockserver \
  -serverPort 1080 \
  -Dmockserver.dynamicallyCreateCertificateAuthorityCertificate=true \
  -Dmockserver.directoryToSaveDynamicSSLCertificate=/dynamic-certs

# 2. Trigger CA generation, then copy the CA out of the container
curl -sk https://localhost:1080/ >/dev/null
docker exec mockserver-proxy cat /dynamic-certs/CertificateAuthorityCertificate.pem > mockserver-ca.pem

# 3. Route OpenCode through the proxy and trust the CA
export HTTPS_PROXY=http://localhost:1080
export NODE_EXTRA_CA_CERTS=$PWD/mockserver-ca.pem   # OpenCode is a Node.js tool
export SSL_CERT_FILE=$PWD/mockserver-ca.pem          # any Python helpers

# 4. Point OpenCode at a REAL provider with YOUR OWN API key, then run it headless
export OPENAI_API_KEY=sk-...            # or ANTHROPIC_API_KEY=sk-ant-...
opencode run "summarise the README and suggest one improvement"

OpenCode's LLM calls are now captured by MockServer. Open the dashboard LLM Optimise tab (immediately after Chaos in the navigation bar), or curl the report, to see signals computed from your real run:

curl -s "http://localhost:1080/mockserver/llm/optimisationReport?format=markdown"

2. Export the Optimisation Brief

Three ways to export — choose whichever fits your workflow:

Dashboard — LLM Optimise Screen

Open the MockServer dashboard and click LLM Optimise in the navigation bar (it sits immediately after Chaos):

http://localhost:1080/mockserver/dashboard

The LLM Optimise screen shows:

• A session picker — pick a single upstream host, or All captured LLM traffic (the default)
• A verdict banner — grade letter (A–F, colour-coded) with the "$X recoverable (Y% of spend)" headline and a one-line rationale
• Hero cards — total estimated cost, input/output tokens, call count, average latency, cache-hit rate, and one-shot rate
• A Detected Opportunities panel listing each signal with severity chip, estimated saving, and structured fix guidance
• A per-call table
• Copy verdict — copies a compact plain-text grade + fix summaries to the clipboard (built from the loaded JSON, no extra fetch)
• Copy optimisation brief — copies the full Markdown brief to the clipboard
• Download bundle — saves the structured JSON report

MCP Tool

If your AI agent is connected to MockServer's MCP control plane (see MCP Setup), use the export_optimisation_report tool:

{
  "method": "tools/call",
  "params": {
    "name": "export_optimisation_report",
    "arguments": {
      "format": "markdown"
    }
  }
}

{
  "method": "tools/call",
  "params": {
    "name": "export_optimisation_report",
    "arguments": {
      "format": "json",
      "host": "api.openai.com"
    }
  }
}

{
  "method": "tools/call",
  "params": {
    "name": "export_optimisation_report",
    "arguments": {
      "format": "csv"
    }
  }
}

REST Endpoint

Call GET /mockserver/llm/optimisationReport directly. This is a MockServer control-plane endpoint; CORS is enabled so the dashboard UI can call it even when the dashboard and control plane are on different hosts or ports.

curl -s "http://localhost:1080/mockserver/llm/optimisationReport?format=markdown"

curl -s "http://localhost:1080/mockserver/llm/optimisationReport?format=json" | python3 -m json.tool

curl -s "http://localhost:1080/mockserver/llm/optimisationReport?format=csv"

# OpenAI-evals JSONL (one {"input":[...],"ideal":"..."} sample per line)
curl -s "http://localhost:1080/mockserver/llm/optimisationReport?format=openai-evals"

# Chat fine-tune JSONL (one {"messages":[...]} conversation per line)
curl -s "http://localhost:1080/mockserver/llm/optimisationReport?format=fine-tune"

# promptfoo test suite (a single {"tests":[...]} JSON document)
curl -s "http://localhost:1080/mockserver/llm/optimisationReport?format=promptfoo"

curl -s -X PUT "http://localhost:1080/mockserver/llm/diffRuns" \
  -H "Content-Type: application/json" \
  -d '{
        "before": {"session": "host:api.openai.com", "provider": "OPENAI"},
        "after":  {"session": "host:api.openai.com", "provider": "OPENAI"}
      }'

# Only OpenAI traffic
curl -s "http://localhost:1080/mockserver/llm/optimisationReport?format=markdown&host=api.openai.com"

# Only Anthropic traffic, JSON
curl -s "http://localhost:1080/mockserver/llm/optimisationReport?format=json&provider=ANTHROPIC"

# A specific named session
curl -s "http://localhost:1080/mockserver/llm/optimisationReport?format=markdown&session=host%3Aapi.openai.com"

Query Parameters

If no LLM traffic has been captured yet, the endpoint returns HTTP 200 with an empty report (JSON) or a brief that says "no LLM traffic captured" (Markdown).

What the Markdown Brief Looks Like

The brief is structured in a fixed order so the downstream LLM can reason about it efficiently:

1. Framing preamble — tells the LLM it is an AI agent optimisation expert and what to do
2. Verdict — A–F grade, rationale, estimated recoverable spend and token count, cache-hit rate, one-shot rate
3. Run summary — provider(s), model(s), call count, token totals, estimated cost, latency, tool-call count, cache-hit rate, one-shot rate
4. Per-call table — one row per call: model, input/output tokens, cost, latency, tools, finish reason. For proxied/forwarded traffic the per-call latencyMs is the measured upstream round-trip time (full-stream duration for streaming responses). It is 0 only when the upstream time could not be captured for that call.
5. Detected opportunities — each detected signal as a section with title, detail, affected call indices, estimated saving, recommendation, and fix guidance (action, config snippet or example expectation, and a docs link where available)
6. Conversations and tool definitions (appendix) — redacted messages and tool schemas so the LLM has the raw material to propose concrete edits

Once you have the brief, paste it into any LLM. No additional context is needed.

3. The Verdict

Every report includes a deterministic verdict — an A–F grade and a "$X recoverable" headline — computed from the detected signals without any LLM call. The grade is your quick answer to "how much is there to fix here?"

The "recoverable" figure is an estimate of how much spend could be saved by acting on the detected signals. It is calculated using per-call MAX attribution: each call's contribution is the maximum saving across all signals that affect it (not a sum), so the total can never exceed the actual session spend. Treat it as a directional estimate, not an invoice.

The rationale line explains the grade in plain English, for example: "Grade C — an estimated 18% of spend ($1.42) is recoverable across 3 findings (1 high, 2 medium)."

Session KPIs

Two new headline metrics appear in the hero cards and run summary:

• Cache hit rate — the fraction of input tokens served from cache (cachedInputTokens / inputTokens). A rate below 50% on a session with repeated prompts suggests caching is not enabled.
• One-shot rate — the fraction of calls that were not retries. A windowed retry detector (window of 3) identifies calls that duplicate a recent request; a low one-shot rate points to retry loops or duplicate request patterns worth de-duplicating.

4. The Nine Optimisation Signals

MockServer analyses the captured calls and emits up to nine deterministic signals, sorted by urgency (a combination of severity and how many calls are affected). Each signal names the problem, quantifies it in tokens and estimated USD, and provides structured fix guidance — including a copy-paste config snippet or example expectation where relevant.

The signals cover two complementary concerns:

• Cost and token efficiency — signals where spend can be reduced by caching, tightening prompts, or routing simpler work to a cheaper model.
• Process, reliability and design — signals where the agent's architecture is the problem: a step that does not need an LLM, a tool surface that is too wide, a tool that returns too much, or retries that repeat work. Fixing these makes the agent more predictable and robust; the cost saving is a consequence, not the headline.

The report lists signals in urgency order (severity × breadth). The Category column below shows which concern each signal addresses — cost savings are immediate, but the process/design signals often have the larger long-term impact on reliability.

Migrating Deterministic Steps into Tools

When DETERMINISTIC_TOOL_CALL or UNUSED_TOOL_SCHEMA fire, the underlying question is whether a step belongs inside the model at all. The captured traffic is the evidence: if a tool call always returns the same result for the same inputs, or if a tool in the schema is never called, these are signals that the tool surface or the call graph needs rethinking.

What to look for: examine the tool names and argument patterns reported in the brief. A tool called repeatedly with identical arguments is a candidate for a direct endpoint call — one that bypasses the model and feeds the result back deterministically. A tool that appears in the schema but never in the call log is dead weight: it costs tokens on every request and widens the surface the model can choose from, increasing the chance of an unexpected selection.

What MockServer analyses: the report works entirely from observed traffic. It does not read prompt files, configuration, or any agent source. A DETERMINISTIC_TOOL_CALL finding tells you a step was deterministic in this captured run — whether that is always true in production is a judgement only you can make from context. Treat the signal as a prompt to investigate, not an automatic prescription.

Acting on a finding:

• For a deterministic lookup (weather, database read, config fetch) that is always routed through the model: replace the tool with a direct HTTP call in your agent code, and return the response to the model as a message rather than going via a tool call. The fix guidance in the brief includes a MockServer expectation you can use to stub the endpoint in tests.
• For an unused tool: remove its definition from the tools array in the request. This reduces input tokens on every call and narrows the model's decision space. If the tool is needed for some sub-tasks but not others, consider passing it conditionally rather than sending the full schema on every turn.

Cost savings follow from these changes, but the more important outcome is a more predictable agent: one that does not depend on sampling to do something deterministic.

Redaction

The optimisation report always strips sensitive headers before including them in either the JSON bundle or the Markdown brief:

• Authorization (Bearer tokens, Basic auth)
• x-api-key / api-key
• Cookie / Set-Cookie
• Proxy-Authorization

Body fields are redacted according to the mockserver.fixtureBodyRedactFields configuration property — the same setting used by record_llm_fixtures. The report includes a redaction object describing what was stripped, so you know exactly what the downstream LLM will not see.

Body content is not automatically redacted beyond the configured field list. Review the Markdown brief before pasting it into an external LLM if your prompts or tool results contain sensitive data not covered by fixtureBodyRedactFields.

Configuration

Current Limitations

• Observed traffic only — MockServer analyses what it captured as a proxy. It has no access to your agent's source code, prompt files, or configuration. The DETERMINISTIC_TOOL_CALL signal, for example, tells you a tool call was deterministic in this run; whether that holds in all runs is something only you can determine. There is no mechanism for parsing or auditing prompt instruction files (sometimes called "skills" or "rules") directly — the signals are the observable evidence from traffic, and you act on them with knowledge of your own codebase.
• Cost estimates — costs are computed from MockServer's built-in provider pricing tables (LlmPricing), which may lag provider price changes. The costIsEstimated field in the JSON bundle is true when the provider did not return real usage tokens and MockServer estimated them from the decoded text; it is false when the provider returned real usage data. Treat all cost figures as directional estimates.
• Non-LLM traffic — only traffic where MockServer can detect a supported LLM provider (OpenAI, Anthropic, Gemini, Bedrock, Azure OpenAI, Ollama), by API path shape, is included — whether mocked or proxied. Other traffic is ignored.
• Session grouping — in v1, the optimisation report groups proxied LLM traffic by upstream host, so the dashboard LLM Optimise session picker offers host-based sessions (and "All captured LLM traffic"). The session REST parameter accepts either the composite host:<host> key or the bare host.

Related Pages

• AI Traffic Inspection — proxy setup, CA trust, and how traffic is captured
• MCP Setup — connect your AI agent to MockServer's MCP control plane to use the export_optimisation_report tool
• MCP Tools Reference — full parameter documentation for export_optimisation_report and related tools
• LLM Response Mocking — mock LLM API responses for deterministic, offline testing
• Configuration Properties — mockserver.llmOptimisationMaxCalls and mockserver.fixtureBodyRedactFields

AI Integration — See Also

• MCP Setup — connect Claude Code, Cursor, Windsurf, Cline, or OpenCode to MockServer's built-in MCP endpoint
• MCP Tools Reference — full documentation of all MCP tools, parameters, and resources
• Debugging with AI — workflows for using AI assistants to debug HTTP traffic via MCP
• AI Traffic Inspection — inspect and record LLM/MCP traffic for debugging and deterministic replay
• OpenAPI Contract Verification — verify recorded traffic and run contract/resiliency tests against an OpenAPI spec
• OpenAPI for AI — use MockServer's OpenAPI spec as a fallback for AI tools without MCP support
• AI Protocol Mocking (MCP & A2A) — mock MCP servers and A2A agents your AI application depends on
• LLM Response Mocking — mock LLM API responses from OpenAI, Anthropic, Gemini, Bedrock, Azure OpenAI, and Ollama with provider-correct formatting, streaming, conversations, and chaos
• AI Agent Optimisation — export a structured optimisation brief from captured LLM traffic covering both spend reduction and agent design: detect repeated tokens, deterministic tool calls that could bypass the model, bloated tool surfaces, and retry patterns
• llms.txt — machine-readable index of MockServer documentation for AI assistants and LLMs