Mock LLM APIs
(OpenAI, Anthropic, Gemini)

When you test an application that calls an LLM, you face two problems: the real API is slow, costs money, and returns different results every run. MockServer solves both: tell it which provider your application targets and what the completion should contain, and it returns a byte-for-byte correct response — including the full streaming SSE framing — without touching the network.

In practice: point your application at MockServer instead of api.openai.com (or the equivalent endpoint for your provider), create one expectation, and your test gets a deterministic, instant, free LLM response every time.

# Start MockServer
docker run -d --rm -p 1080:1080 mockserver/mockserver

# Create an OpenAI-compatible completion mock
curl -s -X PUT http://localhost:1080/mockserver/expectation \
  -H "Content-Type: application/json" \
  -d '{
    "httpRequest":  { "method": "POST", "path": "/v1/chat/completions" },
    "httpLlmResponse": {
      "provider": "OPENAI",
      "model": "gpt-4o",
      "completion": {
        "text": "MockServer is a free, open-source HTTP mock server.",
        "stopReason": "stop",
        "usage": { "inputTokens": 12, "outputTokens": 9 }
      }
    }
  }'

# Your application can now call http://localhost:1080/v1/chat/completions
# and receive a correctly formatted OpenAI response.

MockServer can mock LLM API responses from any major provider using the httpLlmResponse action. You describe the completion in a provider-neutral format and MockServer encodes it into the correct wire format for the target provider — headers, JSON structure, streaming framing, and all. This lets you test AI-powered applications deterministically, without calling a real LLM.

• Basic completion mock
• Supported providers
• Streaming responses
• Tool calls
• Embeddings
• Rerank
• Multimodal request recognition (images and audio)
• Multi-turn conversations
• Session isolation
• Cost budget
• Moderation, content filter & refusal
• Chaos / fault injection
• Realtime voice APIs (OpenAI Realtime, Gemini Live)
• Streaming — client examples (all languages)

Basic Completion Mock

Create an expectation with an httpLlmResponse action. The provider tells MockServer which wire format to produce; the completion describes what to return.

The Java client has a typed fluent API (LlmMockBuilder + HttpLlmResponse). All other clients send the expectation as raw JSON using their generic expectation method (mockAnyResponse in JavaScript; a direct PUT to /mockserver/expectation in other languages).

import static org.mockserver.client.LlmMockBuilder.llmMock;
import static org.mockserver.model.Completion.completion;
import static org.mockserver.model.Provider.OPENAI;
import static org.mockserver.model.Usage.usage;

// OpenAI — POST /v1/chat/completions
llmMock("/v1/chat/completions")
    .withProvider(OPENAI)
    .withModel("gpt-4o")
    .respondingWith(
        completion()
            .withText("MockServer is an open-source HTTP mock server.")
            .withStopReason("stop")
            .withUsage(usage().withInputTokens(12).withOutputTokens(8))
    )
    .applyTo(mockServerClient);

var mockServerClient = require('mockserver-client').mockServerClient;

// OpenAI — POST /v1/chat/completions
mockServerClient("localhost", 1080).mockAnyResponse({
    "httpRequest": {
        "method": "POST",
        "path": "/v1/chat/completions"
    },
    "httpLlmResponse": {
        "provider": "OPENAI",
        "model": "gpt-4o",
        "completion": {
            "text": "MockServer is an open-source HTTP mock server.",
            "stopReason": "stop",
            "usage": { "inputTokens": 12, "outputTokens": 8 }
        }
    }
}).then(
    function () { console.log("expectation created"); },
    function (error) { console.log(error); }
);

import requests

# OpenAI — POST /v1/chat/completions
requests.put(
    "http://localhost:1080/mockserver/expectation",
    json={
        "httpRequest": {
            "method": "POST",
            "path": "/v1/chat/completions"
        },
        "httpLlmResponse": {
            "provider": "OPENAI",
            "model": "gpt-4o",
            "completion": {
                "text": "MockServer is an open-source HTTP mock server.",
                "stopReason": "stop",
                "usage": {"inputTokens": 12, "outputTokens": 8}
            }
        }
    }
)

# Anthropic — POST /v1/messages
requests.put(
    "http://localhost:1080/mockserver/expectation",
    json={
        "httpRequest": {
            "method": "POST",
            "path": "/v1/messages"
        },
        "httpLlmResponse": {
            "provider": "ANTHROPIC",
            "model": "claude-sonnet-4-20250514",
            "completion": {
                "text": "MockServer is an open-source HTTP mock server.",
                "stopReason": "end_turn",
                "usage": {"inputTokens": 12, "outputTokens": 8}
            }
        }
    }
)

require 'net/http'
require 'json'

# OpenAI — POST /v1/chat/completions
uri = URI('http://localhost:1080/mockserver/expectation')
http = Net::HTTP.new(uri.host, uri.port)
req = Net::HTTP::Put.new(uri.path, 'Content-Type' => 'application/json')
req.body = JSON.generate({
  'httpRequest' => { 'method' => 'POST', 'path' => '/v1/chat/completions' },
  'httpLlmResponse' => {
    'provider' => 'OPENAI',
    'model' => 'gpt-4o',
    'completion' => {
      'text' => 'MockServer is an open-source HTTP mock server.',
      'stopReason' => 'stop',
      'usage' => { 'inputTokens' => 12, 'outputTokens' => 8 }
    }
  }
})
http.request(req)

# Anthropic — POST /v1/messages
req = Net::HTTP::Put.new(uri.path, 'Content-Type' => 'application/json')
req.body = JSON.generate({
  'httpRequest' => { 'method' => 'POST', 'path' => '/v1/messages' },
  'httpLlmResponse' => {
    'provider' => 'ANTHROPIC',
    'model' => 'claude-sonnet-4-20250514',
    'completion' => {
      'text' => 'MockServer is an open-source HTTP mock server.',
      'stopReason' => 'end_turn',
      'usage' => { 'inputTokens' => 12, 'outputTokens' => 8 }
    }
  }
})
http.request(req)

package main

import (
    "bytes"
    "encoding/json"
    "net/http"
)

func createLlmExpectations() {
    // OpenAI — POST /v1/chat/completions
    putExpectation("/v1/chat/completions", "OPENAI", "gpt-4o", "stop")
    // Anthropic — POST /v1/messages
    putExpectation("/v1/messages", "ANTHROPIC", "claude-sonnet-4-20250514", "end_turn")
}

func putExpectation(path, provider, model, stopReason string) {
    body, _ := json.Marshal(map[string]interface{}{
        "httpRequest": map[string]interface{}{
            "method": "POST",
            "path":   path,
        },
        "httpLlmResponse": map[string]interface{}{
            "provider": provider,
            "model":    model,
            "completion": map[string]interface{}{
                "text":       "MockServer is an open-source HTTP mock server.",
                "stopReason": stopReason,
                "usage":      map[string]int{"inputTokens": 12, "outputTokens": 8},
            },
        },
    })
    req, _ := http.NewRequest(http.MethodPut,
        "http://localhost:1080/mockserver/expectation", bytes.NewReader(body))
    req.Header.Set("Content-Type", "application/json")
    http.DefaultClient.Do(req)
}

using System.Net.Http;
using System.Text;
using System.Text.Json;

using var client = new HttpClient();

// OpenAI — POST /v1/chat/completions
var openAi = new
{
    httpRequest = new { method = "POST", path = "/v1/chat/completions" },
    httpLlmResponse = new
    {
        provider = "OPENAI",
        model = "gpt-4o",
        completion = new
        {
            text = "MockServer is an open-source HTTP mock server.",
            stopReason = "stop",
            usage = new { inputTokens = 12, outputTokens = 8 }
        }
    }
};
await client.PutAsync(
    "http://localhost:1080/mockserver/expectation",
    new StringContent(JsonSerializer.Serialize(openAi), Encoding.UTF8, "application/json"));

// Anthropic — POST /v1/messages
var anthropic = new
{
    httpRequest = new { method = "POST", path = "/v1/messages" },
    httpLlmResponse = new
    {
        provider = "ANTHROPIC",
        model = "claude-sonnet-4-20250514",
        completion = new
        {
            text = "MockServer is an open-source HTTP mock server.",
            stopReason = "end_turn",
            usage = new { inputTokens = 12, outputTokens = 8 }
        }
    }
};
await client.PutAsync(
    "http://localhost:1080/mockserver/expectation",
    new StringContent(JsonSerializer.Serialize(anthropic), Encoding.UTF8, "application/json"));

use serde_json::json;

// OpenAI — POST /v1/chat/completions
let client = reqwest::blocking::Client::new();
client.put("http://localhost:1080/mockserver/expectation")
    .json(&json!({
        "httpRequest": {
            "method": "POST",
            "path": "/v1/chat/completions"
        },
        "httpLlmResponse": {
            "provider": "OPENAI",
            "model": "gpt-4o",
            "completion": {
                "text": "MockServer is an open-source HTTP mock server.",
                "stopReason": "stop",
                "usage": { "inputTokens": 12, "outputTokens": 8 }
            }
        }
    }))
    .send()
    .unwrap();

// Anthropic — POST /v1/messages
client.put("http://localhost:1080/mockserver/expectation")
    .json(&json!({
        "httpRequest": {
            "method": "POST",
            "path": "/v1/messages"
        },
        "httpLlmResponse": {
            "provider": "ANTHROPIC",
            "model": "claude-sonnet-4-20250514",
            "completion": {
                "text": "MockServer is an open-source HTTP mock server.",
                "stopReason": "end_turn",
                "usage": { "inputTokens": 12, "outputTokens": 8 }
            }
        }
    }))
    .send()
    .unwrap();

<?php
$expectations = [
    // OpenAI — POST /v1/chat/completions
    [
        'httpRequest' => ['method' => 'POST', 'path' => '/v1/chat/completions'],
        'httpLlmResponse' => [
            'provider' => 'OPENAI',
            'model' => 'gpt-4o',
            'completion' => [
                'text' => 'MockServer is an open-source HTTP mock server.',
                'stopReason' => 'stop',
                'usage' => ['inputTokens' => 12, 'outputTokens' => 8],
            ],
        ],
    ],
    // Anthropic — POST /v1/messages
    [
        'httpRequest' => ['method' => 'POST', 'path' => '/v1/messages'],
        'httpLlmResponse' => [
            'provider' => 'ANTHROPIC',
            'model' => 'claude-sonnet-4-20250514',
            'completion' => [
                'text' => 'MockServer is an open-source HTTP mock server.',
                'stopReason' => 'end_turn',
                'usage' => ['inputTokens' => 12, 'outputTokens' => 8],
            ],
        ],
    ],
];
foreach ($expectations as $expectation) {
    $ch = curl_init('http://localhost:1080/mockserver/expectation');
    curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'PUT');
    curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($expectation));
    curl_setopt($ch, CURLOPT_HTTPHEADER, ['Content-Type: application/json']);
    curl_exec($ch);
    curl_close($ch);
}

# OpenAI — POST /v1/chat/completions
curl -v -X PUT "http://localhost:1080/mockserver/expectation" -d '{
  "httpRequest": {
    "method": "POST",
    "path": "/v1/chat/completions"
  },
  "httpLlmResponse": {
    "provider": "OPENAI",
    "model": "gpt-4o",
    "completion": {
      "text": "MockServer is an open-source HTTP mock server.",
      "stopReason": "stop",
      "usage": { "inputTokens": 12, "outputTokens": 8 }
    }
  }
}'

# Anthropic — POST /v1/messages
curl -v -X PUT "http://localhost:1080/mockserver/expectation" -d '{
  "httpRequest": {
    "method": "POST",
    "path": "/v1/messages"
  },
  "httpLlmResponse": {
    "provider": "ANTHROPIC",
    "model": "claude-sonnet-4-20250514",
    "completion": {
      "text": "MockServer is an open-source HTTP mock server.",
      "stopReason": "end_turn",
      "usage": { "inputTokens": 12, "outputTokens": 8 }
    }
  }
}'

When a request matches POST /v1/chat/completions, MockServer returns an OpenAI-formatted JSON response with the specified text, model, stop reason, and token usage — including the correct id, created, and object fields that the OpenAI SDK expects.

A stopReason is converted to the value each provider uses, so you can share one expectation across providers. For example, length becomes max_tokens for Anthropic and Bedrock InvokeModel, MAX_TOKENS for Gemini, and done_reason: "length" for Ollama; stop becomes end_turn for Anthropic. Ollama's final message always has a done_reason: stop unless you set a max-tokens value. When an Ollama reply is streamed, its tool calls arrive in a message before that final one, as Ollama sends them, and the final message holds only the stop reason and token counts.

OPENROUTER reports two values on each choice, as OpenRouter does: finish_reason, normalised to stop, length, tool_calls or content_filter, and native_finish_reason, the raw value of the model behind it. Set stopReason to that model's own value to mock it: with STOP (a Gemini model) or end_turn (a Claude model) a choice has "finish_reason": "stop" and that value as its native_finish_reason. Without a stopReason both fields have the same value. In a stream, native_finish_reason is null until the chunk that carries the finish_reason.

Supported Providers

Each provider value produces the correct API response format for that provider's SDK:

MISTRAL, XAI, DEEPSEEK, GROQ, OPENROUTER and ORCAROUTER serve the same Chat Completions wire format as OpenAI, so they produce identical response bodies. They all match on a /chat/completions path, so it is the HOST that tells them apart — pick the value that names the service you are standing in for. When MockServer is proxying, that is what makes the captured traffic show up as LLM traffic with the right provider and pricing.

AWS Bedrock: Converse and InvokeModel

AWS Bedrock has two APIs that return different JSON, and BEDROCK supports both. MockServer picks the format from the path of the incoming request, so the same expectation works with whichever API your code calls:

If the path doesn't end with one of these (for example, a gateway path), MockServer looks at the request body instead. A Converse-style body (content blocks such as {"text":"..."}, or an inferenceConfig or toolConfig field) gets the Converse format. Anything else gets the InvokeModel format, which is what BEDROCK returned before Converse support was added. For streaming, set "streaming": true on the completion, just as for other providers.

In the Converse format, totalTokens is inputTokens + outputTokens, plus cachedInputTokens and cacheCreationTokens if you set them. Those two appear in the response as cacheReadInputTokens and cacheWriteInputTokens. This follows AWS, where inputTokens counts only the input tokens that weren't read from or written to the cache. metrics.latencyMs is always 0; use streaming physics or a response delay to simulate latency. Conversation matchers read Converse requests too: the system text blocks, the messages[].content[] text, image, toolUse and toolResult blocks. Any stopReason you set is converted to one of the values AWS accepts, so AWS SDKs can always read it. For example, refusal becomes content_filtered, and unrecognised values become end_turn.

Chaos follows the API too. A contentFilterBlockProbability block on a Converse request returns a Converse response with stopReason content_filtered; on InvokeModel it returns the Anthropic refusal (stop_reason refusal with a stop_details object). Injected errors and structured-output enforcement errors use the AWS error body on both APIs, as Bedrock does, for example {"__type":"ThrottlingException","message":"..."} for a 429.

The dashboard shows Bedrock Converse and InvokeModel traffic as Bedrock, with the messages, reply and token counts. InvokeModel calls to models other than Claude and Nova (for example Llama or Titan) are labelled Bedrock but show no reply or token counts.

Streaming Responses

Set streaming to true on the completion to return a stream instead of a single JSON response. MockServer splits the text into token-sized chunks and sends them as streaming chunks in the provider's native streaming format (SSE for most providers; NDJSON for Ollama; binary AWS event-stream for Bedrock). Gemini follows the request, as the real API does: a streamGenerateContent request with ?alt=sse gets SSE, and one without it gets a single JSON array of response chunks (application/json), which is what Google's REST client libraries expect.

By default the text is split into fine, subword-sized chunks that closely match how a real provider streams tokens (so a streamed response emits more, smaller deltas). Set subwordStreaming to false in streamingPhysics if you prefer the coarser whole-word chunking (one chunk per word). The timing controls below behave identically in both modes — subword chunking simply produces more deltas, which makes the overall stream take slightly longer at the same tokensPerSecond rate.

Use streamingPhysics to control timing — useful for testing loading indicators, timeouts, and backpressure handling:

{
  "httpRequest": {
    "method": "POST",
    "path": "/v1/chat/completions"
  },
  "httpLlmResponse": {
    "provider": "OPENAI",
    "model": "gpt-4o",
    "completion": {
      "text": "This response is streamed token by token.",
      "streaming": true,
      "streamingPhysics": {
        "timeToFirstToken": {
          "timeUnit": "MILLISECONDS",
          "value": 200
        },
        "tokensPerSecond": 50,
        "jitter": 0.1,
        "seed": 42
      },
      "usage": {
        "inputTokens": 10,
        "outputTokens": 8
      }
    }
  }
}

Token usage in an OpenAI stream. For OPENAI, AZURE_OPENAI, XAI and ORCAROUTER, a stream carries the usage above only when the request asks for it, as the real APIs do. Four providers send it without being asked, as their real APIs do: MISTRAL puts usage on the chunk with the finish_reason, DEEPSEEK does too (and when asked, also sends "usage": null on every other chunk, with no separate usage chunk), GROQ puts it in x_groq.usage on that chunk, and OPENROUTER always ends the stream with the usage chunk shown below, though its usage chunk has one choice with an empty delta that repeats the finish_reason, as OpenRouter's does, where OpenAI's has none. Some AI SDKs ask by default (the Vercel AI SDK does); in LangChain set stream_usage, and with the OpenAI SDKs pass stream_options. The request sets stream_options:

{
  "model": "gpt-4o",
  "stream": true,
  "stream_options": { "include_usage": true },
  "messages": [{ "role": "user", "content": "Hello" }]
}

Every chunk then carries "usage": null, and one extra chunk with an empty choices array and the token counts comes just before [DONE]:

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":1759478400,"model":"gpt-4o","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"usage":null}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":1759478400,"model":"gpt-4o","choices":[],"usage":{"prompt_tokens":10,"completion_tokens":8,"total_tokens":18}}

data: [DONE]

Without include_usage no chunk has a usage field. A non-streaming response always includes usage.

Tool Calls

To mock an LLM response that invokes tools (function calling), add toolCalls to the completion:

{
  "httpRequest": {
    "method": "POST",
    "path": "/v1/chat/completions"
  },
  "httpLlmResponse": {
    "provider": "OPENAI",
    "model": "gpt-4o",
    "completion": {
      "toolCalls": [
        {
          "id": "call_abc123",
          "name": "get_weather",
          "arguments": "{\"location\": \"London\"}"
        }
      ],
      "stopReason": "tool_use"
    }
  }
}

A tool call's id is returned as the tool-call id your client sends back with the tool result: tool_calls[].id for OpenAI and the OpenAI-compatible providers, the tool_use block's id for Anthropic and Bedrock, call_id for the OpenAI Responses API, toolUseId for the Bedrock Converse API, and functionCall.id for Gemini and tool_calls[].id for Ollama. Leave it out and MockServer generates one in the provider's format (Gemini and Ollama then send none, as their APIs allow).

For OpenAI-compatible mocks you can also set toolChoice on the completion to model the request's tool_choice directive. It accepts auto, none, required, or a named tool. When toolChoice is required and a tool call is configured, the mocked response's finish_reason is forced to tool_calls (for both non-streaming and streaming responses), matching how OpenAI behaves when a caller forces a tool call. Omitting toolChoice leaves the finish reason behaviour unchanged.

{
  "httpLlmResponse": {
    "provider": "OPENAI",
    "model": "gpt-4o",
    "completion": {
      "toolChoice": "required",
      "toolCalls": [
        { "name": "get_weather", "arguments": "{\"location\": \"London\"}" }
      ]
    }
  }
}

With the OPENAI_RESPONSES provider, each tool call is returned as a function_call output item with an item id (fc_…) and a separate call_id (call_…), as the real Responses API does. If you set an id on the tool call, such as call_abc123 above, that becomes the call_id. Your agent sends the tool's result back as a function_call_output carrying that call_id, and that is what containsToolResultFor matches on. This also works when the next request sends only the result plus previous_response_id, whether or not the earlier turn was streamed. When streaming, the arguments arrive as response.function_call_arguments.delta events, and the final response.completed event carries the whole response, including its output. The OpenAI Agents SDK reads each turn from that event.

The streamed arguments are split into small pieces the same way text is, one delta event per piece, and each event is paced by tokensPerSecond when you set streaming physics. A streamed tool call with long arguments therefore takes longer with OPENAI_RESPONSES than with the other providers, which send a tool call's arguments in one piece.

Embeddings

Mock an embeddings endpoint by setting embedding instead of completion. MockServer returns a provider-correct embeddings response with a vector of the requested size. Set deterministicFromInput to true so the same input text always produces the same (L2-normalised) vector — ideal for reproducible tests of retrieval / vector-search code.

When deterministicFromInput is true, the vector is also semantically plausible: it is derived from the words and character n-grams in the input text, so texts that share vocabulary get a higher cosine similarity and unrelated texts are near-orthogonal. This means your retrieval / RAG code can rank related documents above unrelated ones offline — for example "the cat sat on the mat" scores far higher against "a cat sits on a mat" than against "quarterly financial report" — with no real embedding model and deterministic results for the same input, seed, and dimensions.

{
  "httpRequest": {
    "method": "POST",
    "path": "/v1/embeddings"
  },
  "httpLlmResponse": {
    "provider": "OPENAI",
    "model": "text-embedding-3-small",
    "embedding": {
      "dimensions": 1536,
      "deterministicFromInput": true,
      "seed": 42
    }
  }
}

MockServer reads each provider's own input field and returns one vector per input, in the order sent, so a batch of documents gets a batch of vectors. With deterministicFromInput, the same text gets the same vector in every request and on every provider, as long as the seed and dimensions match.

For BEDROCK, the model comes from the request path (/model/{modelId}/invoke), then from the expectation's model. Region-prefixed IDs such as us.cohere.embed-v4:0 work. If neither names an embedding model, a request with texts, inputs or images gets the Cohere shape and any other request gets the Titan V2 shape. With deterministicFromInput, the same image always gets the same vector. Sending both texts and images is rejected, as on Bedrock.

As on Bedrock, a non-streaming BEDROCK InvokeModel response also reports its token counts in the X-Amzn-Bedrock-Input-Token-Count header and, for a completion, the X-Amzn-Bedrock-Output-Token-Count header. Clients such as LangChain's ChatBedrock read usage from these headers. If the completion's usage sets cachedInputTokens or cacheCreationTokens, the response also has the X-Amzn-Bedrock-Cache-Read-Input-Token-Count and X-Amzn-Bedrock-Cache-Write-Input-Token-Count headers, as Bedrock sends for models that support prompt caching; a count you leave unset is sent as 0. For a Titan embedding the input count is inputTextTokenCount; for a Cohere embedding it is an estimate of about four characters per token of the texts, with images not counted. Converse and streaming responses have no token-count headers, as on Bedrock.

dimensions on the expectation sets the vector length. If it is not set, the size the request asks for (dimensions, outputDimensionality or output_dimension) is used, and then the default above. seed makes the deterministic vector reproducible across runs. dimensions on the expectation must be at least 1.

MockServer rejects an embedding request that is over a limit with a 400 in the provider's own error format. Some of the limits copy the real provider's: the Titan and Cohere sizes and embedding types, 96 Cohere texts or inputs, 100 Gemini requests in one batch, 2,048 OpenAI inputs, and OpenAI's 300,000 tokens per request and 8,192 tokens per input (checked only for token arrays, see below). The rest are MockServer's own limits, which keep its responses small: 8,192 dimensions, 2,048 Ollama inputs, and the total number of vector values. A request is rejected when:

• the request asks for more than 8,192 dimensions, or for a size that is not a positive whole number;
• the request asks for a size the provider does not offer: Titan Text Embeddings V2 dimensions must be 256, 512 or 1024, Titan Multimodal embeddingConfig.outputEmbeddingLength 256, 384 or 1024, and Cohere Embed v4 output_dimension 256, 512, 1024 or 1536;
• the request has more than 2,048 inputs, more than 100 Gemini requests, or more than 96 Cohere texts or inputs;
• an OPENAI or AZURE_OPENAI request is certainly over 300,000 tokens, with OpenAI's max_tokens_per_request error. MockServer has no tokeniser, so it counts a token array as its length and each text as just one token: a request of token arrays over the limit is rejected, but text inputs are never rejected for their length, so a batch the real API would accept is never refused. The response's usage.prompt_tokens still estimates text at about four characters a token;
• an OPENAI or AZURE_OPENAI token array is longer than 8,192 tokens, the limit for one input to any OpenAI embedding model, with OpenAI's invalid_request_error ("This model's maximum context length is 8192 tokens, however you requested ..."). A text input is never rejected for its length, because only the model's tokeniser can count it;
• the request asks for a Cohere embedding_types or Titan embeddingTypes value the provider does not have;
• the response would hold more than 262,144 vector values in total (inputs × dimensions, and × the number of Cohere embedding types), for example 171 inputs at 1,536 dimensions. An OpenAI-style request with "encoding_format":"base64", which the OpenAI SDKs send by default, may hold four times as many, 1,048,576 (682 inputs at 1,536 dimensions), because a base64 value is about a quarter of the size;
• with deterministicFromInput, an input is longer than 1,048,576 characters (building a vector from text uses memory in proportion to its length; random vectors have no such limit). This is more than any real embedding provider accepts in one input, so it only stops oversized test data.

These limits keep the largest embedding response at about 5.6 MB, so a small request cannot make MockServer build a very large one. If a test needs a bigger batch, raise the configuration property llmEmbeddingMaxResponseValues (for example to 384000 for a LangChain batch of 1,000 inputs at 1,536 dimensions), send the batch in several requests, or set a smaller dimensions on the expectation: 2,048 inputs fit at 128 dimensions. For longer inputs, raise llmEmbeddingMaxInputCharacters.

ANTHROPIC and OPENAI_RESPONSES have no embeddings endpoint and return an error if used for an embedding mock.

Rerank

Mock a rerank endpoint by setting rerank with the COHERE or VOYAGE provider. MockServer reads the candidate documents from the request body's documents array and returns one result per document — each with its original index and a relevance_score — sorted from most to least relevant. Set deterministicFromInput to true so the same documents always score the same way. MockServer returns as many results as the request asks for (Cohere top_n, Voyage top_k); set topN on the expectation to override that.

{
  "httpRequest": {
    "method": "POST",
    "path": "/v1/rerank"
  },
  "httpLlmResponse": {
    "provider": "COHERE",
    "rerank": {
      "topN": 3,
      "deterministicFromInput": true,
      "seed": 42
    }
  }
}

Each provider returns its own response envelope around the same per-document scores ({"index": N, "relevance_score": F}), sorted by descending relevance:

Documents may be plain strings or objects with a text field (Cohere's structured form) — both are read from the request's documents array. A VOYAGE request with more than 1,000 documents gets a 400 with Voyage's {"detail": "..."} error body, as the real API does. Voyage's limits on query tokens and on total tokens are not checked: a rerank request sends only text, which MockServer cannot count exactly without Voyage's tokeniser.

Multimodal Request Recognition (Images and Audio)

When matching against requests, MockServer recognises image and audio content parts in the decoded conversation, so conversation predicates can assert that a message contained an image or an audio clip (and, where the provider declares it, the media type or audio format). For the OpenAI-compatible format this covers image_url and input_audio content parts. This is request-side recognition only — MockServer notes the presence (and how many) but does not store the media bytes or generate image/audio responses.

Multi-Turn Conversations

For agent testing, you often need to script a sequence of LLM responses that depend on what the agent sent. MockServer supports this through conversation predicates combined with scenario state to create multi-turn conversation flows.

Each turn uses conversationPredicates to match against the conversation history in the request body, and scenario state to track which turn the conversation is on:

[
  {
    "httpRequest": {
      "method": "POST",
      "path": "/v1/chat/completions"
    },
    "httpLlmResponse": {
      "provider": "OPENAI",
      "model": "gpt-4o",
      "completion": {
        "text": "I'll look up the weather for you.",
        "toolCalls": [
          {
            "id": "call_1",
            "name": "get_weather",
            "arguments": "{\"location\": \"London\"}"
          }
        ],
        "stopReason": "tool_use"
      },
      "conversationPredicates": {
        "turnIndex": 0,
        "latestMessageContains": "weather"
      }
    },
    "times": { "remainingTimes": 1 }
  },
  {
    "httpRequest": {
      "method": "POST",
      "path": "/v1/chat/completions"
    },
    "httpLlmResponse": {
      "provider": "OPENAI",
      "model": "gpt-4o",
      "completion": {
        "text": "The weather in London is 18C and sunny.",
        "stopReason": "stop"
      },
      "conversationPredicates": {
        "containsToolResultFor": "get_weather"
      }
    },
    "times": { "remainingTimes": 1 }
  }
]

Conversation predicates

Predicates match against the parsed conversation in the request body (decoded using the provider's message format):

Prompt normalisation

Agent prompts are dynamically assembled, so exact-byte matching can be brittle. Add a normalization object to the predicates to apply deterministic transforms before matching:

"conversationPredicates": {
  "latestMessageContains": "search for weather",
  "normalization": {
    "collapseWhitespace": true,
    "lowercase": true,
    "sortJsonKeys": true
  }
}

Java Client API

The Java client's Llm.conversation() builder (org.mockserver.client.LlmConversationBuilder) produces the same turn expectations without hand-writing the JSON above. Each turn() block sets its match predicates (whenTurnIndex, whenLatestMessageContains, whenLatestMessageMatches, whenLatestMessageRole, whenContainsToolResultFor, whenSemanticMatch) and its response, chained with andThen(); applyTo(...) registers every turn under a single auto-generated scenario name so turns are matched in order:

import static org.mockserver.client.Llm.conversation;
import static org.mockserver.client.Llm.completion;
import static org.mockserver.client.Llm.toolUse;
import static org.mockserver.model.Provider.OPENAI;

conversation()
    .withPath("/v1/chat/completions")
    .withProvider(OPENAI)
    .withModel("gpt-4o")
    .turn()
        .whenTurnIndex(0)
        .whenLatestMessageContains("weather")
        .respondingWith(
            completion()
                .withText("I'll look up the weather for you.")
                .withToolCall(toolUse("get_weather").withArguments("{\"location\": \"London\"}"))
                .withStopReason("tool_use")
        )
    .andThen()
    .turn()
        .whenContainsToolResultFor("get_weather")
        .respondingWith(completion().withText("The weather in London is 18C and sunny."))
    .applyTo(mockServerClient);

Add .isolateBy(Llm.header("x-session-id")) (or Llm.queryParameter(...) / Llm.cookie(...)) before the first turn() to key conversation state per session — see Session Isolation below.

Session Isolation

When multiple agents or test threads share a MockServer instance, each conversation needs its own independent state. Use an isolation source to extract a session key from each request (a header, query parameter, or cookie) so conversation state is tracked per session.

Session isolation is configured via the Java client’s conversation builder (isolateBy(IsolationSource.header("x-session-id"))) or the create_llm_conversation MCP tool’s isolateBy parameter. The isolation source is encoded into the expectation’s scenario name — it is not a separate field in the raw expectation JSON. Common isolation sources:

• Header: extract the session key from a request header (e.g. x-session-id)
• Query parameter: extract from a URL query parameter
• Cookie: extract from a cookie value

Example using the create_llm_conversation MCP tool:

{
  "provider": "OPENAI",
  "path": "/v1/chat/completions",
  "isolateBy": {
    "source": "header",
    "name": "x-session-id"
  },
  "turns": [
    {
      "match": { "turnIndex": 0 },
      "response": { "text": "Hello!", "stopReason": "stop" }
    }
  ]
}

Each unique isolation key value creates an independent conversation state machine, so multiple agents can run conversations in parallel without interfering with each other.

Realtime Voice APIs (OpenAI Realtime, Gemini Live)

The realtime voice APIs — the OpenAI Realtime API and the Google Gemini Live API — are not ordinary HTTP calls: the client opens a WebSocket and exchanges a stream of JSON events. MockServer can mock both so you can test an agent or app that uses them completely offline, with no real API, no cost, and no audio hardware. The mock replies to the client’s control messages with the provider-correct event stream for one scripted spoken turn (transcript deltas, audio-byte deltas, and a final usage/done event).

Point your realtime SDK at MockServer’s WebSocket URL (e.g. ws://localhost:1080/v1/realtime) instead of the real provider. Audio bytes in the mocked stream are short silence placeholders — the aim is to reproduce the event protocol faithfully (so your event-handling code is exercised), not to synthesise real speech. The same scripted turn answers every request on the connection.

Available in the Java client only. Realtime voice mocking is a Java-client convenience (RealtimeMockBuilder): there is no dedicated realtime JSON field. The builder expands the short script below into a full provider-correct httpWebSocketResponse expectation (session events, streamed transcript/audio deltas, and a final usage event) using the server-side realtime codecs. Reproducing that by hand in another client would mean writing out every WebSocket frame, so the other client libraries do not offer this shortcut.

import static org.mockserver.client.RealtimeMockBuilder.openAiRealtime;
import static org.mockserver.llm.realtime.RealtimeTurn.realtimeTurn;

// Mock a spoken response over ws://localhost:1080/v1/realtime
openAiRealtime()
    .withModel("gpt-realtime")
    .respondingWith(
        realtimeTurn("The capital of France is Paris.")
            .withInputTokens(20)
            .withOutputTokens(7)
    )
    .applyTo(mockServerClient);

// On connect the mock pushes session.created; it acknowledges session.update and
// conversation.item.create, and answers each response.create with the full event
// lifecycle ending in response.done (with usage).

import static org.mockserver.client.RealtimeMockBuilder.geminiLive;

// Mock a Gemini Live BidiGenerateContent session
geminiLive()
    .respondingWith("Bonjour le monde")
    .applyTo(mockServerClient);

// The mock answers the client's setup message with setupComplete, then answers each
// clientContent turn with a streamed serverContent chunk sequence, generationComplete,
// and a final turnComplete carrying usageMetadata.

Use .withModality(RealtimeModality.TEXT) for a text-only response, .withTokensPerSecond(...) / .withTimeToFirstToken(...) to control streaming timing, and .respondingWith(realtimeTurn(...)) to set an explicit transcript, synthetic audio bytes, or token usage. Protocol corners that are intentionally not mocked yet (server voice-activity-detection events, tool/function calls, input-audio transcription) are listed in the developer documentation.

Cost Budget

When using MockServer as a proxy in front of a real LLM provider, the llmCostBudgetUsd property sets a cumulative cost ceiling. Once the estimated cost of all forwarded LLM completions exceeds the budget, further LLM forwards are blocked with a 429 response. This is a safety net for CI pipelines or development environments where runaway agents could generate unexpected charges.

The budget is enforced on all forward paths (matched forward actions, proxy-pass, and reverse-proxy routes) and resets on PUT /mockserver/reset. Cost estimation uses an internal pricing table and is approximate — treat it as a safety guard, not an invoice.

Streamed responses are counted too. MockServer reads the token usage the provider reports in the stream and adds the call's cost when the stream ends. The budget is checked before each call is forwarded, so a call that is already streaming when the budget is reached still completes and the next one is blocked. MockServer never estimates usage: a stream that reports none is not counted, and MockServer logs a warning naming the provider and model (no token usage in the streamed ... response, at most once a minute for each provider and model), so check that your client asks for usage where the provider needs it:

Calls through proxyPassMappings reverse-proxy routes are counted the same way, by the host the mapping forwards to. Only calls to a model are counted: completions, and whole embeddings and rerank responses, whose trace spans are named embeddings <model> and rerank <model> instead of chat <model>. Cohere calls are counted by the tokens Cohere bills (a rerank is billed in search units, so it has a span but no token counts), Voyage calls by their total_tokens, and AWS Bedrock InvokeModel calls whose body reports no usage, such as embedding and rerank models, by the token-count headers Bedrock returns. A Bedrock Agent Runtime /rerank call is named by the model in its modelArn, and reports no tokens. On Vertex AI, embedding models called with :predict are counted, and Mistral models are counted as Mistral. Calls to Vertex AI's OpenAI-compatible endpoints (endpoints/openapi/chat/completions and endpoints/openapi/responses, used for Llama, DeepSeek, Qwen, gpt-oss, Grok and other Model Garden models, for Gemini through OpenAI libraries, and the same paths on a self-deployed endpoint) and to AI21 Jamba are counted from their OpenAI-style usage, including on the multi-region aiplatform.us.rep.googleapis.com and aiplatform.eu.rep.googleapis.com hosts. Their cost is estimated at Vertex AI's list prices for Gemini 2.5 and 2.0 models (google/gemini-2.5-flash) and Llama 3.3 70B (meta/llama-3.3-70b-instruct-maas); other models on these endpoints, and AI21 Jamba, are counted in tokens but have no price, so they add nothing to llmCostBudgetUsd. Gemini calls to Vertex AI (:generateContent and :streamGenerateContent on an aiplatform host) are costed at Vertex AI's list prices, which for Gemini 2.0 Flash are higher than the Gemini API's; calls to generativelanguage.googleapis.com keep the Gemini API's prices. The optimisation report, the verify_cost_budget MCP tool and the dashboard's cost estimates price captured calls the same way, by the host the captured request was sent to. A call sent through a proxyPassMappings mapping is captured with MockServer's own host, so those three price it at the Gemini API's prices. A call sent by an httpForward expectation is captured with the host it was forwarded to, so it is priced like a call through the proxy, unless a request breakpoint changed the request before it was forwarded: that call is captured with the host MockServer received it on. The dashboard shows calls to Vertex AI's OpenAI-compatible endpoints, and to Mistral and AI21 Jamba models on Vertex AI, with their token usage, and estimates the cost of the Gemini and Llama 3.3 70B models named above; it has no Mistral prices, so a Mistral call shows its tokens but no cost. An OpenAI response created with background: true and an xAI deferred completion report their usage only when the finished result is fetched, so that fetch is counted, once however many times it is polled. Other requests to an LLM host, such as listing models, fetching an ordinary stored response, counting tokens or Ollama's /api/pull download progress, are relayed but not treated as an LLM call. Behind a gateway or private endpoint that is not a known provider host, set llmProvider so Gemini and Bedrock calls through it are recognised and counted.

Moderation, Content Filter & Refusal

Production agents must handle provider content-filter blocks and refusals. These opt-in fields let you trigger them deterministically. All default to off / not-flagged, so existing mocks are unaffected.

OpenAI Moderations endpoint — set moderation (instead of completion) and a request returns OpenAI's /v1/moderations shape. Categories listed in flaggedCategories are marked flagged; an empty list is a not-flagged verdict.

{
  "httpRequest": {
    "method": "POST",
    "path": "/v1/moderations"
  },
  "httpLlmResponse": {
    "provider": "OPENAI",
    "moderation": {
      "flaggedCategories": [ "hate", "violence" ]
    }
  }
}

Azure content-filter annotations — set contentFilter with per-category severities (hate, sexual, violence, selfHarm: one of safe, low, medium, high) on an AZURE_OPENAI completion. MockServer adds the content_filter_results (per choice) and prompt_filter_results (top-level) annotations agents read to detect filtering (filtered is true at medium/high).

{
  "httpRequest": {
    "method": "POST",
    "path": "/openai/deployments/gpt-4o/chat/completions"
  },
  "httpLlmResponse": {
    "provider": "AZURE_OPENAI",
    "model": "gpt-4o",
    "completion": { "text": "Filtered response" },
    "contentFilter": { "hate": "high", "violence": "safe" }
  }
}

Refusals & content-filter blocks — an Anthropic refusal is just a completion with "stopReason": "refusal". As the real API does, every Anthropic refusal (whole or streamed, including one returned through Bedrock InvokeModel) carries a stop_details object, {"type": "refusal", "category": null, "explanation": null}; a stream carries it in the message_delta event's delta, so SDK code that reads stop_details works against the mock. Every other Anthropic message sends "stop_details": null, also as the real API does. Like the real API, every Anthropic message also carries "container": null (no container tool ran) and "diagnostics": null (the request did not ask for cache diagnostics). To inject a block probabilistically across providers, use the contentFilterBlockProbability chaos field (see below), which emits the provider-correct shape.

Chaos / Fault Injection

Add a chaos object to the httpLlmResponse to test how your application handles LLM failures:

{
  "httpRequest": {
    "method": "POST",
    "path": "/v1/chat/completions"
  },
  "httpLlmResponse": {
    "provider": "OPENAI",
    "model": "gpt-4o",
    "completion": {
      "text": "Normal response text",
      "streaming": true
    },
    "chaos": {
      "errorStatus": 503,
      "errorProbability": 0.3,
      "truncateMode": "MID_STREAM",
      "truncateAtFraction": 0.5,
      "malformedSse": true
    }
  }
}

See Chaos Testing & Fault Injection for more about chaos profiles.

Provider-Correct Error Bodies

LLM chaos error injection emits the body shape the real provider returns — not a generic error body. This lets your client SDK's retry and backoff logic be tested against realistic error responses:

An optional errorKind field (OVERLOAD, RATE_LIMIT, or SERVER_ERROR) on the LLM chaos profile lets you declare the error intent. MockServer emits the active provider's natural HTTP status and its provider-correct body without you having to pick the status manually. An explicit errorStatus still overrides the code while keeping the provider-correct body. A missing or unrecognised errorKind falls back to the generic body.

Failover / Retry Scripting (Java Client)

The chaos profile above injects probabilistic failures. To test a specific, deterministic retry sequence — "the first N attempts fail, then it succeeds" — the Java client's Llm.llmFailover() builder (org.mockserver.client.LlmFailoverBuilder) produces an ordered set of expectations that rely on MockServer's matching order and Times exhaustion: the failure expectations are registered first with a limited Times, so they are matched and consumed before the unlimited success expectation is reached. This lets you point a retry-capable client (an SDK's own retry config, LiteLLM, an AI gateway) at MockServer and assert its failover/retry logic deterministically:

import static org.mockserver.client.Llm.llmFailover;
import static org.mockserver.client.Llm.completion;
import static org.mockserver.model.Provider.OPENAI;

llmFailover()
    .withPath("/v1/chat/completions")
    .withProvider(OPENAI)
    .withModel("gpt-4o")
    .failWith(503)                                    // attempt 1 fails with 503 (generic default error body for this status, not provider-specific)
    .failWith(429, "{\"error\":{\"type\":\"rate_limit_error\"}}") // attempt 2 fails with 429 and a custom body
    .thenRespondWith(completion().withText("The answer is 42."))  // attempt 3+ succeeds
    .applyTo(mockServerClient);                        // or .build() to get the Expectation[] without registering it

failWith(statusCode) and failWith(statusCode, errorBody) add one failed attempt; failWith(statusCode, count) adds count consecutive failures with the same status. The no-errorBody form's default error body is chosen by statusCode alone (429/500/502/503 each get their own error type and message in the same generic {"error":{"type","message"}} shape, anything else a generic fallback) — it is not provider-specific, unlike the LLM chaos profile's errorKind field described above. Pass an explicit errorBody if you need a provider-correct error shape. Consecutive failures with the same status and body are coalesced into a single expectation with Times.exactly(count). statusCode must be between 100 and 599. build() throws IllegalStateException if the path, provider, at least one failure, or the success completion (thenRespondWith(...)) is not set.

Cached and Reasoning Token Usage

org.mockserver.model.Usage has three optional fields in addition to the baseline inputTokens / outputTokens pair. These are back-compatible and optional: omitting them leaves the response without the matching provider fields. There are two exceptions. OPENAI_RESPONSES always returns usage.input_tokens_details and usage.output_tokens_details, as the real Responses API does, with 0 for each count you do not set. ANTHROPIC and BEDROCK (InvokeModel) always return cache_creation_input_tokens, cache_read_input_tokens and the cache_creation breakdown, with 0 for a count you do not set, plus service_tier and, for ANTHROPIC only, inference_geo, as the real APIs do. Streamed Anthropic responses repeat the input and cache counts in the final message_delta usage.

The OpenAI-compatible providers report these counts the way their real APIs do. DEEPSEEK also returns prompt_cache_hit_tokens and prompt_cache_miss_tokens, always returns prompt_tokens_details.cached_tokens (0 when cachedInputTokens is not set), and sends a system_fingerprint on the response and on every streamed chunk, as DeepSeek's API does. XAI leaves reasoning tokens out of completion_tokens and adds them to total_tokens, so outputTokens: 103 with reasoningTokens: 94 is returned as completion_tokens: 9. GROQ adds its timing fields (queue_time, prompt_time, completion_time, total_time, in seconds, taken from the completion's streamingPhysics when set, else 0) and an x_groq request id. OPENROUTER adds cost (when MockServer knows the model's price) and is_byok. A completion's reasoningText is returned as reasoning_content by DEEPSEEK and XAI, and as reasoning by GROQ and OPENROUTER, streamed or not.

Set these fields on the usage object in your expectation JSON to simulate provider responses from models that use prompt caching or chain-of-thought reasoning:

import static org.mockserver.client.LlmMockBuilder.llmMock;
import static org.mockserver.model.Completion.completion;
import static org.mockserver.model.Provider.ANTHROPIC;
import static org.mockserver.model.Usage.usage;

llmMock("/v1/messages")
    .withProvider(ANTHROPIC)
    .withModel("claude-sonnet-4-20250514")
    .respondingWith(
        completion()
            .withText("Paris is the capital of France.")
            .withStopReason("end_turn")
            .withUsage(
                usage()
                    .withInputTokens(150)
                    .withOutputTokens(10)
                    .withCachedInputTokens(120)  // subset of inputTokens
                    .withCacheCreationTokens(0)
                    .withReasoningTokens(0)
            )
    )
    .applyTo(mockServerClient);

var mockServerClient = require('mockserver-client').mockServerClient;

mockServerClient("localhost", 1080).mockAnyResponse({
    "httpRequest": {
        "method": "POST",
        "path": "/v1/messages"
    },
    "httpLlmResponse": {
        "provider": "ANTHROPIC",
        "model": "claude-sonnet-4-20250514",
        "completion": {
            "text": "Paris is the capital of France.",
            "stopReason": "end_turn",
            "usage": {
                "inputTokens": 150,
                "outputTokens": 10,
                "cachedInputTokens": 120,
                "cacheCreationTokens": 0,
                "reasoningTokens": 0
            }
        }
    }
}).then(
    function () { console.log("expectation created"); },
    function (error) { console.log(error); }
);

import requests

requests.put(
    "http://localhost:1080/mockserver/expectation",
    json={
        "httpRequest": {
            "method": "POST",
            "path": "/v1/messages"
        },
        "httpLlmResponse": {
            "provider": "ANTHROPIC",
            "model": "claude-sonnet-4-20250514",
            "completion": {
                "text": "Paris is the capital of France.",
                "stopReason": "end_turn",
                "usage": {
                    "inputTokens": 150,
                    "outputTokens": 10,
                    "cachedInputTokens": 120,
                    "cacheCreationTokens": 0,
                    "reasoningTokens": 0
                }
            }
        }
    }
)

require 'net/http'
require 'json'

uri = URI('http://localhost:1080/mockserver/expectation')
http = Net::HTTP.new(uri.host, uri.port)
req = Net::HTTP::Put.new(uri.path, 'Content-Type' => 'application/json')
req.body = JSON.generate({
  'httpRequest' => { 'method' => 'POST', 'path' => '/v1/messages' },
  'httpLlmResponse' => {
    'provider' => 'ANTHROPIC',
    'model' => 'claude-sonnet-4-20250514',
    'completion' => {
      'text' => 'Paris is the capital of France.',
      'stopReason' => 'end_turn',
      'usage' => {
        'inputTokens' => 150,
        'outputTokens' => 10,
        'cachedInputTokens' => 120,
        'cacheCreationTokens' => 0,
        'reasoningTokens' => 0
      }
    }
  }
})
http.request(req)

package main

import (
    "bytes"
    "encoding/json"
    "net/http"
)

func createCachedUsageExpectation() {
    body, _ := json.Marshal(map[string]interface{}{
        "httpRequest": map[string]interface{}{
            "method": "POST",
            "path":   "/v1/messages",
        },
        "httpLlmResponse": map[string]interface{}{
            "provider": "ANTHROPIC",
            "model":    "claude-sonnet-4-20250514",
            "completion": map[string]interface{}{
                "text":       "Paris is the capital of France.",
                "stopReason": "end_turn",
                "usage": map[string]int{
                    "inputTokens":         150,
                    "outputTokens":        10,
                    "cachedInputTokens":   120,
                    "cacheCreationTokens": 0,
                    "reasoningTokens":     0,
                },
            },
        },
    })
    req, _ := http.NewRequest(http.MethodPut,
        "http://localhost:1080/mockserver/expectation", bytes.NewReader(body))
    req.Header.Set("Content-Type", "application/json")
    http.DefaultClient.Do(req)
}

using System.Net.Http;
using System.Text;
using System.Text.Json;

var expectation = new
{
    httpRequest = new { method = "POST", path = "/v1/messages" },
    httpLlmResponse = new
    {
        provider = "ANTHROPIC",
        model = "claude-sonnet-4-20250514",
        completion = new
        {
            text = "Paris is the capital of France.",
            stopReason = "end_turn",
            usage = new
            {
                inputTokens = 150,
                outputTokens = 10,
                cachedInputTokens = 120,
                cacheCreationTokens = 0,
                reasoningTokens = 0
            }
        }
    }
};
using var client = new HttpClient();
var json = JsonSerializer.Serialize(expectation);
await client.PutAsync(
    "http://localhost:1080/mockserver/expectation",
    new StringContent(json, Encoding.UTF8, "application/json"));

use serde_json::json;

let client = reqwest::blocking::Client::new();
client.put("http://localhost:1080/mockserver/expectation")
    .json(&json!({
        "httpRequest": {
            "method": "POST",
            "path": "/v1/messages"
        },
        "httpLlmResponse": {
            "provider": "ANTHROPIC",
            "model": "claude-sonnet-4-20250514",
            "completion": {
                "text": "Paris is the capital of France.",
                "stopReason": "end_turn",
                "usage": {
                    "inputTokens": 150,
                    "outputTokens": 10,
                    "cachedInputTokens": 120,
                    "cacheCreationTokens": 0,
                    "reasoningTokens": 0
                }
            }
        }
    }))
    .send()
    .unwrap();

<?php
$expectation = [
    'httpRequest' => ['method' => 'POST', 'path' => '/v1/messages'],
    'httpLlmResponse' => [
        'provider' => 'ANTHROPIC',
        'model' => 'claude-sonnet-4-20250514',
        'completion' => [
            'text' => 'Paris is the capital of France.',
            'stopReason' => 'end_turn',
            'usage' => [
                'inputTokens' => 150,
                'outputTokens' => 10,
                'cachedInputTokens' => 120,
                'cacheCreationTokens' => 0,
                'reasoningTokens' => 0,
            ],
        ],
    ],
];
$ch = curl_init('http://localhost:1080/mockserver/expectation');
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'PUT');
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($expectation));
curl_setopt($ch, CURLOPT_HTTPHEADER, ['Content-Type: application/json']);
curl_exec($ch);
curl_close($ch);

curl -v -X PUT "http://localhost:1080/mockserver/expectation" -d '{
  "httpRequest": {
    "method": "POST",
    "path": "/v1/messages"
  },
  "httpLlmResponse": {
    "provider": "ANTHROPIC",
    "model": "claude-sonnet-4-20250514",
    "completion": {
      "text": "Paris is the capital of France.",
      "stopReason": "end_turn",
      "usage": {
        "inputTokens": 150,
        "outputTokens": 10,
        "cachedInputTokens": 120,
        "cacheCreationTokens": 0,
        "reasoningTokens": 0
      }
    }
  }
}'

When MockServer proxies traffic to a real LLM provider, these fields are decoded from the provider's usage shape and stored. The GenAI telemetry spans report them as mockserver.gen_ai.usage.cached_input_tokens, mockserver.gen_ai.usage.cache_creation_tokens, and mockserver.gen_ai.usage.reasoning_tokens.

Streaming — Client Examples

These examples show how to create a streaming LLM expectation with realistic timing physics. The Java client uses the typed fluent API; all other clients send the expectation as raw JSON.

import static java.util.concurrent.TimeUnit.MILLISECONDS;
import static org.mockserver.client.Llm.jitter;
import static org.mockserver.client.Llm.timeToFirstToken;
import static org.mockserver.client.Llm.tokensPerSecond;
import static org.mockserver.client.LlmMockBuilder.llmMock;
import static org.mockserver.model.Completion.completion;
import static org.mockserver.model.Provider.OPENAI;

llmMock("/v1/chat/completions")
    .withProvider(OPENAI)
    .withModel("gpt-4o")
    .respondingWith(
        completion()
            .withText("This response is streamed token by token.")
            .streaming()
            .withStreamingPhysics(
                timeToFirstToken(200, MILLISECONDS),
                tokensPerSecond(50),
                jitter(0.1))
    )
    .applyTo(mockServerClient);

var mockServerClient = require('mockserver-client').mockServerClient;

mockServerClient("localhost", 1080).mockAnyResponse({
    "httpRequest": {
        "method": "POST",
        "path": "/v1/chat/completions"
    },
    "httpLlmResponse": {
        "provider": "OPENAI",
        "model": "gpt-4o",
        "completion": {
            "text": "This response is streamed token by token.",
            "streaming": true,
            "streamingPhysics": {
                "timeToFirstToken": { "timeUnit": "MILLISECONDS", "value": 200 },
                "tokensPerSecond": 50,
                "jitter": 0.1,
                "seed": 42
            }
        }
    }
}).then(
    function () { console.log("expectation created"); },
    function (error) { console.log(error); }
);

import requests

requests.put(
    "http://localhost:1080/mockserver/expectation",
    json={
        "httpRequest": {
            "method": "POST",
            "path": "/v1/chat/completions"
        },
        "httpLlmResponse": {
            "provider": "OPENAI",
            "model": "gpt-4o",
            "completion": {
                "text": "This response is streamed token by token.",
                "streaming": True,
                "streamingPhysics": {
                    "timeToFirstToken": {"timeUnit": "MILLISECONDS", "value": 200},
                    "tokensPerSecond": 50,
                    "jitter": 0.1,
                    "seed": 42
                }
            }
        }
    }
)

require 'net/http'
require 'json'

uri = URI('http://localhost:1080/mockserver/expectation')
http = Net::HTTP.new(uri.host, uri.port)
req = Net::HTTP::Put.new(uri.path, 'Content-Type' => 'application/json')
req.body = JSON.generate({
  'httpRequest' => { 'method' => 'POST', 'path' => '/v1/chat/completions' },
  'httpLlmResponse' => {
    'provider' => 'OPENAI',
    'model' => 'gpt-4o',
    'completion' => {
      'text' => 'This response is streamed token by token.',
      'streaming' => true,
      'streamingPhysics' => {
        'timeToFirstToken' => { 'timeUnit' => 'MILLISECONDS', 'value' => 200 },
        'tokensPerSecond' => 50,
        'jitter' => 0.1,
        'seed' => 42
      }
    }
  }
})
http.request(req)

package main

import (
    "bytes"
    "encoding/json"
    "net/http"
)

func createStreamingLlmExpectation() {
    body, _ := json.Marshal(map[string]interface{}{
        "httpRequest": map[string]interface{}{
            "method": "POST",
            "path":   "/v1/chat/completions",
        },
        "httpLlmResponse": map[string]interface{}{
            "provider": "OPENAI",
            "model":    "gpt-4o",
            "completion": map[string]interface{}{
                "text":      "This response is streamed token by token.",
                "streaming": true,
                "streamingPhysics": map[string]interface{}{
                    "timeToFirstToken": map[string]interface{}{
                        "timeUnit": "MILLISECONDS", "value": 200,
                    },
                    "tokensPerSecond": 50,
                    "jitter":          0.1,
                    "seed":            42,
                },
            },
        },
    })
    req, _ := http.NewRequest(http.MethodPut,
        "http://localhost:1080/mockserver/expectation", bytes.NewReader(body))
    req.Header.Set("Content-Type", "application/json")
    http.DefaultClient.Do(req)
}

using System.Net.Http;
using System.Text;
using System.Text.Json;

var expectation = new
{
    httpRequest = new { method = "POST", path = "/v1/chat/completions" },
    httpLlmResponse = new
    {
        provider = "OPENAI",
        model = "gpt-4o",
        completion = new
        {
            text = "This response is streamed token by token.",
            streaming = true,
            streamingPhysics = new
            {
                timeToFirstToken = new { timeUnit = "MILLISECONDS", value = 200 },
                tokensPerSecond = 50,
                jitter = 0.1,
                seed = 42
            }
        }
    }
};
using var client = new HttpClient();
var json = JsonSerializer.Serialize(expectation);
await client.PutAsync(
    "http://localhost:1080/mockserver/expectation",
    new StringContent(json, Encoding.UTF8, "application/json"));

use serde_json::json;

let client = reqwest::blocking::Client::new();
client.put("http://localhost:1080/mockserver/expectation")
    .json(&json!({
        "httpRequest": {
            "method": "POST",
            "path": "/v1/chat/completions"
        },
        "httpLlmResponse": {
            "provider": "OPENAI",
            "model": "gpt-4o",
            "completion": {
                "text": "This response is streamed token by token.",
                "streaming": true,
                "streamingPhysics": {
                    "timeToFirstToken": { "timeUnit": "MILLISECONDS", "value": 200 },
                    "tokensPerSecond": 50,
                    "jitter": 0.1,
                    "seed": 42
                }
            }
        }
    }))
    .send()
    .unwrap();

<?php
$expectation = [
    'httpRequest' => ['method' => 'POST', 'path' => '/v1/chat/completions'],
    'httpLlmResponse' => [
        'provider' => 'OPENAI',
        'model' => 'gpt-4o',
        'completion' => [
            'text' => 'This response is streamed token by token.',
            'streaming' => true,
            'streamingPhysics' => [
                'timeToFirstToken' => ['timeUnit' => 'MILLISECONDS', 'value' => 200],
                'tokensPerSecond' => 50,
                'jitter' => 0.1,
                'seed' => 42,
            ],
        ],
    ],
];
$ch = curl_init('http://localhost:1080/mockserver/expectation');
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'PUT');
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($expectation));
curl_setopt($ch, CURLOPT_HTTPHEADER, ['Content-Type: application/json']);
curl_exec($ch);
curl_close($ch);

curl -v -X PUT "http://localhost:1080/mockserver/expectation" -d '{
  "httpRequest": {
    "method": "POST",
    "path": "/v1/chat/completions"
  },
  "httpLlmResponse": {
    "provider": "OPENAI",
    "model": "gpt-4o",
    "completion": {
      "text": "This response is streamed token by token.",
      "streaming": true,
      "streamingPhysics": {
        "timeToFirstToken": { "timeUnit": "MILLISECONDS", "value": 200 },
        "tokensPerSecond": 50,
        "jitter": 0.1,
        "seed": 42
      }
    }
  }
}'

Java Client API (Simple Completion)

The simplest LLM expectation returns a single completion for a request. The Java client uses the typed when(...).respond(llmResponse()...) fluent API; all other clients send the same expectation as raw JSON:

import static org.mockserver.model.HttpRequest.request;
import static org.mockserver.model.HttpLlmResponse.llmResponse;
import static org.mockserver.model.Completion.completion;
import org.mockserver.model.Provider;

mockServerClient
    .when(
        request()
            .withMethod("POST")
            .withPath("/v1/chat/completions")
    )
    .respond(
        llmResponse()
            .withProvider(Provider.OPENAI)
            .withModel("gpt-4o")
            .withCompletion(
                completion()
                    .withText("Hello from MockServer!")
                    .withStopReason("stop")
            )
    );

var mockServerClient = require('mockserver-client').mockServerClient;

mockServerClient("localhost", 1080).mockAnyResponse({
    "httpRequest": {
        "method": "POST",
        "path": "/v1/chat/completions"
    },
    "httpLlmResponse": {
        "provider": "OPENAI",
        "model": "gpt-4o",
        "completion": {
            "text": "Hello from MockServer!",
            "stopReason": "stop"
        }
    }
}).then(
    function () { console.log("expectation created"); },
    function (error) { console.log(error); }
);

import requests

requests.put(
    "http://localhost:1080/mockserver/expectation",
    json={
        "httpRequest": {
            "method": "POST",
            "path": "/v1/chat/completions"
        },
        "httpLlmResponse": {
            "provider": "OPENAI",
            "model": "gpt-4o",
            "completion": {
                "text": "Hello from MockServer!",
                "stopReason": "stop"
            }
        }
    }
)

require 'net/http'
require 'json'

uri = URI('http://localhost:1080/mockserver/expectation')
http = Net::HTTP.new(uri.host, uri.port)
req = Net::HTTP::Put.new(uri.path, 'Content-Type' => 'application/json')
req.body = JSON.generate({
  'httpRequest' => { 'method' => 'POST', 'path' => '/v1/chat/completions' },
  'httpLlmResponse' => {
    'provider' => 'OPENAI',
    'model' => 'gpt-4o',
    'completion' => {
      'text' => 'Hello from MockServer!',
      'stopReason' => 'stop'
    }
  }
})
http.request(req)

package main

import (
    "bytes"
    "encoding/json"
    "net/http"
)

func createLlmExpectation() {
    body, _ := json.Marshal(map[string]interface{}{
        "httpRequest": map[string]interface{}{
            "method": "POST",
            "path":   "/v1/chat/completions",
        },
        "httpLlmResponse": map[string]interface{}{
            "provider": "OPENAI",
            "model":    "gpt-4o",
            "completion": map[string]interface{}{
                "text":       "Hello from MockServer!",
                "stopReason": "stop",
            },
        },
    })
    req, _ := http.NewRequest(http.MethodPut,
        "http://localhost:1080/mockserver/expectation", bytes.NewReader(body))
    req.Header.Set("Content-Type", "application/json")
    http.DefaultClient.Do(req)
}

using System.Net.Http;
using System.Text;
using System.Text.Json;

var expectation = new
{
    httpRequest = new { method = "POST", path = "/v1/chat/completions" },
    httpLlmResponse = new
    {
        provider = "OPENAI",
        model = "gpt-4o",
        completion = new
        {
            text = "Hello from MockServer!",
            stopReason = "stop"
        }
    }
};
using var client = new HttpClient();
var json = JsonSerializer.Serialize(expectation);
await client.PutAsync(
    "http://localhost:1080/mockserver/expectation",
    new StringContent(json, Encoding.UTF8, "application/json"));

use serde_json::json;

let client = reqwest::blocking::Client::new();
client.put("http://localhost:1080/mockserver/expectation")
    .json(&json!({
        "httpRequest": {
            "method": "POST",
            "path": "/v1/chat/completions"
        },
        "httpLlmResponse": {
            "provider": "OPENAI",
            "model": "gpt-4o",
            "completion": {
                "text": "Hello from MockServer!",
                "stopReason": "stop"
            }
        }
    }))
    .send()
    .unwrap();

<?php
$expectation = [
    'httpRequest' => ['method' => 'POST', 'path' => '/v1/chat/completions'],
    'httpLlmResponse' => [
        'provider' => 'OPENAI',
        'model' => 'gpt-4o',
        'completion' => [
            'text' => 'Hello from MockServer!',
            'stopReason' => 'stop',
        ],
    ],
];
$ch = curl_init('http://localhost:1080/mockserver/expectation');
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'PUT');
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($expectation));
curl_setopt($ch, CURLOPT_HTTPHEADER, ['Content-Type: application/json']);
curl_exec($ch);
curl_close($ch);

curl -v -X PUT "http://localhost:1080/mockserver/expectation" -d '{
  "httpRequest": {
    "method": "POST",
    "path": "/v1/chat/completions"
  },
  "httpLlmResponse": {
    "provider": "OPENAI",
    "model": "gpt-4o",
    "completion": {
      "text": "Hello from MockServer!",
      "stopReason": "stop"
    }
  }
}'

Related Pages

• Creating Expectations — full request matcher and action reference
• Chaos Testing & Fault Injection — HTTP and LLM chaos profiles
• MCP Tools Reference — mock_llm_completion and create_llm_conversation tools for AI agents
• Inspect AI Agent Traffic — proxy and inspect real LLM API traffic
• Observability — LLM token/cost Prometheus metrics and OpenTelemetry GenAI spans
• AI Agent Optimisation — export a structured optimisation brief from captured LLM traffic covering spend reduction and agent design