Observability

MockServer provides two observability channels: Prometheus metrics for counters, gauges, and histograms; and OpenTelemetry (OTLP) for trace and metric export. Both are opt-in and have zero overhead when disabled.

• Prometheus metrics
• LLM token and cost metrics
• OpenTelemetry (OTLP) export
• GenAI spans
• W3C trace context propagation
• Configuration reference

Prometheus Metrics

Enable Prometheus metrics by setting metricsEnabled to true. MockServer then exposes a scrape endpoint at /mockserver/metrics in Prometheus text exposition format. When metrics are disabled, this endpoint returns 404.

# Start MockServer with metrics enabled
docker run --rm -p 1080:1080 \
  -e MOCKSERVER_METRICS_ENABLED=true \
  mockserver/mockserver:9.0.0

# Scrape metrics
curl http://localhost:1080/mockserver/metrics

Available metrics

Naming convention: the core request-tracking gauges are exposed with unprefixed names (e.g. requests_received_count, expectations_not_matched_count). Counter and histogram metrics, and the operational gauges (mock_server_active_service_chaos, mock_server_expectations_by_type, mock_server_build_info), all use a mock_server_ prefix (e.g. mock_server_request_duration_seconds). Note that the _total suffix is appended to counter names in the exposition output, so mock_server_http_chaos_injected appears as mock_server_http_chaos_injected_total on the /mockserver/metrics endpoint and in PromQL queries and Grafana.

Request and expectation matching

Each of these four gauges (plus llm_chaos_injected_count) is genuinely monotonic (only ever increases). Because the four gauge names above are read by the dashboard UI and existing Grafana dashboards, they are kept unchanged; alongside each one MockServer additionally publishes a proper Prometheus Counter with a _total suffix. Use the _total counters in PromQL rate()/increase() queries — they are true monotonic counters, which those functions model correctly:

# requests-per-second over the last 5 minutes
rate(mock_server_requests_received_total[5m])

Per-expectation match counter (opt-in)

When perExpectationMetricsEnabled is true (alongside metricsEnabled), MockServer registers an additional counter:

This counter is off by default: each active expectation adds one Prometheus label value, so cardinality grows with the number of registered expectations. Enable it only when you need per-expectation visibility. See Per-Expectation Match Counters configuration.

Action execution (one per action type)

Additional action counters exist for template, callback, and other action types. See the full list by scraping the endpoint.

Request latency histogram

mock_server_request_duration_seconds is a Prometheus histogram of request handling duration (receipt to response), with buckets from 0.5 ms to 10 s. Use it to derive latency percentiles:

histogram_quantile(0.95, sum by (le) (rate(mock_server_request_duration_seconds_bucket[1m])))

This histogram times MockServer's request handling only: it stops when the response is handed over to be written. It does not include writing the response to the network, so a slow client, a response sent in delayed chunks (chunkDelay), or time waiting for a busy network thread does not show up in it.

Request latency including the network write

mock_server_request_transport_duration_seconds is a Prometheus histogram of the time from MockServer reading a request's headers to the last byte of its response being written to the network. It covers HTTP/1.1 and HTTP/2 (each HTTP/2 stream is timed separately; HTTP/3 is not timed). Buckets run from 0.5 ms to 10 s, with fine buckets between 1 ms and 100 ms.

Use it to tell whether slow requests seen by a client were slow inside MockServer. If your client sees slow requests but this histogram does not, the time was spent outside MockServer: in the client, the network, or before MockServer read the request. Compare the share of requests over 5 ms, for example:

1 - sum(rate(mock_server_request_transport_duration_seconds_bucket{le="0.005"}[1m]))
  / sum(rate(mock_server_request_transport_duration_seconds_count[1m]))

It counts every request, including calls to MockServer's own API such as creating expectations and scraping metrics. A streamed response (for example server-sent events) is timed until its last byte, so long-lived streams show up as long requests.

Time from reading a request to decoding its headers

mock_server_request_read_to_decode_duration_seconds is a Prometheus histogram of the time just before the one above: from MockServer reading the first bytes of a request from the network to it having decoded the request's headers. It covers HTTP/1.1 only. It is usually well under a millisecond; it grows when a client sends its headers in pieces with pauses between them, or sends several requests at once without waiting for responses (each waits for the ones before it to be handled). Its buckets start at 0.1 ms and include 5 ms, so the same share-over-5-ms query works with its name in place of the one above.

Neither histogram includes time a request spends waiting in the operating system before MockServer reads it.

Build info

mock_server_build_info is a gauge with labels version, major_minor_version, group_id, artifact_id, and git_hash.

JVM runtime

When metrics are enabled, MockServer also exposes JVM health gauges:

Memory and capacity

When metrics are enabled, MockServer exposes how much room the expectation store and the TCP accept queue have left — the "is the server running out of room?" signals. All are read live at scrape time, so they add nothing to the request path.

Event log ring buffer

Before a log entry is available to retrieve, MockServer briefly holds it in an in-memory queue while it finishes being processed. These gauges are read live at scrape time and cover two separate places an entry's memory is held — the short-lived processing queue, and the longer-lived store entries are kept in once processed — so a scrape can tell which one is holding the heap if event-log memory use grows:

Forward/proxy metrics

Forwarded and proxied requests are tracked per upstream host: mock_server_forward_request_duration_seconds (a histogram of forward latency) and mock_server_forward_requests_total (a counter by status_class) are both labelled with upstream_host (host only, never the full URL or path); mock_server_forward_upstream_protocol_total additionally labels the protocol actually negotiated upstream (http2 via ALPN, or http1_1). To bound cardinality, at most 500 distinct upstream hosts get their own label value; any further hosts are grouped under _other. A null or empty host is recorded as unknown.

Chaos metrics

When chaos testing is active, additional metrics track fault injection:

• mock_server_http_chaos_injected_total — counter with a fault_type label (drop, error, latency, truncate, malformed, slow, quota, rateLimit, graphql)
• mock_server_active_service_chaos — gauge per fault_type of currently active chaos profiles
• mock_server_chaos_auto_halt_total — counter that increments each time the chaos auto-halt circuit-breaker triggers

Cluster metrics

MockServer also exposes the size of the cluster it belongs to:

• mock_server_cluster_members — gauge of the number of members in the MockServer cluster, read live at scrape time. It reads 1 for a single-node deployment (the default in-memory backend, or Infinispan in LOCAL mode) and the real fleet size when clustering is enabled. The same membership detail is available as JSON from the GET /mockserver/cluster control-plane endpoint.

LLM Token and Cost Metrics

When both metricsEnabled and llmMetricsEnabled are true, three additional Prometheus counters track LLM usage:

These counters are incremented on both the mock path (when MockServer serves an httpLlmResponse) and the forward/proxy path (when MockServer forwards requests to a real LLM provider). Cost estimation uses an internal pricing table and is approximate. A forwarded call whose response is streamed is counted when the stream ends, from the usage the provider reports in it; a stream that reports no usage adds nothing to these counters (see Cost Budget for what each provider needs).

The cost-budget circuit-breaker (mock_server_llm_cost_budget_tripped_total counter) is documented in LLM Response Mocking → Cost Budget.

# Example: total LLM cost rate per hour
sum(rate(mock_server_llm_cost_usd_total[1h]))

Three further gauges expose the headline verdict of the latest LLM optimisation report. They have no labels (single global gauges) and report the figures from the most recently built optimisation report — 0 until a report has been built (and again after a server reset), so build the report periodically (via the dashboard, the REST endpoint, or the export_optimisation_report MCP tool) to keep them fresh.

OpenTelemetry (OTLP) Export

MockServer can push metrics and traces to an OpenTelemetry Collector (or any OTLP-compatible backend) via OTLP HTTP/protobuf. Set the collector endpoint and enable the signals you want:

docker run --rm -p 1080:1080 \
  -e MOCKSERVER_OTEL_ENDPOINT=http://otel-collector:4318 \
  -e MOCKSERVER_OTEL_METRICS_ENABLED=true \
  -e MOCKSERVER_OTEL_TRACES_ENABLED=true \
  -e MOCKSERVER_METRICS_ENABLED=true \
  mockserver/mockserver:9.0.0

otelEndpoint is the base URL of the OTLP HTTP collector. MockServer appends /v1/metrics and /v1/traces automatically. If MOCKSERVER_OTEL_ENDPOINT is not set, MockServer falls back to the standard OpenTelemetry OTEL_EXPORTER_OTLP_ENDPOINT environment variable, so existing OTel deployments work without extra configuration. See otelEndpoint in the configuration reference for details.

Metrics export interval: otelMetricsExportIntervalSeconds controls how often metrics are pushed (default 60 seconds, minimum 1 second).

GenAI Spans

When otelTracesEnabled is true, MockServer emits OpenTelemetry GenAI semantic-convention spans for LLM completions. Each span includes:

• gen_ai.system — the provider (e.g. openai, anthropic)
• gen_ai.request.model — the model identifier
• Token usage attributes (input and output tokens)
• Finish reason

GenAI spans fire on two paths:

• Mock path — when MockServer serves an httpLlmResponse. A mocked completion gets a span named for the API it answers (see the table below), and a mocked embeddings or rerank response an embeddings or rerank span. An embeddings or rerank span, and the token metrics, carry the input tokens the mocked response reports in the provider's own usage field. A response whose provider reports no tokens there, such as a Cohere rerank (billed in search units) or Ollama's older /api/embeddings endpoint, gets a span with no token counts; a Bedrock Cohere embedding is counted from its X-Amzn-Bedrock-Input-Token-Count header.
• Forward/proxy path — when MockServer forwards requests to a real LLM provider. The provider is detected from the target host (e.g. api.openai.com maps to OpenAI, api.anthropic.com maps to Anthropic). Only calls to a model get a span: a completion, an embeddings call (embeddings) or a rerank call (rerank). Other requests to the provider, such as listing models or fetching a stored response, get none.

Each span is named <operation> <model>, and its gen_ai.operation.name follows the OpenTelemetry GenAI semantic conventions, which require the specific operation name when one applies:

In MockServer 8.0.0 and earlier every completion was named chat. A trace query or dashboard that filters on gen_ai.operation.name == "chat" to find all completions must also match generate_content and text_completion.

W3C Trace Context Propagation

MockServer can extract and propagate W3C traceparent and tracestate headers across requests and responses. This enables distributed tracing correlation when MockServer sits in a service mesh or test harness.

• otelPropagateTraceContext (default false) — when enabled, MockServer copies the incoming trace context headers to the response, so downstream tracing tooling can correlate the mock response with the original request trace.
• otelGenerateTraceId (default false) — when enabled, MockServer generates a new random W3C trace ID for requests that arrive without a traceparent header.

Configuration Reference

Prometheus

OpenTelemetry

Related Pages

• Configuration Properties — full reference for all observability properties
• LLM Response Mocking — LLM mocking and cost budget
• Chaos Testing & Fault Injection — chaos metrics and auto-halt
• Scalability & Latency — performance benchmarks and tuning