Package org.mockserver.llm

Class TokenCounter

java.lang.Object

org.mockserver.llm.TokenCounter

public final class TokenCounter
extends Object

Pure, deterministic approximate token counter and lightweight subword segmenter for LLM text.

This is an estimate, not a real tokenizer. It does not load byte-pair encoding (BPE) merges, SentencePiece, or any provider's exact vocabulary — no dependency, no embedded vocabulary. Instead it approximates GPT-style BPE segmentation with a handful of cheap structural rules, so its counts land close to a provider's billed counts (typically within roughly ±15% for ordinary English prose, further off for code, non-Latin scripts, or long runs of punctuation). It exists so MockServer can populate plausible usage numbers when a mocked completion omits them, back the rough token-quota accounting, and drive subword-granular streaming deltas.

Segmentation heuristic (BPE approximation)

The text is walked once and split into runs, each scored/segmented by class:

• Words (letters/digits, with - ' _ treated as word-internal so open-source stays one word): a leading space is absorbed (real BPE merges a leading space into the following token), the word is split at lower→Upper case boundaries and letter↔digit boundaries (so ChatGPT→Chat,GPT), and each resulting sub-run costs 1 token plus one extra per ~5 characters beyond the first five (so short/common words stay whole while long words split into subword units).
• CJK (Han/Hangul ~1.2 tokens/char, Kana ~0.5): scripts that tokenizers split far more densely than Latin text.
• Punctuation/symbols: ~1 token each — BPE tends to split these into their own tokens.
• Whitespace: a plain space is free (absorbed into the next word); each newline/tab costs ~1 token.

The result is clamped to at least 1 for any non-empty text (every real request costs at least one token) and 0 for null/empty.

The methods are pure and side-effect free: the same input always yields the same result, so they never make a test flaky.

Method Summary

Modifier and Type	Method	Description
static int	estimateCompletionTokens(String text, List<ToolUse> toolCalls)	Estimate the approximate completion (output) token count for the text and tool-call arguments a mocked completion would return.
static int	estimatePromptTokens(ParsedConversation conversation)	Estimate the approximate prompt (input) token count for a decoded conversation: the sum of the per-message text estimates plus a small fixed per-message overhead (real chat formats wrap each message in role markers / delimiters that cost a few tokens).
static int	estimateTokens(String text)	Estimate the approximate token count for a single piece of text using the BPE-approximating segmentation described in the class javadoc.
static List<String>	segmentForStreaming(String text)	Segment text into subword-sized, concatenation-exact pieces approximating BPE tokens for streaming.
static List<String>	streamingTextTokens(String text, StreamingPhysics physics)	Split completion text into the ordered list of streamed delta strings for a streaming response, honouring the streaming physics' subwordStreaming mode.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

estimateTokens

public static int estimateTokens(String text)

Estimate the approximate token count for a single piece of text using the BPE-approximating segmentation described in the class javadoc.

Parameters:

text - the text to estimate (may be null)

Returns:

0 for null/empty text, otherwise an approximate token count >= 1

estimatePromptTokens

public static int estimatePromptTokens(ParsedConversation conversation)

Estimate the approximate prompt (input) token count for a decoded conversation: the sum of the per-message text estimates plus a small fixed per-message overhead (real chat formats wrap each message in role markers / delimiters that cost a few tokens). Tool-call arguments and tool results carried on a message are included in its text estimate.

Parameters:

conversation - the decoded conversation (may be null)

Returns:

0 for a null/empty conversation, otherwise an approximate prompt token count >= 1

estimateCompletionTokens

public static int estimateCompletionTokens(String text,
 List<ToolUse> toolCalls)

Estimate the approximate completion (output) token count for the text and tool-call arguments a mocked completion would return.

Parameters:

text - the response text (may be null)

toolCalls - the response tool calls (may be null/empty)

Returns:

an approximate completion token count (0 when there is no output at all)

streamingTextTokens

public static List<String> streamingTextTokens(String text,
 StreamingPhysics physics)

Split completion text into the ordered list of streamed delta strings for a streaming response, honouring the streaming physics' subwordStreaming mode.

When subwordStreaming is on — which is now the default for a null physics or an unset flag — the text is segmented into subword-sized pieces via segmentForStreaming(String), giving finer, more realistic per-token deltas. Only an explicit subwordStreaming=false selects the legacy whitespace-boundary split (each word and each whitespace run its own delta). Either way the concatenation of the returned pieces equals the input text exactly, and every piece is non-empty.

Parameters:

text - the completion text (may be null/empty)

physics - the streaming physics (may be null)

Returns:

the ordered, non-empty delta pieces (empty list for null/empty text)

segmentForStreaming

public static List<String> segmentForStreaming(String text)

Segment text into subword-sized, concatenation-exact pieces approximating BPE tokens for streaming. A leading space is absorbed into the following word's first piece (BPE-style), words are chunked into 4-character subword units, CJK and punctuation characters are emitted individually. The pieces joined in order equal the input exactly.

Parameters:

text - the text to segment (may be null/empty)

Returns:

the ordered, non-empty pieces (empty list for null/empty text)