Package org.mockserver.llm

Class TokenCounter

java.lang.Object
org.mockserver.llm.TokenCounter
public final class TokenCounter extends Object
Pure, deterministic approximate token counter for LLM text.

This is an estimate, not a real tokenizer. It does not implement byte-pair encoding (BPE), SentencePiece, or any provider's exact vocabulary, so its counts will differ from a provider's billed token counts — typically within roughly ±20% for ordinary English prose, and further off for code, non-Latin scripts, or long runs of punctuation. It exists only so MockServer can populate plausible usage numbers when a mocked completion omits them, and to back the rough character-based estimates already used for token-quota accounting.

Heuristic

The estimate blends two cheap signals that bracket real tokenizer behaviour:
  • Characters ÷ 4 — the long-standing rule of thumb that ~4 characters of English text map to one token.
  • Words × 4/3 — real tokenizers split common words into ~1.3 tokens on average (sub-word units, leading spaces, suffixes).
The two are averaged. Whitespace runs are collapsed for the word count and a small allowance is added for punctuation density, which a BPE tokenizer tends to split into separate tokens. The result is clamped to at least 1 for any non-empty text (every real request costs at least one token) and 0 for null/empty.

The method is pure and side-effect free: the same input always yields the same count, so it never makes a test flaky.

  • Method Details

    • estimateTokens

      public static int estimateTokens(String text)
      Estimate the approximate token count for a single piece of text.
      Parameters:
      text - the text to estimate (may be null)
      Returns:
      0 for null/empty text, otherwise an approximate token count >= 1

    • estimatePromptTokens

      public static int estimatePromptTokens(ParsedConversation conversation)
      Estimate the approximate prompt (input) token count for a decoded conversation: the sum of the per-message text estimates plus a small fixed per-message overhead (real chat formats wrap each message in role markers / delimiters that cost a few tokens). Tool-call arguments and tool results carried on a message are included in its text estimate.
      Parameters:
      conversation - the decoded conversation (may be null)
      Returns:
      0 for a null/empty conversation, otherwise an approximate prompt token count >= 1

    • estimateCompletionTokens

      public static int estimateCompletionTokens(String text, List<ToolUse> toolCalls)
      Estimate the approximate completion (output) token count for the text and tool-call arguments a mocked completion would return.
      Parameters:
      text - the response text (may be null)
      toolCalls - the response tool calls (may be null/empty)
      Returns:
      an approximate completion token count (0 when there is no output at all)