Package org.mockserver.client

Class LlmFailoverBuilder

java.lang.Object
org.mockserver.client.LlmFailoverBuilder
public class LlmFailoverBuilder extends Object
Builder for LLM provider failover/retry test scenarios.

Produces an ordered array of Expectations that simulate a provider returning failures for the first N attempts, then succeeding on subsequent attempts. This lets you point a retry-capable LLM client (LiteLLM, Envoy AI Gateway, an SDK's own retry config) at MockServer and deterministically assert that failover/retry logic works.

The mechanism relies on MockServer's expectation matching order and Times exhaustion: failure expectations are registered first with limited Times, so they are matched and consumed before the unlimited success expectation is reached.

Example: 

 llmFailover()
     .withPath("/v1/chat/completions")
     .withProvider(Provider.OPENAI)
     .withModel("gpt-4o")
     .failWith(503)               // attempt 1 fails with 503
     .failWith(429)               // attempt 2 fails with 429
     .thenRespondWith(completion)  // attempt 3+ succeeds
     .build();                    // returns Expectation[]

  • Method Details

    • llmFailover

      public static LlmFailoverBuilder llmFailover()
      Entry point for building an LLM failover scenario.
      Returns:
      a new LlmFailoverBuilder

    • withPath

      public LlmFailoverBuilder withPath(String path)
      Set the request path to match (e.g. /v1/chat/completions).
      Parameters:
      path - the path
      Returns:
      this builder

    • withProvider

      public LlmFailoverBuilder withProvider(Provider provider)
      Set the LLM provider for the success response encoding.
      Parameters:
      provider - the provider
      Returns:
      this builder

    • withModel

      public LlmFailoverBuilder withModel(String model)
      Set the model name for the success response.
      Parameters:
      model - the model name
      Returns:
      this builder

    • failWith

      public LlmFailoverBuilder failWith(int statusCode)
      Add a single failure attempt returning the given HTTP status code with a default provider-plausible error body.
      Parameters:
      statusCode - the HTTP status code (e.g. 503, 429, 500)
      Returns:
      this builder

    • failWith

      public LlmFailoverBuilder failWith(int statusCode, String errorBody)
      Add a single failure attempt returning the given HTTP status code with a custom error body.
      Parameters:
      statusCode - the HTTP status code (e.g. 503, 429, 500)
      errorBody - the response body to return
      Returns:
      this builder

    • failWith

      public LlmFailoverBuilder failWith(int statusCode, int count)
      Add count consecutive failure attempts returning the given HTTP status code with a default provider-plausible error body.
      Parameters:
      statusCode - the HTTP status code
      count - number of consecutive failures
      Returns:
      this builder

    • thenRespondWith

      public LlmFailoverBuilder thenRespondWith(Completion completion)
      Set the completion to return after all failures are exhausted.
      Parameters:
      completion - the success completion
      Returns:
      this builder

    • applyTo

      public Expectation[] applyTo(MockServerClient client)
      Build and register all expectations with the MockServerClient.
      Parameters:
      client - the MockServerClient
      Returns:
      the created expectations

    • build

      public Expectation[] build()
      Build all expectations without registering them.

      Returns an ordered array of failure expectations followed by a single success expectation with Times.unlimited(). Consecutive failures with the same status code (and body) are coalesced into one expectation with Times.exactly(count); otherwise each failure is its own expectation with Times.exactly(1). Registration order ensures failures are matched first and consumed before the success expectation.

      Returns:
      the array of expectations

    • getFailureCount

      public int getFailureCount()
      Returns the total number of failure attempts configured.
      Returns:
      the failure count