Skip to content

StubDecisionModel

A DecisionModel that answers from a script instead of calling a provider, and records every prompt it was given. It lets you test judges, guardrails and anything else built on the DecisionModel API offline, with no provider and no key.

<dependency>
    <groupId>org.springaicommunity</groupId>
    <artifactId>spring-ai-decision-test</artifactId>
    <scope>test</scope>
</dependency>

The artifact also holds PromptField, for assertions on the input a decision model was given, and ScriptedChatModel, a chat model that replies from a script.

What to test with it

A real model decides what the answers are. Everything after that is your code: the thresholds, the criteria that depend on each other, the code checks, the hazards' actions. That part is deterministic, and worth testing without a model:

  • that a probability just under a threshold fails a criterion, and just over it passes;
  • that a refused question blocks a guardrail turn rather than passing it;
  • that an advisor retries, and what its retry prompt says;
  • that the right fields reach the model.

What a stub cannot tell you is whether your questions are good. For that, run them against a real model in an integration test, and check their stability with DecisionConsistency.

Quick Start

import static org.springaicommunity.decision.test.StubDecisionModel.score;
import static org.springaicommunity.decision.test.StubDecisionModel.yesNo;

class WeatherJudgeTests {

    private final StubDecisionModel model = new StubDecisionModel();

    @Test
    void failsAnImplausibleAnswerOnPlausibilityAlone() {
        this.model.respondWith(Map.of(
                "helpfulness", score(3.0, 0.9, 0.0, 0.0, 0.0, 1.0),
                "is_plausible", yesNo(0.03)));

        DecisionVerdict verdict = judge(this.model).judge(
                "What is the weather in Paris?", "It is -255 degrees Celsius in Paris.");

        assertThat(verdict.failures()).extracting(DecisionFinding::name).containsExactly("is_plausible");
        this.model.verify();
    }
}

Scripting the answers

A stub answers in one of two ways.

Call by call. respondWith(...) and failWith(...) queue the outcome of one call each, consumed in order. A call with nothing queued fails the test, so an unexpected call cannot pass unnoticed, and verify() fails when a queued outcome was never asked for.

StubDecisionModel model = new StubDecisionModel();
model.respondWith(Map.of("is_plausible", yesNo(0.03)))        // first call
     .respondWith(Map.of("is_plausible", yesNo(0.97)))        // second call
     .failWith(new IllegalStateException("provider down"));   // third call

The same every time. answering(...) gives the same answers to every call, or computes them from the prompt. failing(...) fails every call.

StubDecisionModel steady = StubDecisionModel.answering(Map.of("urgent", yesNo(0.9)));

StubDecisionModel byText = StubDecisionModel.answering(prompt -> {
    Object text = ((FieldsInput) prompt.getInstructions()).getFields().get("text");
    boolean flagged = String.valueOf(text).contains("bomb");
    return Map.of("physical_harm", yesNo(flagged ? 0.95 : 0.01));
});

StubDecisionModel down = StubDecisionModel.failing(new IllegalStateException("provider down"));

Only the questions a prompt asks are answered, so one script can serve prompts that ask for a subset of it. A question the script has no answer for fails the call, as a provider that left a question out would.

Building answers

Static factories build each kind of answer:

Factory Builds
yesNo(probability) a yes/no answer
choice(value, confidence) a choice with a confidence and no probabilities
choice(value, confidence, probabilities(...)) a choice with the probability of each option
score(value) a score with neither a confidence nor probabilities
score(value, confidence, probabilities...) a score with the probability of each level, lowest first
refusal(reason) a refusal, for any question
choice("billing", 0.82, probabilities("billing", 0.9, "technical", 0.1));
score(2.34, 0.51, 0.0, 0.07, 0.52, 0.41);   // a four-level rubric
refusal("policy");

The confidence can be null, which is how to test code against a provider that reports none.

Answers are checked like a provider's

The response is built with the DecisionResponse builder, which rejects a score above its rubric, or probabilities that do not match the number of levels. A scripted answer that could not come from a real model fails the test rather than passing through.

Asserting on the prompt

Every prompt is recorded. Read them back with prompts() and lastPrompt():

assertThat(model.prompts()).hasSize(2);
assertThat(model.lastPrompt().getOptions()).isSameAs(options);

To check a prompt at the moment of the call, start the script with expect(). PromptField asserts on one value of the prompt's fields, addressed by a JSON path:

import static org.springaicommunity.decision.test.PromptField.field;

model.expect()
    .andExpect(field("$.assistant_answer").value("It is 15 degrees Celsius in Paris."))
    .andExpect(field("$.tool_calls[0].name").value("weather"))
    .andExpect(field("$.tool_calls.length()").value(1))
    .andExpect(field("$.supporting_context").doesNotExist())
    .respondWith(Map.of("is_plausible", yesNo(0.97)));

value(...) also takes a Hamcrest matcher, such as containsString("Paris"). A broken expectation fails the test with an AssertionError when the call is made.

PromptField reads a fields input, which is what the judge, the guardrail and the chat adapter send. For a text or conversation input, assert on prompt.getInstructions() directly with andExpect(prompt -> ...).

ScriptedChatModel

A ChatModel that replies with a scripted sequence of answers and records the prompts it was given. Use it to test an advisor chain end to end:

StubDecisionModel judgeModel = new StubDecisionModel();
judgeModel.respondWith(FAILING).respondWith(PASSING);
ScriptedChatModel chatModel = new ScriptedChatModel(
        "It is -255 degrees Celsius in Paris.", "It is 15 degrees Celsius in Paris.");

String content = ChatClient.builder(chatModel)
    .defaultAdvisors(DecisionSelfRefineAdvisor.builder().judge(judge(judgeModel)).build())
    .build()
    .prompt("What is the weather in Paris?")
    .call()
    .content();

assertThat(content).isEqualTo("It is 15 degrees Celsius in Paris.");
assertThat(chatModel.callCount()).isEqualTo(2);
assertThat(chatModel.userMessageOfCall(1)).contains("Your previous answer was rejected");

Two scripted steps go beyond plain text:

Step The model replies with
ScriptedChatModel.toolCall(name, arguments) a tool call, so ChatClient runs its tool loop
ScriptedChatModel.afterThinking(thinking, answer) a thinking block as a generation of its own, ahead of the answer

receivedPrompts(), callCount() and userMessageOfCall(index) read back what the model was asked.

See Also