DecisionJudge¶
A model-as-a-judge built out of atomic questions rather than one rubric prompt. Every criterion is answered against the same input in a single call, and each keeps its own threshold.
<dependency>
<groupId>org.springaicommunity</groupId>
<artifactId>spring-ai-decision-judge</artifactId>
</dependency>
The judge needs only the DecisionModel API, so it can be used in a test suite or a batch
evaluation without ChatClient.
What it does¶
A judge holds a list of criteria. Most are a question plus what counts as passing it:
- a yes/no question passes when its probability of yes reaches a minimum
- a score passes when at least half of its probability is on the levels at or above your minimum
- a choice passes when at least half of its probability is on the options you accept
See how a verdict is decided for the details.
The rest are code checks: plain Java predicates for anything the input already settles, such as whether a tool was called. They never reach the model.
judge(...) sends every question in one request and returns a DecisionVerdict: whether
everything passed, a finding per criterion, the raw response, and feedback synthesised from
the criteria that did not.
Class diagram¶
A judge is built from criteria, and each criterion pairs a question with its own pass condition:
classDiagram
direction TB
class Builder {
+options(DecisionOptions) Builder
+yesNo(String, YesNoQuestion, double) Builder
+score(String, ScoreQuestion, double) Builder
+choice(String, ChoiceQuestion, String[]) Builder
+check(String, Predicate~DecisionJudgeInput~, String) Builder
+criterion(DecisionCriterion) Builder
+minConfidence(double) Builder
+failOnError(boolean) Builder
+failFast(boolean) Builder
+failOnInconclusive(boolean) Builder
+build() DecisionJudge
}
class DecisionJudge {
+judge(String, String) DecisionVerdict
+judge(DecisionJudgeInput) DecisionVerdict
+judge(DecisionInput) DecisionVerdict
+criteria() List~DecisionCriterion~
+builder(DecisionModel)$ Builder
}
class DecisionCriterion {
<<sealed interface>>
+name() String
}
class QuestionCriterion {
<<record>>
+String name
+DecisionQuestion question
+double minimum
+Set~String~ acceptedOptions
+Predicate~DecisionJudgeInput~ appliesWhen
+Dependency dependsOn
}
class CodeCriterion {
<<record>>
+String name
+Predicate~DecisionJudgeInput~ check
+String defect
}
class DecisionQuestion {
<<sealed interface>>
}
Builder ..> DecisionJudge : builds
DecisionJudge "1" *-- "1..*" DecisionCriterion : holds
DecisionCriterion <|.. QuestionCriterion
DecisionCriterion <|.. CodeCriterion
QuestionCriterion --> DecisionQuestion : the question asked
note for DecisionQuestion "permits YesNoQuestion,<br/>ChoiceQuestion, ScoreQuestion"
The shape to take from this: a judge is a list of criteria and nothing else. It holds a
DecisionModel and spends at most one call per judge(...), whatever the number of
criteria. It makes none when no question applies, or when failFast skips it. Code checks
run locally and add nothing to that.
How the judge, the advisor and the evaluator fit together¶
These three are a hub and two spokes, not a chain. DecisionJudge is the engine; the
other two are independent adapters that put it behind a Spring AI extension point. Neither
spoke uses the other.
flowchart TB
J["DecisionJudge<br/><small>criteria in, verdict out</small>"]
S["DecisionSelfRefineAdvisor<br/><small>CallAdvisor</small>"]
E["DecisionEvaluator<br/><small>Spring AI Evaluator SPI</small>"]
S --> J
E --> J
| Type | Use it when |
|---|---|
DecisionJudge (this page) |
you want a verdict in your own code, with the per-criterion findings |
DecisionSelfRefineAdvisor |
a ChatClient should judge its own answer and retry when it falls short |
DecisionGuardrailAdvisor |
a turn should be refused rather than retried: an unsafe answer is not a draft |
DecisionEvaluator |
existing code is written against Spring AI's Evaluator, and you want a decision model behind it |
DecisionGuardrailAdvisor is the odd one out: it builds its own hazard batteries rather
than taking a DecisionJudge, because the questions and thresholds are prescribed by the
guardrail policy rather than supplied per use.
Why not just ask a chat model¶
Spring AI ships RelevancyEvaluator and FactCheckingEvaluator. Both prompt a second chat
model and read a yes or a no out of its reply. That fails in two directions. The model may
answer in a form the parser does not expect, and one yes-or-no cannot say which of several
things was wrong. A judge built from typed questions cannot come back malformed, and keeps
the dimensions separate.
| A judge model | DecisionJudge |
|
|---|---|---|
| Verdict | prose to parse | typed answers |
| Dimensions | one overall rating | one threshold per criterion |
| Cost of a third check | a third prompt, or a longer one | negligible: same call |
| Feedback | generated, varies run to run | synthesised from your own rubric |
| Undecided | indistinguishable from failed | reported as INCONCLUSIVE |
| Could not judge | indistinguishable from failed | reported as ERROR |
Quick Start¶
DecisionJudge judge = DecisionJudge.builder(decisionModel)
.score("helpfulness", ScoreQuestion.builder()
.text("How well does `assistant_answer` address `user_question`?")
.level("Terrible: irrelevant or off-topic")
.level("Mostly unhelpful: misses the main point")
.level("Mostly helpful: minor gaps remain")
.level("Excellent: fully and correctly addressed")
.build(), 2.0d)
.yesNo("is_plausible", YesNoQuestion.builder()
.text("Are the numeric values in `assistant_answer` physically plausible?")
.whenNo("At least one value is impossible")
.build(), 0.7d)
.build();
DecisionVerdict verdict = judge.judge(question, answer);
if (!verdict.passed()) {
System.out.println(verdict.feedback());
}
The input a judge builds¶
judge(String question, String answer) builds a two-field input:
When there is more to judge against, build a DecisionJudgeInput. Its fields become the
input under fixed names, the same wherever a judge is used, so write your questions against
them:
| Builder method | Input field | Shape |
|---|---|---|
question(String) |
user_question |
string |
answer(String) |
assistant_answer |
string |
expected(Object) |
expected_output |
a string, number, boolean, map or list: the reference to judge against |
context(String) / context(List<String>) |
supporting_context |
list, one entry per document |
toolCall(ToolCall) / toolCalls(List<ToolCall>) |
tool_calls |
list of {name, arguments, result} |
field(String, Object) |
the name you give | anything else |
DecisionVerdict verdict = judge.judge(DecisionJudgeInput.builder()
.question("Will I need an umbrella in Dublin tomorrow?")
.answer(answer)
.expected("Rain is likely; bring an umbrella.")
.field("search_required", true)
.toolCalls(toolCalls)
.build());
Empty context and tool-call lists are left out rather than sent as empty lists, so a
criterion can tell "no evidence" from "evidence that says nothing". The constants live on
DecisionJudgeInput (QUESTION_FIELD, EXPECTED_FIELD, TOOL_CALLS_FIELD, ...).
A value the model input cannot carry, such as an arbitrary object, is rejected when the input is built, with a message naming the field.
For anything that is not a question-and-answer pair, pass the input yourself. Any
DecisionInput works: fields, a plain text or a conversation.
DecisionVerdict verdict = judge.judge(DecisionInput.fields(Map.of(
"passage", retrievedText,
"claim", claimUnderTest)));
Code criteria¶
Some criteria need no model, because the answer is already in the input: whether the agent searched when it had to, whether the answer parses, whether it matches the expected output exactly. Declare those as checks:
DecisionJudge judge = DecisionJudge.builder(decisionModel)
.check("matches_search_expectation",
input -> !input.toolCalls().isEmpty()
== Boolean.TRUE.equals(input.field("search_required", Boolean.class)),
"searched when no search was required, or skipped a required search")
.yesNo("is_grounded", groundedQuestion, 0.7d)
.build();
A check is a Predicate<DecisionJudgeInput> plus the defect to report when it returns
false.
- It never reaches the model. Checks run locally before the call. Only the questions are sent, still in one request.
- It lands in the same verdict. A failed check is a
FAILEDfinding: it fails the verdict, its defect goes into the feedback and it shows insummary(). Itsanswerisnull, since there is no model answer. - A check that throws is a bug, not a verdict. The exception propagates out of
judge(...)before any request is made. - A judge needs at least one question. A judge of checks alone would be a plain predicate.
judge(DecisionInput) works with checks too, as long as the input is a fields input: they
read it through DecisionJudgeInput.of(fieldsInput) and its field(name, type) reader. A
text or conversation input with code checks is rejected with an IllegalArgumentException.
Do not ask a model a question code can answer
A question that a one-line comparison settles exactly is better as a check. It is free, it is never wrong, and it removes a source of variance from the verdict.
Builder Configuration¶
| Builder method | Default | Description |
|---|---|---|
options(DecisionOptions) |
the model's defaults | Options sent with every judging call, such as the model name. |
yesNo(String, YesNoQuestion, double) |
— | A criterion passing when the probability of yes is at or above the minimum. |
score(String, ScoreQuestion, double) |
— | A criterion passing when at least half its probability is on the levels at or above the minimum. |
choice(String, ChoiceQuestion, String...) |
— | A criterion passing when at least half its probability is on the accepted options. Validates that the options exist on the choice. |
check(String, Predicate<DecisionJudgeInput>, String) |
— | A code check: passes when the predicate returns true, otherwise reports the defect. |
criterion(DecisionCriterion) |
— | Add a pre-built criterion of either kind. |
minConfidence(double) |
0.6 |
How much of a choice's or score's probability must support its verdict; below it, INCONCLUSIVE. See undecided, not failed. Yes/no criteria are unaffected. |
failOnInconclusive(boolean) |
false |
Treat an undecided criterion as a failure. |
failOnError(boolean) |
false |
Treat a criterion the model refused to answer (ERROR) as a failure. |
failFast(boolean) |
false |
When a code check fails, skip the model call; the questions become NOT_APPLICABLE. |
feedbackRenderer(Function<List<DecisionFinding>, String>) |
DecisionJudge::defaultFeedback |
Replace the wording fed back to the model. |
Criterion names must be unique across both kinds, and a judge needs at least one question criterion. A score's minimum above the top of its rubric is rejected when the judge is built.
Criteria that only sometimes apply¶
A question can be made conditional with appliesWhen. When the predicate returns false
the question is not sent, and its finding is NOT_APPLICABLE: it neither passes nor fails.
A groundedness check on a turn with no retrieved context is the typical case:
DecisionJudge judge = DecisionJudge.builder(decisionModel)
.criterion(DecisionCriterion.yesNo("is_grounded", groundedQuestion, 0.7d)
.appliesWhen(input -> !input.context().isEmpty()))
.yesNo("is_relevant", relevantQuestion, 0.7d)
.build();
When no question applies, no call is made and verdict.response() is null.
Criteria that depend on other answers¶
Rubrics branch. An answer that asks which Springfield is meant should not be marked down for
leaving out the forecast's timing. whenChosen and whenPassed make a question depend on
another criterion's outcome:
DecisionJudge judge = DecisionJudge.builder(decisionModel)
.choice("mode", ChoiceQuestion.builder()
.text("How does `assistant_answer` respond to `user_question`?")
.option("answered", "Gives the weather")
.option("clarification_needed", "Asks which place is meant")
.build(), "answered", "clarification_needed")
.criterion(DecisionCriterion.yesNo("has_details", detailsQuestion, 0.7d)
.whenChosen("mode", "answered"))
.build();
- The branch costs nothing. Every question is answered in the one call, so
has_detailsis asked anyway, and the dependency is resolved in Java afterwards. Judge frameworks built on chat models walk such a graph one model call per node. - The branch follows the probability, not just the top option. It is taken when at
least half of the choice's probability, and at least
minConfidence, is on the optionswhenChosennames. A choice split 0.51 to 0.49 selects nothing. - When the choice lands elsewhere or doesn't settle, the dependent is
NOT_APPLICABLE, with a detail such ashas_details: does not apply, mode was "clarification_needed". It neither passes nor fails. - An undecided dependency (
INCONCLUSIVE,ERRORorNOT_APPLICABLE) makes its dependentsNOT_APPLICABLEtoo: nothing was selected for them to rely on. whenPassed("name")depends on a criterion having passed. That criterion can be a code check.- A dependency must be declared before its dependents, which rules out cycles. A
whenChosenoption must be one of the choice's options. Both are checked when the judge is built.
appliesWhen and a dependency combine. appliesWhen is judged on the input before the
call, and a false result means the question is not sent at all. The dependency is judged on
the findings after the call.
Reading the verdict¶
judge(...) returns one verdict carrying one finding per criterion, so the dimensions stay
separate rather than collapsing into a number:
classDiagram
direction TB
class DecisionVerdict {
<<record>>
+boolean passed
+List~DecisionFinding~ findings
+DecisionResponse response
+String feedback
+failures() List~DecisionFinding~
+inconclusive() List~DecisionFinding~
+errors() List~DecisionFinding~
+notApplicable() List~DecisionFinding~
+summary() String
}
class DecisionFinding {
<<record>>
+DecisionCriterion criterion
+DecisionAnswer answer
+Outcome outcome
+String detail
+name() String
+isFailure() boolean
}
class Outcome {
<<enumeration>>
PASSED
FAILED
INCONCLUSIVE
ERROR
NOT_APPLICABLE
}
class DecisionAnswer {
<<sealed interface>>
}
DecisionVerdict "1" *-- "1..*" DecisionFinding : one per criterion
DecisionFinding --> Outcome : how it landed
DecisionFinding --> DecisionAnswer : what came back
note for DecisionAnswer "permits YesNoAnswer, ChoiceAnswer,<br/>ScoreAnswer, RefusalAnswer"
response is the untouched DecisionResponse, so the usage counts and the response id are
there when you need them. It is null when no question was asked: every question was not
applicable, or failFast skipped the call.
summary() is a one-line view of how every criterion landed:
Each DecisionFinding carries its criterion, the raw answer, an Outcome, and a detail
sentence ready to hand back to a model. The answer is null for a code check, for a
criterion that was not asked, and for one set aside because its dependency was not met;
what the model said about the last kind is still in verdict.response(). Only FAILED
blocks the verdict:
| Outcome | Meaning | Blocks? |
|---|---|---|
PASSED |
the criterion was met | no |
FAILED |
the criterion was not met | yes |
INCONCLUSIVE |
too little probability supported the verdict | only with failOnInconclusive(true) |
ERROR |
the model refused to answer this criterion | only with failOnError(true) |
NOT_APPLICABLE |
not asked (appliesWhen was false, or failFast skipped the call), or asked but its dependency (whenChosen, whenPassed) was not met, so its answer is set aside |
no |
ERROR and NOT_APPLICABLE findings are left out of the feedback, since neither is a
defect the model can fix. failOnInconclusive(true) and failOnError(true) make a finding
block by reporting it as FAILED: it then appears in failures() and in the feedback, not
in inconclusive() or errors().
Feedback is synthesised, not generated¶
A decision model returns numbers, never prose, so the feedback is built from the answer and the rubric you wrote. It is deterministic and more specific than a judge model's commentary:
- helpfulness: rated "Mostly unhelpful: misses the main point" (1.20), needs to reach 2.00
which is "Mostly helpful: minor gaps remain"
- is_plausible: At least one value is impossible (scored 0.04, needs at least 0.70)
A failed yes/no criterion quotes the question's whenNo text, which is why it pays to write
it as a statement about the input.
Two details worth knowing. The level quoted is the one the score's value lands on, not the
most probable level: the two can disagree on a spread distribution, and quoting a label that
contradicts the number would be worse than useless. And the wording is formatted with
Locale.ROOT, so 0.70 does not become 0,70 on a comma-decimal JVM and change what the
model reads.
One case reads differently. When most of the probability sits below the required level but a share on the top levels lifts the score's value over it, quoting the value would contradict the verdict, so the feedback names what failed instead:
- helpfulness: 0.65 of the probability is below "Mostly unhelpful: misses the main point",
the level it needs to reach
Low confidence is undecided, not failed¶
A score or a choice is decided by where its probability lies.
A score passes when at least half of its probability sits on the levels at or above
minimum. A choice passes when at least half sits on the accepted options, even if the
single most likely option is another one.
Deciding from the probability, rather than from the score's value, is what makes a minimum mean "this level or better". An answer rated at level 2 with 90% of the probability, and at the level below with the other 10%, has a value of 1.9. It passes a minimum of 2 all the same:
A minimum is a level
The levels of a rubric are whole numbers, so a fractional minimum counts from the next
whole level: 1.2 asks for level 2 or better. Prefer whole numbers, which say what
they mean.
Whether that verdict can be acted on is not a question of how sure the provider says it is,
but of how much of the probability supports the verdict: the share on the passing side
when it passes, on the failing side when it fails. When that share is below minConfidence
(0.6 by default: a clear majority, where a coin flip is 0.5), the criterion is
INCONCLUSIVE, and it does not block unless you set failOnInconclusive(true).
The difference from confidence matters. This distribution over a four-level rubric has a confidence of only 0.51, because it is split between two levels, yet both levels pass a minimum of 2:
A split close to even is the undecided case. Here the score would pass, but barely more than half of the probability says so:
When the provider reports no probabilities, there is no split to read. The score's value must then be at least the minimum, the selected option must be an accepted one, and the answer's own confidence is used in place of the support. When the provider reports no confidence either, there is nothing to gate on, and the answer is acted on as given.
A criterion the model refuses to answer is reported as ERROR rather than thrown, with
the reason the provider gave. A refusal says nothing about the answer under judgement, so it
costs you that one criterion, not the whole verdict.
Testing with DecisionJudge¶
A judge is a useful test oracle for an LLM feature. It can check meaning where
contains(...) would be brittle, and it answers every criterion in one call. Asserting on
verdict.passed() alone, however, fails with Expecting value to be true but was false,
which hides which criterion failed. Pass the summary and the feedback as the assertion's
description:
DecisionVerdict verdict = judge.judge(question, answer);
assertThat(verdict.passed())
.as("%s%n%s", verdict.summary(), verdict.feedback())
.isTrue();
A failure then names the criterion, the score it reached and the score it needed:
[passed=false [helpfulness=PASSED, is_plausible=FAILED]
- is_plausible: At least one value is impossible (scored 0.04, needs at least 0.70)]
Expecting value to be true but was false
The %n puts the feedback on its own lines, under the summary.
Three more habits make these tests trustworthy:
- Build the test's judge with
failOnInconclusive(true)andfailOnError(true). By default a verdict passes when nothing failed, which suits a live self-refine loop. In a test it would let a criterion the model never decided pass silently. - Assert on the failing criteria when you expect a failure, not just on
passed():assertThat(verdict.failures()).extracting(DecisionFinding::name).containsExactly("is_plausible"). - Keep tests that call a real model out of an ordinary build. Each
judge(...)is one model call. Put them in*ITclasses behind a Maven profile, and guard them on the provider's key or server being available.
To test the judge's own wiring (your thresholds, dependencies and code checks) without a
model, script the answers with
StubDecisionModel.
For a criterion that sits close to its threshold, check how much it moves between runs with DecisionConsistency before you build a test on it.
See Also¶
- DecisionSelfRefineAdvisor: judge and retry inside a
ChatClient - DecisionEvaluator: the same judge behind Spring AI's
EvaluatorSPI - Confidence
- Decision Models