Skip to content

JevEvaluator

A JevJudge behind Spring AI's Evaluator interface, so Jev drops into code already written against the framework's evaluation SPI.

Why

Spring AI's org.springframework.ai.evaluation.Evaluator is a one-method interface:

EvaluationResponse evaluate(EvaluationRequest request);

Its two shipped implementations — RelevancyEvaluator and FactCheckingEvaluator — prompt a chat model and parse a verdict out of the prose. JevEvaluator implements the same interface with typed questions instead, so anything holding an Evaluator gets a stricter one without changing a line.

Quick Start

Evaluator evaluator = new JevEvaluator(judge);

EvaluationResponse result = evaluator.evaluate(
        new EvaluationRequest(userQuestion, retrievedDocuments, assistantAnswer));

if (!result.isPass()) {
    log.warn("{} (score {})", result.getFeedback(), result.getScore());
}

The mapping

EvaluationRequest Judged state field
getUserText() user_question
getResponseContent() assistant_answer
getDataList() supporting_context (only when non-empty)
EvaluationResponse From
isPass() verdict.passed()
getScore() the fraction of criteria that passed
getFeedback() verdict.feedback()
getMetadata() the full JevVerdict and a per-criterion outcome map 
JevVerdict verdict = (JevVerdict) result.getMetadata().get(JevEvaluator.VERDICT_METADATA_KEY);
Map<String, String> findings =
        (Map<String, String>) result.getMetadata().get(JevEvaluator.FINDINGS_METADATA_KEY);
// {helpfulness=PASSED, is_plausible=FAILED}

The single score is lossy on purpose

EvaluationResponse has room for one float, while a judge holds one threshold per criterion. The score is the fraction of criteria that passed, and the per-criterion detail is preserved in the metadata. If that view is what you are after, call JevJudge.judge directly and read the JevVerdict.

An inconclusive criterion counts as neither passed nor failed, so it lowers the score without being treated as a failure — the same stance JevVerdict.passed() takes.

Supporting documents

getDataList() carries the retrieved Documents, which is the evidence a groundedness criterion needs. They reach the state as supporting_context, so write such a criterion against that field:

JevJudge grounded = JevJudge.builder(typeSafeClient)
    .noul("is_grounded", Noul.builder()
        .instructions("Is every claim in `assistant_answer` supported by `supporting_context`?")
        .whenFalse("Introduces facts the context does not support")
        .build(), 0.7d)
    .build();

This is also how a groundedness check gets real evidence in a RAG setting, which the self-refine advisor cannot do when tool execution is internal.

See Also