Stop Begging Your LLM for Valid JSON: Self-Correcting Structured Output in Spring AI 2.0

Stop Begging Your LLM for Valid JSON: Self-Correcting Structured Output in Spring AI 2.0



Tópico: Stop Begging Your LLM for Valid JSON: Self-Correcting Structured Output in Spring AI 2.0
Categoria: Tutoriais | Programação & Tecnologia
Idioma Principal: Português (Conteúdo de Tecnologia)

Descrição do Conteúdo / Informações:
-------------------------------------------------------------------------
Every developer who has worked with LLMs has been there. You ask the model for JSON. You describe the schema. You say "please only respond with valid JSON." And sometimes, it still breaks.

Your application crashes because the model returned a string where you expected an integer. Or it wrapped the JSON in markdown code blocks. Or it omitted a required field.

Spring AI 2.0 has a solution that treats this like a real engineering problem instead of a prayer.



The Problem


When you use structured output in Spring AI, the workflow goes like this:

• You define a Java type (a record, class, or enum)

• Spring AI generates a JSON schema from that type

• The schema gets appended to the prompt sent to the LLM

• The model returns a response

• Spring AI attempts to deserialize the response into your type

This works well with frontier models like Claude and GPT-4. But smaller open-source models, like Llama 3.2 1B running locally via Ollama, fail more often. They might return null for a primitive field, omit required fields, or produce malformed JSON.

When it fails, you get a deserialization exception. Your endpoint returns a 500 error. Spring AI provides no built-in recovery mechanism.



The Old Approach: Hope


Consider a conference talk submission system. Speakers submit messy, unstructured abstracts. You want to extract structured data:

public record TalkSubmission(
String title,
String abstractText,
Level level,        // BEGINNER, INTERMEDIATE, ADVANCED
Track track,
int duration,
List<String> tags,
String speakerHandle
) {}

Here is what the basic typed response looks like:

@PostMapping("/typed")
public TalkSubmission typed(@RequestBody String rawSubmission) {
return chatClient.prompt()
.system(systemPrompt)
.user(spec -> spec.text("Extract the talk submission: {submission}")
.param("submission", rawSubmission))
.call()
.entity(TalkSubmission.class);
}

You define your type. Spring AI generates the schema and appends it to the prompt. The model gets the instruction. And you hope it works.

Dan Vega, Spring Developer Advocate at Broadcom, puts it bluntly in his video demonstration: "That's not engineering. That's hoping."



The New Solution: validateSchema()


Spring AI 2.0 introduces self-correcting schema validation. When the model returns invalid JSON, Spring AI validates the response against the schema, detects the error, feeds the error message back to the model, and asks it to fix the response.

The change is a single method call:

@PostMapping("/validated")
public TalkSubmission validated(@RequestBody String rawSubmission) {
return chatClient.prompt()
.system(systemPrompt)
.user(spec -> spec.text("Extract the talk submission: {submission}")
.param("submission", rawSubmission))
.call()
.entity(TalkSubmission.class, spec -> spec.validateSchema());
}

That is it. The spec -> spec.validateSchema() consumer turns on the self-correcting retry loop.



How It Actually Works


According to the official documentation by Christian Tzolov (Spring AI team), the validation loop works as follows:

• The model responds

• Spring AI validates the response against the generated schema

• If validation passes, you get your typed record back

• If validation fails, the specific validation error (e.g., "expected int, got null for field duration") is appended to the user prompt and the call is re-issued

• The model sees the exact error on each retry, not a blind re-try

This is powered by StructuredOutputValidationAdvisor, a recursive advisor that is auto-registered when you call validateSchema(). Default is 3 retry attempts. The model knows exactly what went wrong and can correct it on the next attempt.

To customize the retry count, build your own advisor instance:

var validationAdvisor = StructuredOutputValidationAdvisor.builder()
.outputType(TalkSubmission.class)
.maxRepeatAttempts(5)
.build();

ChatClient chatClient = ChatClient.builder(chatModel)
.defaultAdvisors(validationAdvisor)
.build();



Provider-Native Structured Output


Some frontier models support structured output at the API level. Instead of appending the schema to the prompt text, the schema is sent as an API constraint. The provider's runtime enforces conformance, meaning invalid responses cannot be emitted at all.

Spring AI 2.0 exposes this through useProviderStructuredOutput():

TalkSubmission result = chatClient.prompt()
.system(systemPrompt)
.user(spec -> spec.text("Extract the talk submission: {submission}")
.param("submission", rawSubmission))
.call()
.entity(TalkSubmission.class, spec -> spec
.useProviderStructuredOutput()
.validateSchema());

Supported providers as of Spring AI 2.0:

•
OpenAI: GPT-4o and later models with JSON Schema support

•
Anthropic: Claude 3.5 Sonnet and later models

•
Google GenAI: Gemini 1.5 Pro and later models

•
Mistral AI: Mistral Small and later models with JSON Schema support

•
Ollama: Models with JSON Schema support (model-specific)

Native structured output is off by default because support varies across models. If a model does not support it, the flag is silently ignored and the prompt-based approach is used instead.

Note: .entity() is only available on .call(), not on .stream(). Typed parsing requires the complete response, so streaming responses cannot be deserialized into a typed object.



Known Limitations


OpenAI does not accept top-level JSON arrays. If you need a List<T>, wrap it in a container record first:

// Does NOT work with OpenAI native structured output:
List<TalkSubmission> list = chatClient.prompt()
.call()
.entity(new ParameterizedTypeReference<List<TalkSubmission>>() {},
spec -> spec.useProviderStructuredOutput()); // fails

// Works: wrap in a container
record SubmissionList(List<TalkSubmission> submissions) {}
SubmissionList result = chatClient.prompt()
.call()
.entity(SubmissionList.class, spec -> spec.useProviderStructuredOutput());

Ollama with reasoning models (like Qwen variants) may emit internal reasoning traces as plain text instead of JSON. Use a non-reasoning model, or combine with validateSchema() so malformed responses are automatically retried.



When To Use What


Not every scenario needs all features enabled. Based on the official docs and video demonstration:

Frontier models (Claude, GPT-4, Gemini):

•
useProviderStructuredOutput() for API-level enforcement

•
validateSchema() as a safety net for edge cases

• These models rarely fail, but the combination gives you two layers of protection

Open-source models (Llama, Mistral via Ollama):

•
useProviderStructuredOutput() may have no effect (model-specific)

•
validateSchema() is essential

• These models fail more often, especially with complex schemas or small parameter counts

Production systems:

• Always enable validation. The overhead is minimal compared to a 500 error

• Log validation failures to identify which prompts or models need improvement

• Customize maxRepeatAttempts based on your latency budget



The Bigger Picture


This feature represents a shift in how we think about LLM integration. For too long, the industry treated unreliable model output as a prompt engineering problem. Write a better prompt. Be more specific. Add examples. Pray harder.

Spring AI 2.0 treats it as a systems problem. Validate. Retry. Self-correct. The same principles we apply to any unreliable external service: network calls, database queries, third-party APIs. LLMs are no different.

If you are building production applications with LLMs, schema validation is not optional. It is basic engineering.

Sources:

• Self-Correcting Structured Output in Spring AI 2.0 - Dan Vega (YouTube)

• Self-Correcting Structured Output in Spring AI 2.0 - Christian Tzolov (Spring Blog)

• Spring AI Structured Output Reference Docs