The first document summarizer I put in front of a real team returned beautiful prose and completely unusable data. The model would write a fluent paragraph, then occasionally wrap it in markdown fences, occasionally omit a required field, and occasionally invent a confidence score that was not in the schema. Downstream code that expected clean JSON kept throwing parse errors. After the third failure I stopped asking the model to “please return JSON” and started treating structured output as an engineering problem with explicit validation, clear failure modes, and a small retry path.
This post builds the smallest useful version of that solution: a document summarizer that returns a validated JSON object with a fixed shape. We will define the schema, call the model, validate the result, and handle the cases where the output is still wrong. No agent loop, no tools, no external storage—just a reliable extraction step you can drop into a larger pipeline later.
As always:
Build the smallest useful version, then test what breaks.
What You Will Have Working by the End
A fixed JSON schema for a document summary
A single request that asks the model for that shape
Validation that rejects missing fields, wrong types, or extra keys
A short retry path when validation fails
Three realistic failure modes and how to notice them early
The output will be something you can hand to another service without additional cleaning.
The Target Schema
We will extract four pieces of information from a short document:
JSON
{
"title": "string",
"one_sentence_summary": "string",
"key_points": ["string"],
"sentiment": "positive" | "neutral" | "negative"
}In Python we can express the same shape with a simple dictionary schema or with Pydantic. For clarity we will start with a plain JSON Schema and a manual validator; later posts can introduce stricter libraries.
Python
SUMMARY_SCHEMA = {
"type": "object",
"properties": {
"title": {"type": "string"},
"one_sentence_summary": {"type": "string"},
"key_points": {
"type": "array",
"items": {"type": "string"},
"minItems": 1,
"maxItems": 5
},
"sentiment": {
"type": "string",
"enum": ["positive", "neutral", "negative"]
}
},
"required": ["title", "one_sentence_summary", "key_points", "sentiment"],
"additionalProperties": False
}The additionalProperties: False rule is deliberate. Models sometimes add helpful extra fields; those extras break downstream consumers that expect an exact shape.
The Smallest Useful Request
We will use the Messages API and ask for JSON directly. Modern Anthropic models support structured outputs via output_config (or the SDK’s messages.parse helper). For teaching purposes we will show an explicit approach that still works even if you are on an older SDK version, then note the newer helpers.
Python
import json
import os
from dotenv import load_dotenv
from anthropic import Anthropic
load_dotenv()
client = Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
SYSTEM = """You are a precise document summarizer.
Return ONLY a valid JSON object that matches this schema:
{
"title": string,
"one_sentence_summary": string,
"key_points": array of 1-5 strings,
"sentiment": "positive" | "neutral" | "negative"
}
Do not wrap the JSON in markdown fences.
Do not add any keys that are not in the schema.
Do not include explanatory text before or after the JSON.
"""
def summarize_document(text: str) -> dict:
response = client.messages.create(
model="claude-sonnet-5", # verify current ID in official docs
max_tokens=512,
system=SYSTEM,
messages=[
{
"role": "user",
"content": f"Summarize the following document:\n\n{text}"
}
]
)
raw = response.content[0].text.strip()
return rawAt this point we have a string that is supposed to be JSON. We have not yet validated it.
Validation: The Part Most Tutorials Skip
Python
def validate_summary(data: dict) -> list[str]:
"""Return a list of error messages. Empty list means valid."""
errors = []
if not isinstance(data, dict):
return ["Root value is not an object"]
required = ["title", "one_sentence_summary", "key_points", "sentiment"]
for key in required:
if key not in data:
errors.append(f"Missing required key: {key}")
if "title" in data and not isinstance(data["title"], str):
errors.append("title must be a string")
if "one_sentence_summary" in data and not isinstance(data["one_sentence_summary"], str):
errors.append("one_sentence_summary must be a string")
if "key_points" in data:
if not isinstance(data["key_points"], list):
errors.append("key_points must be an array")
else:
if not (1 <= len(data["key_points"]) <= 5):
errors.append("key_points must contain 1-5 items")
for i, item in enumerate(data["key_points"]):
if not isinstance(item, str):
errors.append(f"key_points[{i}] must be a string")
if "sentiment" in data and data["sentiment"] not in ("positive", "neutral", "negative"):
errors.append("sentiment must be one of: positive, neutral, negative")
extra = set(data.keys()) - set(required)
if extra:
errors.append(f"Unexpected keys: {sorted(extra)}")
return errorsA production system would use a real JSON Schema validator or a Pydantic model. The manual version above makes every rule visible, which is useful while you are still learning the failure modes.
Parsing and Retrying
Python
def parse_and_validate(raw: str) -> tuple[dict | None, list[str]]:
try:
data = json.loads(raw)
except json.JSONDecodeError as e:
return None, [f"JSON parse error: {e}"]
errors = validate_summary(data)
if errors:
return None, errors
return data, []
def summarize_with_retry(text: str, max_attempts: int = 2) -> dict:
last_errors = []
current_text = text
for attempt in range(1, max_attempts + 1):
print(f"Attempt {attempt}")
raw = summarize_document(current_text)
print(f"Raw output: {raw[:200]}...")
data, errors = parse_and_validate(raw)
if data is not None:
return data
last_errors = errors
print(f"Validation failed: {errors}")
# Feed the errors back so the model can correct itself
current_text = (
f"The previous summary was invalid for these reasons:\n"
f"{json.dumps(errors, indent=2)}\n\n"
f"Please try again and return ONLY valid JSON.\n\n"
f"Original document:\n{text}"
)
raise ValueError(f"Failed after {max_attempts} attempts. Last errors: {last_errors}")Two attempts are usually enough for well-formed schemas. More than that often indicates the schema itself is ambiguous or the document is too noisy.
A Complete Runnable Example
Python
SAMPLE_DOC = """
Acme Corp announced record quarterly revenue of $42 million,
up 18% from the previous year. CEO Jane Rivera credited strong
demand in the Midwest and a successful product launch in March.
The company also noted ongoing supply-chain pressure that may
affect margins next quarter. Overall the tone of the release
was optimistic.
"""
if __name__ == "__main__":
result = summarize_with_retry(SAMPLE_DOC)
print("\n=== Validated Summary ===")
print(json.dumps(result, indent=2))When it works you receive a clean dictionary you can pass to any other Python code without further cleaning.
Three Failure Modes Worth Testing
1. Markdown Fences and Extra Prose
Even with a strict system prompt the model sometimes returns:
text
Here is the summary:
```json
{ ... }
```The json.loads call fails. The retry path that feeds the parse error back to the model usually corrects it on the second attempt. Logging the raw output makes the problem obvious.
2. Missing or Extra Keys
The model omits sentiment or adds a helpful confidence field. The validator catches both cases. Feeding the exact error list back in the retry message is more effective than a generic “please follow the schema.”
3. Wrong Types Inside Arrays
key_points comes back as a list of objects instead of a list of strings, or as a single string. The manual validator surfaces the precise index that failed. That precision helps the model correct the mistake.
I keep a small folder of “ugly documents” that historically trigger these three failures. Running the summarizer against that folder is part of every change to the schema or the prompt.
Practical Rules I Use for Structured Extraction
Define the schema before you write the prompt. The schema is the contract; the prompt is only a way to ask for it.
Reject extra keys. Helpful additional fields become breaking changes for downstream code.
Keep the schema small. Large, deeply nested schemas increase the chance of partial or malformed output.
Always log the raw model output when validation fails. The raw text is the only way to see what actually went wrong.
Prefer a short retry with explicit error feedback over a long, complicated prompt that tries to prevent every possible mistake in advance.
Modern SDK Helpers (Worth Knowing)
Newer versions of the Anthropic Python SDK offer client.messages.parse and output_config / output_format options that can constrain decoding more tightly and integrate with Pydantic models. Those helpers reduce the amount of manual validation you need to write. They do not eliminate the need to think about failure modes, retries, and logging. Use the helpers when they are available; keep the validation mindset regardless.
What This Summarizer Does Not Do Yet
It does not:
Handle documents longer than the context window
Split large documents into chunks and merge results
Attach source citations or page numbers
Run as a background job with retries and dead-letter queues
Store results in a database or object store
Those capabilities belong to later production posts. The goal today is a single, reliable extraction step that returns valid JSON or fails loudly with clear reasons.
Verification Checklist
Run the sample document and confirm you receive a dictionary that matches the schema.
Temporarily break the system prompt so the model returns markdown fences; confirm the retry path recovers or raises a clear error.
Change the schema to require an extra field and confirm validation fails with a precise message.
Feed a document that is pure noise and confirm the system either returns valid (if limited) JSON or fails after the retry limit.
Inspect the logs and confirm every validation failure includes the raw model output.
If all five steps pass, you have a working structured summarizer.
Update Note
Anthropic’s structured-output features continue to evolve. Model identifiers, the exact parameter names (output_config, output_format, etc.), and the degree of decoding constraints change. Always verify the current recommended approach and model ID in the official documentation before you copy examples into longer-lived code. The validation-and-retry pattern shown here remains useful even when the platform offers stronger guarantees.
Looking Ahead
The next post builds a human-reviewed support-ticket triage workflow on top of the same structured-output foundation. Once you can reliably extract a fixed JSON shape, it becomes straightforward to route tickets, assign priorities, and keep a person in the loop for the cases that still need judgment.
Keep the summarizer. When you later embed it inside a larger pipeline, you will already know how to detect and recover from the ways structured output still breaks.
Build the smallest useful version, then test what breaks.
— Caleb Fletcher
Boise, Idaho
No feedback yet — submit the first.