Lab assignment
Structured extraction and batch evaluation
Compare extraction strategies against labeled synthetic data without hiding critical-field failures.
The brief
Your assignment
Generate 30 synthetic club-reimbursement documents locally from a deterministic template and label the answer set before evaluation. Extract a small schema, comparing one pass, deterministic field validation, and an independent review pass. Then compare synchronous processing with a message batch. Batch completion may take longer than the active work time.
Keep the generator simple—vary layout, missing fields, and document type from a recorded seed. Use a held-out evaluation set; examples must not leak its answers.
Before you build
Set up a safe workspace
- Write a deterministic generator and record its seed.
- Define the extraction schema before creating examples.
- Reserve a held-out labeled set that examples cannot reveal.
Build in checkpoints
One observable behavior at a time
- 01Generate and label 30 synthetic documents.
- 02Run one-pass extraction and structural validation.
- 03Add deterministic field checks and an independent review pass.
- 04Create a batch manifest with stable
custom_idvalues. - 05Report per-field and per-document-type results.
What you'll practice
- Define criteria before examples or evaluation.
- Separate structural validation from semantic accuracy.
- Correlate every batch item with a stable identifier and status.
- Justify human review using labeled outcomes rather than model confidence alone.
Your lab files
Four files for each lab
- 01
decision.mdState the requirement, chosen pattern, rejected alternatives, and the reason for the tradeoff.
- 02
implementation/Keep the smallest working example, configuration, diagram, or synthetic-trace analysis.
- 03
evidence.mdRecord test cases, observed results, failure behavior, limitations, and evidence locations.
- 04
review.mdExplain what must be deterministic, what may remain model-directed, and where a human is required.
What to save
- A dataset card, generator seed, and gold labels.
- A schema and versioned prompts/examples.
- Validation code and a batch manifest.
- A per-field, per-document-type report and error analysis.
Test bench
What good work includes
- Absent fields may remain null and every structural result validates.
- Semantic inconsistencies remain detectable.
- Every request and result has a stable
custom_id. - Partial failures are independently traceable and selectively retriable.
- Critical fields are reported separately.
- Review thresholds are evaluated against labels.
Review questions
- What does aggregate accuracy hide?
- Which failures can retry fix?
- When is batch latency acceptable?
- How does independent review change cost and correlated failure?
Strong solutions tend to
- Synthetic reproducible data and held-out evaluation.
- Explicit denominators and versioned prompts/schemas.
- Evidence-based human-review thresholds.
Partial example
What useful evidence can look like
This is one small example, not a complete solution or a template you must copy. Use the notes to see why the evidence is useful.
Open annotated example from evidence.md / evaluation slice
document_type: handwritten-receipt
field: total
correct: 6 / 10
review_triggered: 4 / 10
review_fixed: 3 / 4- 6 / 10
- Keeps the denominator visible and avoids hiding a weak document type inside one overall score.
- review_fixed
- Measures whether the proposed human-review rule actually catches useful cases.
Review your work
Use evidence from your work—not confidence alone. For each area, run the check and choose the strongest stage your work fully demonstrates.
Dataset and evaluation split
The synthetic documents, gold labels, and held-out set are reproducible and do not leak evaluation answers into examples.
Regenerate the documents from the recorded seed, compare them with the dataset card, and verify that no prompt example contains a held-out document or answer.
Needs another pass
Labels are created after seeing model output, the generator cannot be reproduced, or evaluation answers appear in examples.
Partly demonstrated
A seed and labels exist, but the split, document-type coverage, or separation between examples and held-out cases is unclear.
Demonstrated
The recorded seed reproduces all documents, gold labels were fixed before evaluation, document variations are documented, and the held-out set is isolated from prompt examples.
Structural and semantic checks
Schema validation checks output format while separate code checks whether extracted values make sense together.
Run missing-field, malformed-output, and inconsistent-value fixtures. Identify which schema rule or semantic check catches each problem.
Needs another pass
Schema-valid output is treated as automatically correct, missing values are invented, or inconsistent amounts and dates pass unnoticed.
Partly demonstrated
The schema and some value checks work, but null handling, cross-field rules, or failure reporting is incomplete.
Demonstrated
Every output matches the schema, absent values remain null, deterministic checks expose cross-field inconsistencies, and source claims are preserved rather than silently corrected.
Batch identity and recovery
Every batch result maps to its original request and partial failures can be retried without repeating successful work.
Shuffle returned results, fail one item, and use custom_id values plus the manifest to identify and retry only the failed request.
Needs another pass
Results are matched by position, failed items cannot be distinguished, or a partial failure forces the entire batch to run again.
Partly demonstrated
custom_id values exist, but status handling, manifest records, or selective retry evidence is incomplete.
Demonstrated
Stable custom_id values connect every request and result regardless of order, each status is traceable, and transient failed items can be retried independently.
Measured quality and review
Results expose weak fields and document types, and human-review rules are tested against labels rather than chosen from model confidence alone.
Recalculate one per-field and one per-document-type result from the raw labels, then compare outcomes with and without each proposed review threshold.
Needs another pass
Only one overall accuracy is reported, denominators are hidden, or review is routed solely by untested confidence.
Partly demonstrated
Some slices are reported, but critical fields, denominators, threshold impact, or error examples are missing.
Demonstrated
The report shows explicit denominators, critical fields, document-type slices, and errors; review thresholds are evaluated against labeled outcomes, impact, and available capacity.