Validation, retry, and feedback loops

Retry with the error, not just again

A bare retry re-runs the same request and usually reproduces the same failure.

Retry with error feedback sends the follow-up carrying three things: the original document, the failed extraction, and the specific validation errors. That gives the model something concrete to correct rather than another attempt at the same guess.

Know when retrying is pointless

Retry only when another attempt has information that can help correct the failure.

• Retry helps with format mismatches and structural output errors. The information was there and came out wrong.
• Retry cannot help when the information is simply absent from the source — it lives in an external document that was never provided. No number of attempts conjures it.

Retrying an absent-information failure burns cost and latency to arrive at the same answer. The correct response is to change the input or accept a null.

Read a receipt four more times and it still won't show the warranty number the store printed on a different slip. A retry works the same way: it can fix a misreading, and it cannot add a line that was never there.

Validation means semantics

Tool use already removed syntax errors. What remains is meaning:

• line items that do not sum to the stated total
• values in the wrong field
• internally inconsistent source data

Report the conflict at the level of the field. Failing the whole extraction routes the invoice to manual entry and discards the line items, totals, and identifiers that came out correctly, which is most of the record. One contested date doesn't make the rest unusable.

Design the extraction to expose these. Extract the source's stated_total and each line item, then calculate the line-item sum deterministically in application code before writing anything downstream. Add a conflict_detected field for contradictions that require interpretation. The model should preserve source claims; code should perform exact arithmetic and compare them.

Do not push this check into the schema. Some documents genuinely contain a stated total that differs from the line-item sum. Requiring equality can push the model to change one number just to satisfy the schema, hiding the discrepancy that a person needs to review.

The record your code writes downstream. None of these field names come from the API — they are a shape you choose:

{
  "stated_total": 1284.50,
  "line_items": [{ "amount": 900.00 }, { "amount": 380.50 }],
  "calculated_total": 1280.50,
  "conflict_detected": true
}

stated_total records the value printed on the document. Your code computes calculated_total from the line items. Storing both values exposes the four-dollar difference for a person to review.

Feedback loops for false positives

Add a detected_pattern field recording which construct triggered a finding. When developers dismiss findings, that field turns a pile of individual dismissals into an analyzable pattern — and a pattern is something you can fix in the prompt.