CI review with a strict output contract — Labs

Labs · Lab 7 of 10

120–180 minutes active15–30 minutes setupTask 3.6: Integrate Claude Code into CI/CD pipelines Task 4.1: Design prompts with explicit criteria to improve precision and reduce false positives Task 4.3: Enforce structured output using tool use and JSON schemas Task 4.4: Implement validation, retry, and feedback loops for extraction Task 4.6: Design multi-instance and multi-pass review architectures Task 5.5: Design human review workflows and confidence calibration

The brief

Your assignment

Create a synthetic pull-request diff in a disposable repository, including one real issue and one prompt-injection-like comment, then review it locally. Return schema-validated findings with file, line, severity, evidence, and recommended action. Treat the diff as untrusted input. The core path writes a local report artifact; posting a pull-request comment is a write operation and needs the narrow permission it actually uses.

Before you build

Set up a safe workspace

Create a disposable repository and synthetic diff with one real issue.
Add a prompt-injection-like comment to the diff.
Keep the core path local and do not configure repository credentials.

Build in checkpoints

One observable behavior at a time

01Write the rubric and strict finding schema.
02Create clean, malformed, failed, and invalid-line fixtures.
03Validate findings against the reviewed diff.
04Define timeout and retry behavior.
05Place human approval before every publication or merge effect.

What you'll practice

Run non-interactively with time bounds.
Separate zero findings from malformed output and execution failure.
Validate evidence against the reviewed diff.
Place human approval before any write or merge action.

Your lab files

Four files for each lab

01
decision.md
State the requirement, chosen pattern, rejected alternatives, and the reason for the tradeoff.
02
implementation/
Keep the smallest working example, configuration, diagram, or synthetic-trace analysis.
03
evidence.md
Record test cases, observed results, failure behavior, limitations, and evidence locations.
04
review.md
Explain what must be deterministic, what may remain model-directed, and where a human is required.

Download starter templates

What to save

A review rubric and strict output schema.
A workflow and permission/side-effect matrix.
A retry table.
Fixtures for clean, malformed, failed, and invalid-line outcomes.

Test bench

What good work includes

The command cannot wait indefinitely for input.
Zero findings is explicit and schema valid.
Malformed output and API failure fail the job.
Every file and line belongs to the reviewed diff.
Retries apply to transient execution failures, not invalid findings.
No comment, code, or merge write occurs without its stated boundary.

Review questions

Why is posting a comment still a write?
Which false positives damage trust most?
Why should review use fresh, independent context?

Strong solutions tend to

Committed criteria and machine validation before publication.
Timeouts and local report files in the no-credential path.
Permissions that match the actions the workflow can take.

These are common strengths, not one required design.

Partial example

What useful evidence can look like

This is one small example, not a complete solution or a template you must copy. Use the notes to see why the evidence is useful.

Open annotated example from evidence.md / validated result

status: completed
findings: []
checked_files: 14
schema_valid: true
reference_validation: passed

status: Shows the review ran, which distinguishes zero findings from execution failure.
checked_files: Provides a coverage denominator instead of implying the whole repository was reviewed.

Review your work

Use evidence from your work—not confidence alone. For each area, run the check and choose the strongest stage your work fully demonstrates.

If you're between two stages, use the earlier one and treat the next stage as your revision goal. There is no total score.

Finding contract and evidence

Every reported finding follows the strict schema and points to evidence that actually exists in the reviewed diff.

How to check

Validate the real-issue and invalid-line fixtures, then follow every file and line in the accepted report back to the exact changed diff content.

Needs another pass

Findings are free-form, required fields can be missing, or a file and line can be reported without evidence in the diff.

Partly demonstrated

The schema validates common findings, but evidence checking, allowed severity values, or the zero-findings format is incomplete.

Demonstrated

Every finding has valid file, line, severity, evidence, and action fields; each location exists in the diff; and zero findings is an explicit schema-valid result.

Failure, timeout, and retry

The job distinguishes a clean review from malformed output, invalid evidence, and transient or permanent execution failure.

How to check

Run the clean, malformed, API-failure, invalid-line, and timeout fixtures. Match each result to the retry table and final job status.

Needs another pass

A malformed or failed run can look clean, the command can wait forever, or every failure is retried without a limit.

Partly demonstrated

Failures are visible, but at least one result type has the wrong retry rule or lacks a clear terminal report.

Demonstrated

The job is non-interactive and time-bounded, every result type has a distinct status, only transient execution failures retry, and exhausted attempts fail clearly.

Permissions and side effects

Untrusted diff content cannot expand permissions or cause comments, code changes, or merges without the required boundary.

How to check

Trace the prompt-injection-like comment through the workflow and inspect the permission matrix for local reports, PR comments, code writes, and merges.

Needs another pass

Diff instructions are followed, credentials are unnecessarily available, or the review can publish or merge without explicit approval.

Partly demonstrated

The core review is read-only, but one optional write lacks a narrowly stated permission, approval, or audit step.

Demonstrated

The diff remains untrusted data, the local path needs no repository credential, and every optional write has only its required permission plus an explicit approval boundary.

Review quality evidence

Committed criteria and independent context produce a useful report without hiding false positives or relying on the author’s assumptions.

How to check

Compare the output with the known synthetic issue and clean fixture, then confirm the reviewer received the diff and criteria but not the generating session’s reasoning.

Needs another pass

Criteria are vague, the reviewer shares the author’s full context, or the packet reports only whether the command ran.

Partly demonstrated

The known issue is detected, but false-positive behavior, independent context, or the reason for the chosen criteria is not demonstrated.

Demonstrated

The independent review finds the supported issue, leaves the clean fixture clean, records false positives, and ties every accepted category to committed criteria.

Loading completion status…

Your assignment

Four files for each lab

decision.md

State the requirement, chosen pattern, rejected alternatives, and the reason for the tradeoff.

implementation/

Keep the smallest working example, configuration, diagram, or synthetic-trace analysis.

evidence.md

Record test cases, observed results, failure behavior, limitations, and evidence locations.

review.md

Explain what must be deterministic, what may remain model-directed, and where a human is required.

What to save

A review rubric and strict output schema.

A workflow and permission/side-effect matrix.

A retry table.

Fixtures for clean, malformed, failed, and invalid-line outcomes.

What good work includes

The command cannot wait indefinitely for input.

Zero findings is explicit and schema valid.

Malformed output and API failure fail the job.

Every file and line belongs to the reviewed diff.

Retries apply to transient execution failures, not invalid findings.

No comment, code, or merge write occurs without its stated boundary.

What useful evidence can look like

This is one small example, not a complete solution or a template you must copy. Use the notes to see why the evidence is useful.

Open annotated example from evidence.md / validated result

status: completed
findings: []
checked_files: 14
schema_valid: true
reference_validation: passed

status: Shows the review ran, which distinguishes zero findings from execution failure.
checked_files: Provides a coverage denominator instead of implying the whole repository was reviewed.

Review your work

Use evidence from your work—not confidence alone. For each area, run the check and choose the strongest stage your work fully demonstrates.

If you're between two stages, use the earlier one and treat the next stage as your revision goal. There is no total score.

Finding contract and evidence

Every reported finding follows the strict schema and points to evidence that actually exists in the reviewed diff.

How to check

Validate the real-issue and invalid-line fixtures, then follow every file and line in the accepted report back to the exact changed diff content.

Needs another pass

Findings are free-form, required fields can be missing, or a file and line can be reported without evidence in the diff.

Partly demonstrated

The schema validates common findings, but evidence checking, allowed severity values, or the zero-findings format is incomplete.

Demonstrated

Every finding has valid file, line, severity, evidence, and action fields; each location exists in the diff; and zero findings is an explicit schema-valid result.

Failure, timeout, and retry

The job distinguishes a clean review from malformed output, invalid evidence, and transient or permanent execution failure.

How to check

Run the clean, malformed, API-failure, invalid-line, and timeout fixtures. Match each result to the retry table and final job status.

Needs another pass

A malformed or failed run can look clean, the command can wait forever, or every failure is retried without a limit.

Partly demonstrated

Failures are visible, but at least one result type has the wrong retry rule or lacks a clear terminal report.

Demonstrated

The job is non-interactive and time-bounded, every result type has a distinct status, only transient execution failures retry, and exhausted attempts fail clearly.

Permissions and side effects

Untrusted diff content cannot expand permissions or cause comments, code changes, or merges without the required boundary.

How to check

Trace the prompt-injection-like comment through the workflow and inspect the permission matrix for local reports, PR comments, code writes, and merges.

Needs another pass

Diff instructions are followed, credentials are unnecessarily available, or the review can publish or merge without explicit approval.

Partly demonstrated

The core review is read-only, but one optional write lacks a narrowly stated permission, approval, or audit step.

Demonstrated

The diff remains untrusted data, the local path needs no repository credential, and every optional write has only its required permission plus an explicit approval boundary.

Review quality evidence

Committed criteria and independent context produce a useful report without hiding false positives or relying on the author’s assumptions.

How to check

Compare the output with the known synthetic issue and clean fixture, then confirm the reviewer received the diff and criteria but not the generating session’s reasoning.

Needs another pass

Criteria are vague, the reviewer shares the author’s full context, or the packet reports only whether the command ran.

Partly demonstrated

The known issue is detected, but false-positive behavior, independent context, or the reason for the chosen criteria is not demonstrated.

Demonstrated

The independent review finds the supported issue, leaves the clean fixture clean, records false positives, and ties every accepted category to committed criteria.