Lab assignment
Reliability, provenance, and escalation evaluation
Evaluate autonomy, escalation, failure propagation, and claim-level provenance across a synthetic scenario suite.
The brief
Your assignment
Create a fictional support or research evaluation suite. Include autonomous resolution, ambiguity, an explicit human request, a policy exception, a timeout, a partial agent failure, conflicting sources, insufficient evidence, and high-impact review. Do not use real customer conversations, personal data, or confidential company material.
Before you build
Set up a safe workspace
- Choose a fictional support or research setting.
- Define the outcome labels and metric denominators before generating cases.
- Create only synthetic cases with no personal or confidential data.
Build in checkpoints
One observable behavior at a time
- 01Build at least one case for every required scenario type.
- 02Define the failure envelope and escalation table.
- 03Add claim-level source mappings.
- 04Run the suite and record outcome labels.
- 05Stratify results and document residual risk.
What you'll practice
- Define labeled outcome classes and metric denominators.
- Propagate structured failures through every handoff.
- Preserve claim-level provenance.
- Route humans using ambiguity, impact, and evidence.
Your lab files
Four files for each lab
- 01
decision.mdState the requirement, chosen pattern, rejected alternatives, and the reason for the tradeoff.
- 02
implementation/Keep the smallest working example, configuration, diagram, or synthetic-trace analysis.
- 03
evidence.mdRecord test cases, observed results, failure behavior, limitations, and evidence locations.
- 04
review.mdExplain what must be deterministic, what may remain model-directed, and where a human is required.
What to save
- Evaluation cases and gold labels.
- An escalation decision table and failure envelope.
- A claim/source schema and metric definitions.
- Stratified results and a residual-risk note.
Test bench
What good work includes
- Every required scenario type appears.
- Explicit human requests always hand off and ambiguity prompts clarification.
- Valid-empty differs from inaccessible.
- Partial failures create coverage gaps.
- Sources are called conflicting only after metric, scope, and period are comparable.
- Unsupported claims are detectable and results are stratified.
Review questions
- Which is costlier here: inappropriate autonomy or unnecessary escalation?
- How does scenario mix change the aggregate?
- Which threshold is supported by data rather than intuition?
Strong solutions tend to
- Explicit labels and rules enforced in code.
- Failure details that support the next step and claim-level source links.
- Clear denominators and representative fictional cases.
Partial example
What useful evidence can look like
This is one small example, not a complete solution or a template you must copy. Use the notes to see why the evidence is useful.
Open annotated example from evidence.md / scenario slice
slice: explicit-human-request
cases: 8
immediate_handoff: 8 / 8
unnecessary_investigation: 0 / 8
unsupported_claims: 0 / 8- slice
- Groups cases with the same expected behavior instead of relying on one blended score.
- unnecessary_investigation
- Checks the failure mode that matters for this slice: continuing after the person asked for a human.
Review your work
Use evidence from your work—not confidence alone. For each area, run the check and choose the strongest stage your work fully demonstrates.
Scenario coverage and labels
The suite includes every required situation with outcome labels and metric denominators defined before results are measured.
Map every required scenario type to at least one case, gold label, expected evidence, and metric denominator. Look for missing or overrepresented categories.
Needs another pass
Required scenario types are absent, labels are assigned after seeing output, or reported rates have no clear denominator.
Partly demonstrated
All categories appear, but at least one label rule, denominator, or important scenario variation is unclear.
Demonstrated
Every required category has a prewritten case and gold label, denominators are explicit, and the scenario mix is documented well enough to interpret the aggregate.
Autonomy and escalation
Clear in-policy work stays autonomous while ambiguity, explicit human requests, policy exceptions, and high-impact uncertainty follow distinct routes.
Run one case for each route and compare the observed action with the escalation table, including the evidence available at the decision point.
Needs another pass
Sentiment alone forces escalation, explicit human requests are delayed, or ambiguous and high-impact cases continue without a safe boundary.
Partly demonstrated
The main routes work, but one trigger, clarification step, or threshold lacks a clear rule supported by the case evidence.
Demonstrated
Routine cases resolve autonomously, ambiguity asks a focused question, explicit human requests hand off immediately, and policy or high-impact cases follow documented evidence-based gates.
Failure and partial coverage
Timeouts, unavailable sources, and partial specialist failures remain visible and lead to an explicit recovery or disclosure decision.
Follow the timeout and partial-failure cases through every handoff. Identify the status, attempted work, retained findings, recovery choice, and final coverage statement.
Needs another pass
Failures become empty success, all successful work is discarded, or the final answer claims coverage that the failed path did not provide.
Partly demonstrated
Failure is visible, but useful partial evidence, retry history, next actions, or the exact coverage gap is missing.
Demonstrated
Structured failure context survives each handoff, useful findings remain available, and the system retries, reroutes, escalates, or proceeds with a precise disclosed gap.
Provenance and measured results
Every supported claim keeps its source, conflicts remain visible, and results show important differences between scenario groups.
Trace one supported, one conflicting, and one insufficient-evidence claim, then recalculate one stratified result from the raw case outcomes.
Needs another pass
Claims have only a general source list, conflicting numbers are averaged, unsupported claims pass, or only one overall result is shown.
Partly demonstrated
Most claims and slices are traceable, but source locations, comparable periods, denominators, or residual risks are incomplete.
Demonstrated
Claims keep exact source mappings, conflicts preserve comparable values and dates, unsupported claims are detected, and stratified results with denominators support the stated threshold and residual-risk note.