Lab assignment
Production architecture review
Synthesize one fictional system into a coherent, reviewable architecture packet.
The brief
Your assignment
Choose a scenario family from the official guide, then create a new fictional organization, constraints, and data. Do not reproduce or imply access to a live exam scenario. Work across several sittings if needed; the source does not prescribe a completion time.
Create one internally consistent packet that connects requirements, system boundaries, context, reliability, provenance, security, evaluation, cost, latency, and tradeoffs.
Before you build
Set up a safe workspace
- Choose lessons you want to synthesize.
- Invent a fictional organization, requirements, and data.
- Plan several sittings and keep a decision log between them.
Build in checkpoints
One observable behavior at a time
- 01Define measurable requirements and success criteria.
- 02Draw the sequence or state diagram and trust boundaries.
- 03Specify agent, tool, deterministic, and human-review boundaries.
- 04Add failure, provenance, security, and evaluation plans.
- 05Review every decision against requirements, alternatives, cost, and residual risk.
What you'll practice
- Trace architecture decisions to measurable requirements.
- Make critical invariants enforceable outside prompts.
- Design observable failure and escalation behavior.
- State uncertainty, assumptions, and rejected alternatives clearly.
Your lab files
Four files for each lab
- 01
decision.mdState the requirement, chosen pattern, rejected alternatives, and the reason for the tradeoff.
- 02
implementation/Keep the smallest working example, configuration, diagram, or synthetic-trace analysis.
- 03
evidence.mdRecord test cases, observed results, failure behavior, limitations, and evidence locations.
- 04
review.mdExplain what must be deterministic, what may remain model-directed, and where a human is required.
What to save
- Requirements and measurable success criteria.
- A sequence diagram or state machine and trust-boundary diagram.
- Agent/tool boundaries and deterministic/human-review gates.
- Context, persistence, provenance, security, evaluation, cost, and latency sections.
- A requirement-to-decision matrix, source register, assumptions, and residual risks.
Test bench
What good work includes
- Every major decision traces to a requirement.
- Every critical invariant has a non-prompt enforcement mechanism.
- Every failure state has retry, reroute, escalation, or disclosure behavior.
- Permissions match actual operations and claims map to sources.
- Costs and latency are labeled estimates unless measured.
- At least one rejected alternative would fit different constraints.
Review questions
- Which changed requirement would reverse the architecture choice?
- What is the smallest system that satisfies the brief?
- What still needs validation before production?
Strong solutions tend to
- Coherent boundaries and least privilege.
- Evidence-driven evaluation and proportionate oversight.
- Observable failures, explicit uncertainty, and clearly stated alternatives.
Partial example
What useful evidence can look like
This is one small example, not a complete solution or a template you must copy. Use the notes to see why the evidence is useful.
Open annotated example from decision.md / tradeoff excerpt
Requirement: high-impact actions need review
Choice: deterministic gate before the mutation tool
Rejected: prompt-only warning
Reason: a prompt can guide selection but cannot guarantee enforcement
Evidence: adversarial case A-06 blocked before mutation- Rejected + Reason
- Shows why the alternative fails the requirement, not merely which feature was chosen.
- Evidence
- Connects the architecture decision to an observed test result.
Review your work
Use evidence from your work—not confidence alone. For each area, run the check and choose the strongest stage your work fully demonstrates.
Requirements and decisions
Every major architecture choice answers a measurable requirement and records a real alternative that would fit different constraints.
Choose each major component in the diagram and follow it through the requirement-to-decision matrix to its requirement, evidence, tradeoff, and rejected alternative.
Needs another pass
Components are chosen because they are familiar, requirements are not measurable, or alternatives are listed without explaining why they lost.
Partly demonstrated
Most decisions trace to requirements, but at least one major choice lacks measurable success criteria, evidence, or a realistic alternative.
Demonstrated
Every major choice traces to a measurable requirement, states its tradeoff and evidence, and names an alternative plus the changed constraint that would make it preferable.
Boundaries and enforcement
Agent, tool, data, trust, permission, and human boundaries agree across the diagrams and place critical rules outside prompt wording.
Trace one normal and one protected action across the sequence and trust diagrams. At every boundary, name the allowed data, permission, validation, and enforcing component.
Needs another pass
Roles overlap, permissions exceed actual operations, or a critical safety or authorization rule exists only in a prompt.
Partly demonstrated
The main boundaries are drawn, but at least one protected action lacks matching permission, validation, authorization, or human approval in the written design.
Demonstrated
Diagrams and contracts agree, each role has least privilege, inputs and outputs are validated, and every critical invariant is enforced by deterministic code, a safe hook, or an explicit human decision.
State, evidence, and failure
The architecture preserves exact state and claim sources while making every important failure and escalation path observable.
Follow one exact fact and one source-backed claim through the system, then inject a tool failure, partial result, stale state, and explicit human request into the relevant paths.
Needs another pass
Exact facts depend on conversation memory, claim sources disappear, or a failure can look successful without a recovery or disclosure path.
Partly demonstrated
State and common failures are covered, but freshness, partial coverage, source locations, escalation context, or a terminal status is missing.
Demonstrated
Durable facts retain evidence and freshness, claims keep exact sources, and every modeled failure ends in retry, reroute, escalation, or a clear report of what remains incomplete.
Evaluation and production readiness
The packet explains how quality, security, cost, latency, and remaining risk will be measured without presenting assumptions as facts.
Trace every target, cost, and latency figure to a measurement or an estimate label, then verify that the evaluation set covers critical slices and that residual risks have owners or next checks.
Needs another pass
The design relies on one aggregate score, unlabeled estimates, assumed security, or claims that the system is production-ready without unresolved risks.
Partly demonstrated
An evaluation and risk plan exists, but important slices, denominators, estimate assumptions, security tests, or next validation steps are incomplete.
Demonstrated
Success metrics have explicit denominators and critical slices, security boundaries have tests, estimates are labeled with assumptions, and residual risks, owners, and pre-production validation steps are stated plainly.