# Final Assessment: A/B Testing at Scale

## Assignment

Design an experiment review process for an organization running many concurrent experiments.

Your process must define risk tiers, intake requirements, metric contracts, automated checks, human review criteria, launch readiness, readout order, archive requirements, and operating rhythm.

## Required Artifact

Submit an experiment review-board operating model.

Minimum sections:

- review charter
- risk tiers
- intake fields
- metric contract requirements
- automated checks
- human review criteria
- launch readiness checks
- readout order
- experiment card schema
- operating rhythm
- strongest critique

## Rubric

| Criterion | Strong | Needs revision |
|---|---|---|
| Charter | Balances learning velocity with validity and safety | Optimizes only speed or only control |
| Risk tiers | Review depth is proportional to user and business risk | Applies same process to every test |
| Intake | Requires decision, hypothesis, population, treatment, control, metrics, and owner | Collects fields without decision ownership |
| Metrics | Uses metric contracts with numerator, denominator, window, exclusions, and owner | Relies on metric names alone |
| Launch checks | Requires assignment, exposure, dashboard, SRM, rollback, and owner readiness | Defers instrumentation checks |
| Readout | Starts with validity before primary metric and guardrails | Starts with whichever metric looks best |
| Archive | Preserves reusable learning and follow-up | Stores only winner and p-value |
| Critique | Identifies how the process can become slow or performative | Assumes governance always helps |

## Pass Criteria

Pass if the operating model would help teams run more trustworthy experiments without burying low-risk learning in needless process.

## Submission Checklist

- [ ] Risk tiers are defined.
- [ ] Required intake fields are listed.
- [ ] Metric contracts are specified.
- [ ] Automated checks are separated from human review.
- [ ] Launch readiness includes instrumentation.
- [ ] Readout order starts with validity.
- [ ] Archive schema preserves reusable learning.

## Certificate Language

Completed A/B Testing at Scale by designing a risk-tiered experiment review process with metric contracts, launch checks, readout discipline, and decision memory.