# Workshop Guide: Experimental Design

## Audience

Product managers, analysts, researchers, and operators who need to turn experiment ideas into review-ready designs.

## 60-Minute Agenda

1. 0-10 min: Pick one real decision that could be tested.
2. 10-20 min: Rewrite the decision as a causal question.
3. 20-35 min: Draft estimand, unit, exposure, primary metric, and guardrails.
4. 35-50 min: Peer review for invalidation risks.
5. 50-60 min: Share one ship/no-ship decision rule and one unresolved risk.

## 90-Minute Agenda

1. 0-10 min: Introduce the design memo and worked example.
2. 10-25 min: Teams draft decision, causal question, and estimand.
3. 25-45 min: Teams add metrics, randomization unit, power/MDE, and validity checks.
4. 45-60 min: Hostile review swap.
5. 60-75 min: Revise memo from critique.
6. 75-90 min: Group readout and facilitator synthesis.

## Team Exercise

Each team chooses one experiment idea and produces:

- causal question
- estimand
- primary metric
- guardrails
- randomization unit
- validity checks
- decision rule

## Discussion Prompts

- What result would be statistically significant but not worth acting on?
- What result would be practically important but inconclusive?
- What would invalidate the experiment before analysis?
- Which guardrail would make you stop the rollout?

## Facilitator Notes

Push teams to name a decision, not just a question. If a team cannot say what would change after the experiment, the design is not ready.

Common failure modes:

- randomization unit does not match interference structure
- too many primary metrics
- no minimum worthwhile effect
- decision rule postponed until after results

## Review Standard

Use `final-assessment.md` as the rubric. A strong memo is specific enough that another analyst could implement the experiment and another reviewer could judge whether the result is valid.