# Answer Key: Channel Conversion Model Criticism

Use this as a calibration guide, not a single correct answer.

## Core Readout

The channel-level conversion differences should not be read as five independent truths or as one universal effect. Complete pooling can hide real channel heterogeneity; no pooling can overreact to small cells. Partial pooling is a sensible default because it stabilizes noisy channel estimates while still allowing important differences to remain visible.

## What A Strong Answer Should Say

- Compute channel-level treatment-control differences and inspect total users per channel.
- Explain why smaller channels deserve more shrinkage toward the overall pattern.
- Explain why large strategic channels may still need channel-specific decisions.
- Propose posterior predictive checks: channel-level conversion distribution, extreme channel rates, aggregate lift, and calibration by treatment arm.
- Recommend a staged rollout or holdback where uncertainty is strategically important.

## Common Mistakes

- Declaring the highest-lift channel the winner without checking sample size.
- Treating complete pooling as objective because it produces one clean estimate.
- Treating no pooling as more honest when it is mostly noisier.
- Skipping posterior predictive checks and going straight to rollout.

## Instructor Notes

Ask learners what decision changes if the smallest channel has the largest estimated lift. The useful answer is not "ignore it"; it is "shrink it, inspect uncertainty, and decide whether that channel matters enough to gather more data."