ab-test-readout

1# A/B Test Readout Skill
2 
3The hard part of an experiment is the readout: not "B won" but "is this real, is it big enough to matter, and should we ship?" This skill turns results into an honest decision — and flags the ways A/B results lie.
4 
5## Working from a brief
6 
7Given results (even partial), **write the full readout anyway**. If significance isn't provided, reason about it from the numbers and flag what's needed to confirm. Mark assumed figures. Never declare a winner without addressing significance and sample.
8 
9## Required Inputs
10 
11Ask for (if not already provided):
12- **The hypothesis** and the **primary metric**
13- **Results** — control vs variant: conversions/rate, sample size per arm, duration
14- **Guardrail metrics** (revenue, retention, latency, complaints) that mustn't regress
15- **Pre-registered decision rule** (what would count as a win) if one exists
16 
17## Output Format
18 
19### 1. Verdict (one line)
20*Ship / Don't ship / Inconclusive — keep running* — with the headline number.
21 
22### 2. The result
23 
24| Metric | Control | Variant | Relative lift | Significant? |
25|---|---|---|---|---|
26| Primary | | | | p / CI |
27| Guardrail(s) | | | | |
28 
29State **statistical** significance (p-value / confidence interval) *and* **practical** significance (is the lift big enough to matter given the cost?).
30 
31### 3. Did it really win?
32Address the ways A/B tests mislead:
33- **Sample / power** — was the test adequately powered, or under-sampled?
34…