ab-test-setup

1# A/B Test Setup
2 
3## 1️⃣ Purpose & Scope
4 
5Define an experiment that can answer a specific product question, and verify its assumptions before exposing users. This procedure cannot guarantee validity by itself.
6 
7- Documents the stopping rule
8- Estimates sample needs under stated assumptions
9- Makes the hypothesis and decision criteria reviewable
10 
11---
12 
13## 2️⃣ Pre-Requisites
14 
15You must have:
16 
17- A clear user problem
18- Access to an analytics source
19- Roughly estimated traffic volume
20 
21### Hypothesis Quality Checklist
22 
23A valid hypothesis includes:
24 
25- Observation or evidence
26- Single, specific change
27- Directional expectation
28- Defined audience
29- Measurable success criteria
30 
31---
32 
33## 3️⃣ Hypothesis Lock (Hard Gate)
34 
35Before designing variants or metrics, you MUST:
36 
37- Present the **final hypothesis**
38- Specify:
39  - Target audience
40  - Primary metric
41  - Expected direction of effect
42  - Minimum Detectable Effect (MDE)
43 
44Use the hypothesis already agreed in the task. If a launch-critical choice is missing, present the concrete choice for confirmation while continuing independent analysis. Do not repeatedly request approval for a decision already authorized.
45 
46---
47 
48## 4️⃣ Assumptions & Validity Check (Mandatory)
49 
50Explicitly list assumptions about:
51 
52- Traffic stability
53- User independence
54- Metric reliability
55- Randomization quality
56- External factors (seasonality, campaigns, releases)
57 
58If assumptions are weak or violated:
59 
60- Warn the user
61- Recommend delaying or redesigning the test
62 
63---
64…