ab-test-analysis

1## A/B Test Analysis
2 
3Evaluate A/B test results with statistical rigor and translate findings into clear product decisions.
4 
5### Context
6 
7You are analyzing A/B test results for **$ARGUMENTS**.
8 
9If the user provides data files (CSV, Excel, or analytics exports), read and analyze them directly. Generate Python scripts for statistical calculations when needed.
10 
11### Instructions
12 
131. **Understand the experiment**:
14   - What was the hypothesis?
15   - What was changed (the variant)?
16   - What is the primary metric? Any guardrail metrics?
17   - How long did the test run?
18   - What is the traffic split?
19 
202. **Validate the test setup**:
21   - **Sample size**: Is the sample large enough for the expected effect size?
22     - Use the formula: n = (Z²α/2 × 2 × p × (1-p)) / MDE²
23     - Flag if the test is underpowered (<80% power)
24   - **Duration**: Did the test run for at least 1-2 full business cycles?
25   - **Randomization**: Any evidence of sample ratio mismatch (SRM)?
26   - **Novelty/primacy effects**: Was there enough time to wash out initial behavior changes?
27 
283. **Calculate statistical significance**:
29   - **Conversion rate** for control and variant
30   - **Relative lift**: (variant - control) / control × 100
31   - **p-value**: Using a two-tailed z-test or chi-squared test
32   - **Confidence interval**: 95% CI for the difference
33   - **Statistical significance**: Is p < 0.05?
34   - **Practical significance**: Is the lift meaningful for the business?
35…