with the skill
# Pricing page test: don't ship yet
The headline result looks like a win, but the test fails a validity check. I'd hold the decision until the cause is found.
## Headline numbers
| Metric | Control | Variant | Diff | Relative lift | p-value | 95% CI (abs. diff) |
|---|---|---|---|---|---|---|
| Paid conversion | 4.00% (2,016 / 50,412) | 4.30% (2,072 / 48,190) | +0.30 pp | +7.5% | ~0.018 | +0.05 to +0.55 pp |
Taken at face value (two-tailed z-test, z ≈ 2.37), this is statistically significant. The confidence interval is wide, though: the true lift could be anywhere from about +1% to +14% relative.
## The problem: sample ratio mismatch (SRM)
You planned a 50/50 split, but the arms are 51.1% control and 48.9% variant.
- Expected: about 49,301 users per arm. The variant is about 1,100 users short.
- Chi-squared ≈ 50, p ≈ 10⁻¹² (about a 7σ deviation).
- Random assignment by user ID should not produce this. A p-value below 0.001 on an SRM check means the assignment or measurement pipeline is broken somewhere.
Show the rest (26 more lines)
**Why this matters:** users are going missing from one arm, and they are probably not missing at random. Examples:
- If the new layout loads slower, errors out, or has tracking that fails for some browsers, the people who drop out are likely low-intent visitors. That leaves a more motivated variant population and inflates conversion.
- Bot filtering or caching rules that differ by arm would have a similar effect.
**Sensitivity check:** suppose the variant should have had as many users as control (50,412), and the roughly 2,200 missing users would all have been non-converters. Variant conversion becomes 4.11% vs 4.00%, a +2.8% lift with p ≈ 0.37, which is not significant. That is a worst case, but it shows the observed win is fragile to whatever is causing the mismatch.
## What passes and what's unknown
- **Duration:** 14 days covers two full weekly cycles, which is good. A novelty effect is still possible on a pricing page, so check whether the lift is stable across days.
- **Power:** you didn't give the MDE, so I can't judge this. The CI width suggests the test can detect roughly 0.5 pp but not much less.
- **Guardrails:** I have no data on these. For pricing, I'd want revenue per user, plan mix (a shift to cheaper plans can raise conversion while lowering revenue), refund and churn rates, and page load time.
## Recommendation: **Investigate**
1. **Find the cause of the SRM.**
- Check where assignment is logged versus where users are counted (assignment vs. exposure).
- Compare variant load time, error rates, and redirects.
- Look at the mismatch by device, browser, country, and traffic source. The segment with the biggest deviation usually points to the bug.
- Check bot and internal-traffic filtering, caching or CDN behavior, and whether the experiment config changed mid-test.
2. **Don't use these results for the decision** unless the cause is found and shown to be unrelated to user intent.
3. **If it's fixable, rerun the test** with the same 50/50 split and an SRM check in the first few days. Pre-register the primary metric and guardrails, and agree on the MDE up front.
4. **Pull the guardrail metrics** from the existing data in the meantime. If revenue per user is flat or down, that's a reason to be cautious even without the SRM.
If you can share the daily or segment-level breakdown, or the raw user-level data, I can run the SRM segmentation and check the lift for stability over time. Those two checks would show whether this is salvageable.
*Figures were calculated by hand from the numbers you provided, not from raw data. I can generate a script to verify them.*