with the skill
## A/B Test Results: New Pricing Page
**Recommendation: Investigate before shipping.** The variant shows a statistically significant conversion increase, but the traffic allocation has a serious **sample ratio mismatch (SRM)** that could invalidate the result.
**Duration:** 14 days · **Randomization:** User ID · **Planned split:** 50/50
### Conversion results
| Metric | Control | Variant |
|---|---:|---:|
| Users | 50,412 | 48,190 |
| Paid conversions | 2,016 | 2,072 |
| Paid conversion rate | 4.00% | 4.30% |
- **Absolute lift:** +0.301 percentage points
- **Relative lift:** +7.52%
- **Two-sided pooled z-test:** p ≈ 0.018
- **95% confidence interval for the absolute difference:** +0.051 to +0.550 percentage points
On its face, this is a promising result: approximately **3 additional paid conversions per 1,000 users**. These statistical results assume the experiment data are valid.
Show the rest (16 more lines)
### Blocking issue: the split is unexpectedly uneven
With 98,602 users and a planned 50/50 split, we would expect **49,301 users per arm**. Instead, the observed split is **51.13% / 48.87%**.
A chi-squared SRM test gives **χ² ≈ 50.07, p ≈ 1.5 × 10⁻¹²**. This imbalance is extraordinarily unlikely under the intended allocation.
It does **not** prove the variant is worse or the experiment is broken, but it is a strong warning of assignment, eligibility, exposure, or logging problems. Comparing conversion rates does not automatically correct selection bias.
### What to do next
1. **Audit allocation and inclusion.** Compare assignment logs with the users included in this table. Check missing events, exclusions, redirects, identity handling, and whether inclusion depends on behavior affected by the layout.
2. **Locate the mismatch.** Break assignment counts down by day, device, browser, and acquisition channel.
3. **Check conversion maturity and guardrails.** Confirm equal conversion windows and review revenue per user, plan mix, refunds/cancellations, and page performance.
4. **Confirm the planned analysis.** Fourteen days covers two weekly cycles, but adequate power cannot be established without a prespecified minimum detectable effect. Also verify whether this was a fixed-horizon test or whether repeated significance checks influenced stopping.
**Bottom line:** The conversion lift is encouraging, but **do not roll out to 100% yet**. Resolve the SRM first, then reanalyze trustworthy data—or rerun if the allocation or measurement issue cannot be repaired.