with the skill
# A/B Test Results: Shorter Onboarding Checklist
**I wouldn't write this up as a confirmed win yet.** The numbers lean positive, but the way the test was stopped makes the p = 0.044 much weaker evidence than it looks. A write-up that says "significant, ship it" would overstate what you know.
## The numbers
| Metric | Control | Variant | Difference | p-value | Significant at 0.05? |
|---|---|---|---|---|---|
| Activation rate | 4.20% (412 / 9,800) | 4.80% (468 / 9,750) | +0.60 pp (+14.2% relative) | 0.044 (two-tailed z-test) | Nominally yes |
- **95% CI for the difference:** +0.02 pp to +1.18 pp, or about +0.4% to +28% relative. The data are consistent with a negligible effect and with a large one.
- **SRM check:** 9,800 vs 9,750 against an expected 50/50 split gives p ≈ 0.72, so randomization looks fine.
- **Guardrails:** none were provided (e.g., 7-day retention, support tickets, downstream conversion). I can't say whether any degraded.
## Why this isn't ship-ready
1. **Optional stopping.** You checked daily and stopped at the first day p < 0.05. Each look is another chance for noise to cross the line. With about 6 looks, the real false-positive rate is roughly 12–15%, not 5%. A sequential design such as O'Brien-Fleming would require a z-score of about 3.5 or more this early. Yours is z ≈ 2.01.
2. **Only about a third of the planned sample.** You have about 9,800 users per arm against 30,000 planned. Effects estimated at the moment a test first crosses significance tend to be inflated (winner's curse), so the true lift is probably smaller than +14%.
3. **Less than one business cycle.** Six days doesn't cover a full weekly cycle, so weekday/weekend mix could be skewing who shows up. Novelty effects on a new onboarding flow also haven't had time to wear off.
4. **A borderline result.** p = 0.044 with a CI that nearly touches zero is the kind of result that often fails to replicate.
Show the rest (16 more lines)
## Recommendation: Extend the test (don't ship on this result)
**Reasoning:** The trend is positive and plausible, since a shorter checklist reducing friction is a sensible mechanism. But the evidence doesn't meet the bar you set at the start.
**Next steps:**
1. **Resume the test and run it to day 21.** Don't look at p-values in the meantime. If you want to monitor, watch only guardrails and sample-ratio health.
2. **Analyze two ways at the end:** the full 21 days (the pre-registered analysis), and days 7–21 alone as a clean replication. If the effect holds in the later window, that's strong evidence.
3. **Pull guardrails now:** downstream retention, time to first key action, support contacts. A shorter checklist can lift activation by skipping steps that matter later.
4. **Decide the ship criteria in advance.** For example: ship if the day-21 difference is positive, p < 0.05, and the guardrails are flat.
5. **For future tests,** use a sequential testing method (or a fixed horizon with no peeking) if you want the option to stop early.
## If you need to ship sooner
That's a business call and a reasonable one if the change is low-risk and easy to revert. In that case, frame it honestly: *"Early data showed a positive but unconfirmed lift (+0.6 pp, p = 0.044, stopped at 33% of planned sample). We're rolling out with monitoring and will confirm against the full test."* Keep a holdout group if you can, so the test keeps collecting data after launch.
If you can share guardrail metrics or daily results, I can check for day-of-week patterns and write the final day-21 analysis once it's done.