with the skill
## A/B Test Results: Shorter Onboarding Checklist
**Recommendation: Do not ship yet—confirm the result with a valid testing plan.**
The variant shows a promising activation increase, but the test stopped at the first daily p-value below 0.05. That makes the nominal significance insufficient to declare a winner.
### Experiment summary
**Hypothesis:** A shorter onboarding checklist increases activation.
**Planned:** 21 days, approximately 30,000 users per arm, 50/50 allocation.
**Actual:** Stopped on day 6 after daily significance checks; 19,550 users total—approximately 33% of the planned sample.
| Metric | Control | Variant | Change |
|---|---:|---:|---:|
| Users | 9,800 | 9,750 | — |
| Activated users | 412 | 468 | — |
| Activation rate | 4.20% | 4.80% | **+0.60 percentage points** |
| Relative activation lift | — | — | **+14.2%** |
**Conventional fixed-horizon statistics:**
Show the rest (23 more lines)
- Two-sided, two-proportion z-test: **p ≈ 0.0445**
- Approximate 95% confidence interval for the absolute difference: **+0.015 to +1.177 percentage points**
**Important:** These are *nominal, unadjusted* statistics. Their usual false-positive and confidence-coverage guarantees do not apply to this stop-on-significance procedure.
### Validity checks
- **Repeated checking and early stopping:** Checking daily and stopping at the first p < 0.05 increases false-positive risk unless a valid sequential method was specified. The final counts alone are insufficient to calculate an appropriate sequentially adjusted result.
- **Duration:** Six days did not cover even one complete weekly cycle, much less the planned three weeks. Weekday mix and novelty effects remain unresolved.
- **Traffic allocation:** No evidence of sample ratio mismatch: the observed allocation is consistent with 50/50 (**SRM p ≈ 0.72**). This does not establish that all other aspects of randomization were correct.
- **Power:** The test collected substantially less than its planned sample. Whether it met an 80% power target cannot be established without the prespecified minimum detectable effect and power assumptions.
- **Guardrails and measurement:** No guardrail results were provided. Confirm downstream retention, revenue, and onboarding quality, and verify that both arms had equal time to complete the activation window.
### Decision and next steps
**Treat this as a promising signal, not a confirmed win.** The observed lift would mean approximately six additional activations per 1,000 users if it persists, but durability and downstream value remain unverified.
1. Preserve the daily results and document the early-stopping decision.
2. Run a confirmatory test with a prespecified primary metric, activation window, meaningful effect threshold, guardrails, and stopping rule.
3. Either use a fixed-horizon design without significance-based stopping or a properly designed sequential test that permits interim decisions.
4. Ship after valid confirmation and acceptable guardrail outcomes.
**Stakeholder summary:** “The shorter checklist increased observed activation from 4.20% to 4.80%. However, because we stopped at the first significant daily result, this is not yet a statistically confirmed win. We recommend confirmation before full rollout.”