with the skill
Here's a draft for Slack:
---
:rotating_light: *Incident Summary: Checkout Degradation (Oct 25)*
*Summary*
On Oct 25, checkout was degraded for 47 minutes (14:06-14:53 UTC). About 3% of checkout attempts failed, or roughly 2,100 orders. No data was lost.
*Root cause*
A config change merged at 14:02 UTC lowered the payment service DB connection pool from 50 to 5. Under normal traffic the pool was exhausted, and payment requests began to fail.
*Why it got through*
Config changes skip the load test that code changes run, so nothing exercised the new pool size before it reached production.
*Timeline (Oct 25, UTC)*
- 14:02: Config change merged, lowering the payment service DB connection pool from 50 to 5
- 14:06: Checkout error rate begins to climb
- 14:11: On-call (Mei) paged
- 14:30: Mei identifies pool exhaustion in the payment-service logs
Show the rest (22 more lines)
- 14:49: Config reverted
- 14:53: Error rate back to normal
*What went well*
- On-call identified the cause from the logs about 19 minutes after the page.
- The revert was clean, and no data was lost.
*What we're improving*
- Detection took too long. The first alert came about 5 minutes after errors began, and nothing flagged the pool itself.
- Config changes had no pre-production safety net.
*Follow-ups*
- [ ] Add config changes to the load test pipeline. Owner: Lena, by Nov 6
- [ ] Alert on DB connection pool usage above 80%. Owner: Mei, by Oct 30
Questions or additional context? Reply in the thread.
---
Notes:
- I kept it blameless and described the change rather than naming the author. If your org's norm is to name people, it's an easy edit.
- If you have a link to the incident doc or postmortem, add it under the summary.