with the skill
# Postmortem: Label API outage from a blocking index migration
**Date of incident**: 2026-10-14 (all times UTC)
**Authors**: TBD
**Status**: Draft
**Severity**: SEV2 (proposed, to be confirmed)
**Duration**: 43 minutes (13:58 to 14:41). Writes to `shipments` were blocked for 31 minutes (13:58 to 14:29).
## Executive Summary
On 2026-10-14, deploy `2026.10.14-3` automatically ran migration 0142, which created an index on `shipments(carrier_id, created_at)` without `CONCURRENTLY`. `shipments` has about 48M rows, so the build held a lock that blocked all writes to the table. Label creation hung for 31 minutes. The on-call engineer traced the problem to the index build and cancelled it, after which writes resumed and the backlog drained.
Several gaps let this reach production and lengthened the incident. Nothing in the pipeline flags or limits a blocking migration, the latency alert needs 10 minutes to fire, and there is no runbook for lock contention.
**Impact**
- ~6,200 label requests failed or timed out
- 214 merchants affected
- 57 support tickets
- 38 duplicate labels created by client retries, voided by hand ($412 in carrier fees refunded)
- No data lost
Show the rest (104 more lines)
## Timeline
| Time | Event |
| ---- | ----- |
| 13:58 | Deploy 2026.10.14-3 runs migration 0142 (`CREATE INDEX` on `shipments`, not concurrent). Writes to `shipments` begin blocking. |
| 14:03 | First support ticket (Bramble Goods): "label creation hanging" |
| 14:06 | 11 more tickets |
| 14:09 | Support asks in #eng whether anything is down |
| 14:12 | PagerDuty fires: label-api p99 latency > 10s for 10 min. Dana Okafor paged. |
| 14:15 | Dana acks and starts checking label-api pods, which look healthy |
| 14:24 | Dana finds ~3,900 queries waiting on a lock in `pg_stat_activity` and traces it to the index build |
| 14:27 | Decision to cancel the index build |
| 14:29 | Build cancelled. Writes resume and the backlog starts draining. |
| 14:41 | p99 back to normal. Incident closed. |
## Root Cause Analysis
### What happened
Migration 0142 built an index on a 48M-row table without `CONCURRENTLY`. A non-concurrent build blocks writes for its whole duration. The migration ran automatically as part of the deploy, so label creation queued behind the lock until the build was cancelled.
### Why it happened
1. **Proximate cause**: a non-concurrent `CREATE INDEX` ran against a large, write-heavy table.
2. **Contributing factors**:
- **No automated check caught it.** The migration linter checks naming only and does not flag `CREATE INDEX` without `CONCURRENTLY`.
- **No limit on lock waits.** The migration runner sets no `lock_timeout`, so the migration could block production writes indefinitely.
- **Migrations run automatically on deploy** with no gate for risky operations on large tables.
- **Slow detection.** The latency alert requires 10 minutes over threshold, so customers (12 tickets by 14:06) noticed before the page at 14:12. Support's question in #eng at 14:09 had no path to a page.
- **Slow diagnosis.** There is no runbook for lock contention. Dana spent about 9 minutes (14:15 to 14:24) checking healthy pods before finding the lock.
- **Retries amplified the damage.** Client retries produced 38 duplicate labels with real carrier fees. This suggests label creation is not idempotent, which should be confirmed.
### 5 Whys
1. **Why did label creation fail?** Writes to `shipments` were blocked by a lock.
2. **Why was there a lock?** A non-concurrent index build on a 48M-row table was running.
3. **Why did that build run in production?** The migration was written that way, and nothing in review, linting, or the runner flagged or limited it.
4. **Why did nothing flag it?** The linter covers naming only, and the runner has no `lock_timeout` or risky-operation gate.
5. **Why were those safeguards missing?** Migration safety has not been treated as a platform-level concern, so a single migration's correctness depended on the author knowing this Postgres behavior.
### A note on the "rookie mistake" comment
The debrief included a comment that the migration author should have known better. This postmortem does not treat the author's actions as a cause. The migration was a reasonable-looking change that passed every check we have. Postgres's locking behavior is easy to miss, and our process has no step that would catch it. If the system relies on every author remembering this, it will fail again with someone else. The fixes below add that safety net.
## Detection
**What worked**
- The latency alert fired and paged on-call, and Dana acknowledged within 3 minutes.
**What didn't**
- The 10-minute alert window meant a 14-minute gap between the deploy and the page.
- Support saw the problem first (14:03) but had no path to a page.
- No alert correlates a deploy with a migration that is still running.
## Response
**What worked**
- Once the lock was found, the decision to cancel took 3 minutes (14:24 to 14:27).
- Cancelling the build resumed writes immediately, and recovery took about 12 more minutes.
**What could be improved**
- Diagnosis took 9 minutes because the first instinct (check the pods) was reasonable but wrong, and nothing pointed toward the database.
- No runbook covers lock contention or `pg_stat_activity` triage.
- Duplicate labels had to be voided by hand.
## Lessons Learned
**What went well**
- On-call acknowledged quickly and found the root cause with the tools at hand.
- Cancelling the build was a clean, fast mitigation.
- No data was lost.
**What went wrong**
- A blocking migration reached production unchecked.
- Customers detected the issue before our alerting did.
- Retries created duplicate labels and refunded fees.
**Where we got lucky**
- The build could be cancelled without damaging the table.
- The backlog drained within about 12 minutes of the cancel.
## Action Items
Owners are filled in only where the notes assign them. Tickets and due dates still need to be created. The proposed priorities are open for discussion in the review meeting.
| Priority | Action | Owner | Due | Ticket |
| -------- | ------ | ----- | --- | ------ |
| P0 | Add a `lock_timeout` to the migration runner so migrations fail fast instead of blocking writes | Dana Okafor (offered) | TBD | TBD |
| P0 | Extend the migration linter to flag `CREATE INDEX` without `CONCURRENTLY` (and other blocking operations) and fail the deploy | Platform team (Sofia Reyes) | TBD | TBD |
| P1 | Write a runbook for lock contention (`pg_stat_activity` triage, how to cancel safely) and link it from the label-api alerts | TBD | TBD | TBD |
| P1 | Shorten alert detection for label-api (for example, a faster-burn latency or error-rate alert) | TBD | TBD | TBD |
| P1 | Investigate making label creation idempotent, or otherwise preventing duplicates on retry | TBD | TBD | TBD |
| P2 | Add a path from a surge in support tickets to a page, or at least to on-call | TBD | TBD | TBD |
| P2 | Consider a gate or separate step for migrations on large tables rather than auto-running on every deploy | TBD | TBD | TBD |
## Open Questions
- What is the current state of migration 0142? Was the index eventually created concurrently, and was the migration marked as applied or rolled back?
- Is label creation currently idempotent? Do clients send an idempotency key?
- Was the deploy rolled back, or did the rest of `2026.10.14-3` stay live?
- What severity does this meet under our definitions?
## Related Incidents
None identified in the notes. Check for earlier migration-related lock incidents.
I left out ticket numbers, due dates, and unassigned owners rather than invent them. Fill those in at the review meeting.