with the skill
# Postmortem: Label Creation Disruption from Blocking Index Migration
**Incident date:** October 14, 2026
**Status:** Draft for review
**Severity:** Not recorded
**Incident window:** 13:58–14:41 UTC — 43 minutes
**Affected service:** Label API
## Executive Summary
On October 14, deployment `2026.10.14-3` automatically ran migration `0142`, which created an index on `shipments(carrier_id, created_at)`. The table contains approximately 48 million rows. Because the index was created without `CONCURRENTLY`, the operation blocked writes to `shipments`, causing label requests to hang, fail, or time out.
The incident affected 214 merchants and approximately 6,200 label requests. Client retries created 38 duplicate labels, which were manually voided; $412 in carrier fees was refunded. No data was lost.
The on-call engineer identified the blocking index build at 14:24. It was canceled at 14:29, allowing writes to resume. The backlog drained and p99 latency returned to normal at 14:41.
The failure was enabled by automated execution of a write-blocking migration without safeguards appropriate for a large production table. Migration linting checked naming only, and there was no lock-contention runbook. Detection also lagged customer reports because the latency alert required ten minutes above threshold.
## Customer Impact
Show the rest (84 more lines)
- Approximately **6,200 label requests** failed or timed out.
- **214 merchants** were affected.
- Support received **57 tickets**.
- Client retries created **38 duplicate labels**, subsequently voided manually.
- **$412 in carrier fees** was refunded.
- **No data was lost.**
Writes resumed at 14:29, but customer-facing latency did not fully recover until 14:41 as the backlog drained.
## Timeline
All times UTC on October 14, 2026.
| Time | Event |
|---|---|
| 13:58 | Deployment `2026.10.14-3` runs migration `0142`: `CREATE INDEX` on `shipments(carrier_id, created_at)` without `CONCURRENTLY`. Writes to the approximately 48-million-row table block. |
| 14:03 | First support ticket arrives from Bramble Goods reporting that label creation is hanging. |
| 14:06 | Eleven additional support tickets have arrived. |
| 14:09 | Support asks in `#eng` whether a service is down. |
| 14:12 | PagerDuty alerts on label-api p99 latency exceeding ten seconds for ten minutes. Dana Okafor is paged. |
| 14:15 | Dana acknowledges and begins investigating label-api pods, which appear healthy. |
| 14:24 | Dana finds approximately 3,900 queries waiting on a lock in `pg_stat_activity` and traces the contention to the index build. |
| 14:27 | Decision made to cancel the index build. |
| 14:29 | Index build canceled. Writes resume and the backlog begins draining. |
| 14:41 | p99 latency returns to normal. Incident closed. |
## Cause and Contributing Factors
### Direct cause
Migration `0142` used a non-concurrent index build on `shipments`. That operation blocked writes for the duration of the build, causing requests dependent on those writes to accumulate.
### Systemic contributing factors
- **Deployment automatically executed migrations.** A potentially long-running, write-blocking schema change ran as part of the normal deployment path.
- **Migration checks did not assess operational safety.** The linter checked naming but did not flag `CREATE INDEX` without `CONCURRENTLY`.
- **No migration `lock_timeout` was configured.** Migration lock acquisition had no configured wait bound. However, adding `lock_timeout` alone would not prevent this incident: it limits time spent *waiting to acquire* a lock, not how long an acquired lock is held during an index build.
- **Latency alerting delayed the page.** The alert required ten minutes above threshold. The page arrived 14 minutes after the migration began and nine minutes after the first customer report.
- **No lock-contention runbook existed.** Initial investigation focused on apparently healthy application pods for nine minutes before database lock contention was identified.
- **Retries amplified the impact.** Client retries produced duplicate labels. The notes do not establish which retry or duplicate-prevention behavior allowed this; that requires follow-up investigation.
The corrective focus is on migration controls and response tooling, not individual judgment. The deployment path allowed this migration to reach production without detecting or containing its operational risk.
## Detection and Response
### What worked
- Support surfaced customer symptoms before the automated page.
- Database inspection exposed the lock queue and its blocking operation.
- Canceling the index build restored writes.
- Recovery was confirmed through normalized p99 latency.
- Duplicate labels were identified and manually voided.
### What could improve
- Automated paging followed customer reports rather than preceding them.
- Healthy pods did not rule out a blocked database dependency; responders lacked a runbook directing them to lock diagnostics.
- Approximately 3,900 queries accumulated behind the blocking operation.
- Recovery required manual cancellation and a further 12 minutes of backlog drainage.
- Duplicate-label remediation required manual work.
## Follow-up Actions
**Owners below are proposed unless explicitly noted; assignments and due dates must be confirmed before this postmortem is finalized.**
| Priority | Action and completion criteria | Owner | Due |
|---|---|---|---|
| P0 | Add migration linting that rejects non-concurrent index creation on existing production tables unless an explicit exception is approved. Include regression tests for migration `0142`’s pattern. | Platform — Sofia Reyes, proposed accountable lead | TBD |
| P0 | Add migration `lock_timeout` handling with a safe failure path. Verify that a migration unable to acquire its lock exits rather than waiting indefinitely. | Dana Okafor — volunteered; assignment to confirm | TBD |
| P0 | Define and implement a safe production index-build procedure, including concurrent execution outside a transaction where required, monitoring, cancellation, and cleanup of failed builds. Validate before retrying migration `0142`. | TBD | TBD |
| P1 | Define execution-time safeguards for migrations, separate from lock-acquisition timeouts, and test cancellation behavior for long-running operations. | TBD | TBD |
| P1 | Create a lock-contention runbook covering blocker identification, `pg_stat_activity`, cancellation decisions, and recovery verification. Validate it in a response exercise. | TBD | TBD |
| P1 | Review the ten-minute latency-alert window and add actionable detection for sustained write blocking or growing lock queues. Test detection with a representative blocking scenario. | TBD | TBD |
| P1 | Investigate the 38 duplicate labels and implement or strengthen retry-safe label creation. Verify that retrying a timed-out request cannot create an additional label. | TBD | TBD |
## Key Lessons
1. Schema changes must be evaluated for locking behavior at production scale, not just SQL correctness.
2. Automated deployment paths need automated migration-safety checks.
3. Lock-acquisition timeouts and limits on long-running operations address different risks; both require deliberate design.
4. Database-lock diagnostics should be part of the initial response to hanging requests, even when application pods appear healthy.
5. Request timeouts and retries can create secondary customer impact unless label creation is retry-safe.
**Before finalization:** Confirm severity, action owners, and due dates; attach available deployment logs, lock diagnostics, latency graphs, and remediation records.