with the skill
Below is a blameless rewrite. I kept to the facts in your draft and used placeholders where I'd otherwise be inventing owners, dates, or tickets.
## Root Cause
The production TLS certificate for `api.fernway.example` expired on 2026-10-03, and nothing in our process or tooling caught it before it did.
Several conditions allowed this:
1. **Manual, yearly renewal.** The certificate is bought annually and installed by hand on the load balancer. A once-a-year task with no automation is easy to miss and depends entirely on someone remembering.
2. **Broken warning channel.** Expiry warnings went to `certs@fernway.example`, an alias deleted in the June mailbox cleanup. The warnings were sent but reached no one. Nothing in the cleanup process flagged that the alias was used for external notifications.
3. **No monitoring for expiry.** Our uptime monitor checks only that the API returns HTTP 200. It doesn't check certificate validity or time to expiry, so there was no independent warning.
4. **Single point of ownership.** Renewal depended on one person, with no backup owner and no shared tracking.
5. **A known weakness that recurred.** A renewal was also missed in 2025-09. The fix then did not remove the underlying dependency on a manual process. This is a pattern, which points to a systemic gap rather than an individual lapse.
**What worked:** Staging certificates are issued automatically by Let's Encrypt and have never lapsed. That shows automated issuance works in our environment.
## Action Items
| Priority | Action | Type | Owner | Due | Ticket |
|---|---|---|---|---|---|
Show the rest (18 more lines)
| P0 | Renew and verify the production cert chain (if not already done as part of the incident fix) | Mitigation | TBD | TBD | TBD |
| P0 | Add certificate-expiry monitoring for all public endpoints, alerting at 30,14, and 7 days to the on-call channel (not an individual mailbox) | Detection | TBD | TBD | TBD |
| P1 | Automate production certificate issuance and renewal (ACME/Let's Encrypt, or the provider's automation), following the staging setup. Confirm load balancer support | Prevention | TBD | TBD | TBD |
| P1 | Replace `certs@` with a monitored shared distribution list or ticketing queue, and document who receives it | Prevention | TBD | TBD | TBD |
| P1 | Audit other deleted or retired aliases for external services that send notifications (domains, vendors, billing, certs) | Prevention | TBD | TBD | TBD |
| P2 | Add a "check external dependencies on this alias/mailbox" step to the mailbox cleanup checklist | Prevention | TBD | TBD | TBD |
| P2 | Inventory all certificates (owner, expiry, renewal method) and assign a primary and backup owner to any that stay manual | Prevention | TBD | TBD | TBD |
| P2 | Review the 2025-09 postmortem actions and record why they didn't prevent recurrence | Learning | TBD | TBD | TBD |
I dropped "Tom to set a calendar reminder" and "Tom to be more careful." Both rely on individual vigilance, which is the failure mode that already happened twice. They are also hard to verify and put the burden on one person. If a reminder is wanted as a stopgap until automation lands, make it a shared team calendar event with a backup owner.
## Open Questions
- How was the outage detected? Did the uptime monitor fire, and if not, why? A failed TLS handshake may not have registered as a non-200 response, which would be worth its own note under Detection.
- Does 07:50 UTC match the actual expiry time? If so, the detection gap is worth stating.
- What were the 2025-09 action items, and what happened to them?
- Does the load balancer support automated certificate management, or will it need a different approach?
I can also rewrite the impact, timeline, or "Where we got lucky" sections in the same style if you'd like.