Opinionated conventions for handling production incidents, on-call and blameless postmortems.
Turns request counts into SLO answers: SLI, error budget for a 28d or 30d window, budget consumed and remaining, burn rates over 5m to 3d, multiwindow burn-rate alerts (page at 14.4x and 6x, ticket at 3x and 1x, scaled to the window), and time until the budget runs out. Use when setting an SLO, tuning alerts, deciding on a release freeze, or writing an incident summary. Includes a stdlib Python calculator with --self-test.
Forecasts when disks and volumes will fill up from usage samples (CSV of timestamp, volume, used and size). Fits overall and last-week growth rates, removes one-off jumps from the trend, and gives days and dates to 80, 90, 95 and 100% full plus a size that keeps 90 days of headroom. Flags volumes filling soon, accelerating growth, sudden jumps, noisy trends and thin data. Includes a stdlib Python forecaster with --self-test.
Turns noisy application, server, and access logs into a ranked list of error patterns with counts, first and last seen, spikes, and patterns that are new versus a known-good baseline, then separates root causes from symptoms and writes a short incident triage report. Includes a tested stdlib Python log clusterer.
Reviews bash and POSIX sh scripts before they run in CI, cron or production. Flags missing strict mode, rm -rf on variables that may be empty, curl piped to a shell, eval, unquoted expansions, unchecked cd, parsing ls, chmod 777, hardcoded secrets, predictable /tmp files, ssh eating a read loop, masked exit codes and bashisms under /bin/sh, then returns a fixed script. Includes a stdlib Python linter with --self-test.
Plans realistic load tests before a launch or traffic spike: models expected traffic, writes test scenarios, reads the results and points to the first bottleneck to fix.
After an incident it collects the alert timeline, chat messages, deploys and graphs into a blameless postmortem draft with impact, root cause, and follow-up actions.
Reviews database migrations in pull requests for locking, data loss and rollback risk before they reach production.