incidents

Post-incident reviews (PIRs) and operational runbooks for the pgmac homelab Kubernetes infrastructure.

Published at https://incidents.pgmac.net.au/

Contents

• Incidents — PIRs documenting what went wrong, why, and how it was fixed
• Runbooks — Step-by-step recovery procedures for known failure modes
• Templates — PIR and runbook templates, with section-by-section guidance
• Decisions — ADRs covering how the site itself works

Local Development

Requires mise and Python 3.13.

mise run install   # create venv and install dependencies
mise run serve     # serve at http://localhost:8000 with live reload
mise run build     # build static site
mise run build-strict  # strict build (matches CI)

Skills

Live incidents are triaged and PIRs/runbooks are authored with Claude Code / OpenCode skills from pgmac-net/claude-plugins.

Skill	Purpose
/start-incident	Work a live incident: open a tracking issue immediately, triage via Nagios/Slack, match a runbook, gate every mutating action on confirmation — hands off to /create-pir on resolution
/create-pir	Generate a post-incident review: Infinite How's analysis, runbook evaluation, GitHub Issues, commit + PR. Reads a /start-incident tracking issue as its primary source when one exists
/pickup-ticket	Work a GitHub Issue end-to-end: read, grill, plan, implement, PR, document
/grilling	Stress-test plans and decisions one question at a time
/domain-modeling	Build and sharpen domain model: glossary, ADRs, terminology

Why public?

Why do I make all of this public? A few reasons:

1. I enjoy working in public.
2. I'm challenged by working in public. It forces me to produce work I am prepared to show.
3. Helps future-me by reducing the number of assumptions and assumed knowledge I sometimes(/usually) leave.
4. Hopefully it helps other people not only pick up some SRE knowledge, but also inspire some to work in public and share their experiences, too.

Contributing

New PIR

1. Name: YYYY-MM-DD-brief-description.md
2. Location: src/incidents/ — the nav discovers it automatically, newest-first
3. Frontmatter must carry title, date, severity (P1–P4), duration, and normally resolution and impact. main.py fails the build on a missing field or an unknown severity. Never use status: — it is reserved by Material; use resolution:
4. Add a row to the top of src/incidents/index.md (newest-first), with severity as a badge: <span class="sev sev--p2">P2</span>
5. Follow src/doc-templates/pir-template.md — it documents the full frontmatter contract. Do not write the Date/Duration/Severity/Status block into the body; it is rendered
6. Use the /create-pir skill to automate the full flow — see Skills above

New Runbook

1. Name: <service>-<failure-description>.md
2. Location: src/runbooks/
3. Add a row to src/runbooks/index.md
4. Follow src/doc-templates/runbook-template.md (simple or multi-mode pattern)
5. Cross-link from the PIR that documented the failure
6. The /create-pir skill evaluates runbook needs during PIR generation — see Skills above

CI

• validate.yml — strict build of both configs on every PR
• deploy.yml — builds and deploys to GitHub Pages on merge to main

Build configuration

mkdocs-base.yml holds everything shared. mkdocs.yml (internal) and incidents-mkdoc.yml (public) are INHERIT plus their own site_url/site_dir, so a theme change made once applies to both. Neither declares a nav — pages are discovered from src/ and ordered by .nav.yml files.

main.py generates the home page's recent-incidents list and section counts from PIR frontmatter, and validates that frontmatter at build time. Macro rendering is opt-in (render_macros: true), because two documents quote Kubernetes errors containing literal {{ }} that Jinja would otherwise try to evaluate.

Rationale for these choices is recorded in src/decisions.md.