A production incident runbook gives an on-call engineer a reliable path from the first alert to recovery and follow-up. This guide provides a copyable template, severity and escalation matrix, first-15-minutes checklist, communication timeline, service-health evidence checklist, mitigation decision tree, and review process. It also shows how to keep access, secrets, ownership, and operational links current so the runbook remains useful during a stressful event.
Overview
An incident runbook should reduce decision friction without trying to predict every failure. It tells responders what to verify, who can help, which actions are safe to take, and when to escalate. The best runbooks are short enough to use under pressure but specific enough to prevent avoidable mistakes.
Use this production incident checklist as a starting point for customer-facing outages, security-relevant events, failed deployments, data integrity concerns, and severe degradation. Adapt the details to your services, alerting system, cloud infrastructure, and access model. A runbook should never depend on one person's memory or on undocumented permissions.
For security and access management, document the approved break-glass path, the people authorized to use it, and the audit trail that must be retained. Do not place passwords, tokens, private keys, or other secrets in the runbook. Link to the approved secrets management system instead. The secrets management tools and rotation playbook can help teams review how credentials are stored and rotated.
Copyable incident runbook template
Incident ID:
Start time (UTC):
Incident commander:
Technical lead:
Communications lead:
Scribe:
Affected service or component:
Customer or internal impact:
Current severity:
Status page or communication channel:
Primary dashboard:
Relevant logs and traces:
Change or deployment window:
Known workaround:
Next update time:
Immediate objective:
- Stabilize the service, protect data, and establish the next verified fact.
Actions taken:
- [Time] [Person] [Action] [Result]
Decisions and rationale:
- [Time] [Decision] [Owner] [Reason]
Follow-up owner:
- [Action] [Owner] [Due date]
What to track
1. Incident identity, ownership, and scope
Record the incident ID, start time, affected services, environment, region, and the alert or report that initiated the response. Assign roles early, even if one person temporarily holds several roles:
- Incident commander: Sets priorities, coordinates the response, and makes or delegates decisions.
- Technical lead: Directs diagnosis, mitigation, and validation.
- Communications lead: Owns internal updates, customer-facing messages, and stakeholder coordination.
- Scribe: Maintains the timeline, commands or changes performed, evidence, and decisions.
Write down who is acting in each role. This prevents several engineers from investigating the same symptom while nobody coordinates the overall response.
2. Severity and escalation
Severity should describe impact and urgency, not the seniority of the person reporting the problem. A simple matrix can be adjusted to match your service-level objectives and risk tolerance:
| Severity | Typical condition | Response and escalation |
|---|---|---|
| SEV1 | Broad outage, critical data risk, or a rapidly worsening customer impact | Declare immediately, page the primary escalation group, assign an incident commander, and provide frequent updates |
| SEV2 | Material degradation, important feature failure, or a limited but sustained outage | Page the owning team, set a response lead, involve the next escalation level if progress is blocked |
| SEV3 | Limited impact, workaround available, or a non-urgent reliability problem | Assign an owner, document the issue, and resolve through the normal support or engineering workflow |
Define escalation triggers in advance. Examples include no working hypothesis after a set period, evidence of data loss or unauthorized access, failure of the rollback, exhaustion of the primary responder group, or an impact that crosses a documented severity threshold. Keep escalation contacts in a maintained directory rather than embedding personal details that quickly become obsolete.
3. Service-health evidence
Capture evidence before changing the system when doing so is safe. Track customer symptoms and technical signals separately:
- Request success rate, error rate, latency, and traffic volume.
- Health by region, availability zone, tenant, endpoint, or deployment version.
- Recent releases, configuration changes, infrastructure changes, feature flags, and scheduled jobs.
- Container restarts, pod scheduling failures, resource saturation, queue depth, and dependency health.
- Relevant application logs, traces, database metrics, and cloud-provider events.
- Authentication failures, unusual permission changes, secret access, or other security signals when access or identity may be involved.
Link directly to the dashboards, log searches, deployment records, and service ownership pages responders need. If Kubernetes is involved, resource requests, limits, scheduling, and autoscaling may explain the symptom; use the Kubernetes resource requests and limits troubleshooting guide and the HPA guide as supporting references.
4. The first 15 minutes
- 0–3 minutes: Acknowledge the alert, open an incident record, confirm the affected service, and assign an initial severity.
- 3–5 minutes: Name the incident commander and technical lead. Start a shared timeline and response channel.
- 5–10 minutes: Confirm impact using at least two useful signals, check recent changes, and identify whether the issue is isolated or broad.
- 10–15 minutes: Select a safe investigative or mitigating action, page additional owners if needed, and publish the first status update when customer impact is confirmed.
Avoid making several unrelated changes at once. Every action should have an owner, a purpose, a risk assessment, and a way to verify its result.
Cadence and checkpoints
A runbook is operational documentation, not a one-time project. Use a predictable review cadence and update it whenever the system changes.
During the incident
Set a communication rhythm based on severity and impact. A practical timeline is:
- Initial update: Confirm what is known, what is affected, and that investigation is underway. Do not speculate about a root cause.
- Progress updates: State the current impact, completed actions, working hypothesis, next action, and next update time.
- Mitigation update: Explain what changed, what evidence indicates improvement, and what remains under observation.
- Resolution update: Confirm that key health signals have recovered and state whether monitoring continues.
Keep customer-facing language factual and concise. Internal channels can hold technical detail, but access should follow least-privilege expectations. Do not copy sensitive logs, credentials, personal data, or security investigation details into broad channels.
After the incident
Within the team's normal review window, complete a timeline and separate observed facts from assumptions. Record the trigger, detection quality, decisions, mitigation, recovery validation, and any delayed or missing escalation. The purpose is to improve systems and processes, not to assign blame.
For deployment-related incidents, compare the response with the team's CI/CD release checklist. If a build or rollback contributed to the event, use the CI/CD pipeline troubleshooting guide to identify gaps that should become automated checks.
Quarterly runbook review
At least quarterly, verify every operational dependency: alert destinations, service owners, escalation mappings, dashboard links, log queries, deployment and rollback commands, dependency diagrams, access groups, break-glass procedures, and secret references. Ask an engineer who did not write the runbook to follow it in a staging exercise or game day. Confusion during the exercise is useful evidence that the instructions need revision.
How to interpret changes
Incident signals rarely point to a cause by themselves. Interpret changes by comparing timing, scope, and behavior.
- Impact begins immediately after a release or configuration change: Compare versions and consider pausing further rollout. Roll back only when the rollback is understood, authorized, and safer than the current state.
- Error rate rises while traffic is stable: Examine application health, dependencies, authentication, database capacity, and recent code paths.
- Latency rises before errors: Check saturation, queue depth, downstream calls, connection pools, and resource limits.
- Only one region, tenant, or version is affected: Compare the affected slice with a healthy slice to narrow the change boundary.
- Access or secret-related signals appear: Stop treating the event as a routine availability incident. Restrict unnecessary access, preserve relevant evidence, involve the approved security owner, and follow the organization's credential rotation process.
Use a mitigation decision tree: first protect people and data, then contain the blast radius, then restore the most important service path, and finally investigate the underlying cause. If a change is reversible and its effect can be measured, it may be a reasonable mitigation. If it is irreversible, affects data, expands access, or cannot be validated, escalate before proceeding.
Recovery is not proven by one green dashboard. Validate the customer-facing path, background processing, dependent services, error rates, latency, capacity, and security controls. Continue observation long enough to detect recurrence.
When to revisit
Revisit this runbook on a quarterly schedule and after every major incident, on-call handoff, new service launch, game day, significant architecture change, identity or access change, migration, or deployment-process change. Review it immediately when an alert routes to the wrong team, a dashboard link fails, a responder lacks required permission, a rollback command is outdated, or an escalation contact cannot be reached.
Use this post-incident follow-up template to turn observations into owned work:
What happened:
What was the customer or business impact:
How was the incident detected:
What actions reduced impact:
What actions increased risk or caused confusion:
Which alerts, dashboards, or permissions were missing:
Which secrets, credentials, or access paths require review:
What should be automated:
Action items:
- [Action] [Owner] [Due date] [Verification method]
Runbook changes:
- [Section or link] [Change] [Reviewer] [Review date]
Next exercise or checkpoint:
Before closing the review, assign an owner and verification method to each important action. “Improve monitoring” is not a sufficient task; “add an alert for sustained queue growth and test it in the next game day” is actionable. Refresh the runbook after the change is deployed, then ask the on-call team to use the updated version.
To put this into practice, copy the template, fill in the service-specific owners and links, confirm the least-privilege access path, and run a short tabletop exercise. Schedule the next quarterly review now. A runbook earns its value when responders can follow it without guessing, even when the system, team, or incident is unfamiliar.