Human error is not a root cause

The short answer

When a post-incident review ends with “human error,” it records an action but leaves the cause unresolved. Someone selected the wrong environment, approved a risky change, missed an alert, or followed an outdated runbook. Each action matters. None explains why the production system allowed one ordinary mistake to become a customer-facing incident.

A better investigation looks at the conditions around the action. It examines the interface, permissions, workload, information available at the time, review process, deployment design, recovery path, and organisational pressure. That work produces changes the team can implement and test.

This does not remove accountability. It makes accountability useful.

Why “human error” ends the investigation too early

Imagine that an engineer runs a database command against production instead of staging. The incident report says the root cause was operator error and recommends more training.

That account leaves several questions unanswered:

• Why did staging and production use similar names or prompts?
• Why could one credential modify both environments?
• Why was there no preview or dry-run step?
• Why did the command affect such a large set of data?
• Why did recovery take longer than the change itself?
• Had the team seen a similar near miss before?
• Was the engineer working from an accurate runbook?
• What information did the engineer have when the decision was made?

Training may help when someone lacks knowledge. It has little value when the real problem is an ambiguous interface, excessive access, time pressure, or a workflow that depends on perfect attention.

The UK Health and Safety Executive advises incident investigators to identify why people failed instead of stopping at “operator error.” Its guidance also points to performance-influencing factors such as interface design, distraction, workload, time pressure, competence, and communication. Software operations has different consequences from industrial safety, but the investigation principle transfers well.

Error, mistake, and deliberate violation are different

Teams often place several kinds of behaviour under the same label. Separating them leads to better corrective work.

Slips and lapses

A slip happens when the intended action is correct but the execution is wrong. An engineer selects the production cluster from a dropdown while intending to select staging. A lapse is a missed step, such as forgetting to restore an alert after planned maintenance.

Slips and lapses happen during familiar work. Another training session rarely removes them. Clear environment markers, safer defaults, checklists, automation, and reversible actions are stronger controls.

Mistakes in judgment

A mistake happens when someone follows an incorrect plan while believing it is right. An on-call engineer may restart healthy replicas because the dashboard makes replication lag look like CPU saturation. The person executed the plan correctly. The mental model, information, or diagnostic support was wrong.

Useful actions may include better telemetry, clearer dashboards, decision trees, scenario training, and access to an experienced responder during escalation.

Violations and workarounds

A violation is a deliberate departure from a rule or procedure. That sounds more serious, but the reason still matters. People often bypass a control because the approved path is too slow, broken, or incompatible with the work they are expected to complete.

Investigate the trade-off the person believed they were making. A rule that teams bypass every week is weak control, even if the document containing it is clear.

Reckless or malicious behaviour requires a separate management and security response. A learning-focused incident review should not be used to ignore misconduct. Most operational incidents, however, involve people trying to complete legitimate work with imperfect tools and incomplete information.

Reconstruct what made sense at the time

Hindsight changes how an incident looks. After the timeline is complete, the missed signal appears obvious. During the incident, the responder may have faced ten alerts, partial telemetry, a recent deployment, an unreliable dashboard, and pressure to restore a revenue-generating service.

Ask what the person could see and reasonably understand at each point in the timeline:

• What information was available at that moment?
• Which signals were missing, delayed, or misleading?
• What goal was the person trying to achieve?
• What competing risks did they consider?
• Which procedure or prior experience shaped the decision?
• What made the chosen action appear safer or faster?
• How did workload, fatigue, handoffs, or time pressure affect the situation?

This reconstruction should use evidence where possible. Review deployment logs, chat history, alert timestamps, access records, runbook versions, dashboards, and command history. Avoid asking a responder to recreate every detail from memory several days later.

Google’s SRE guidance describes blameless postmortems as a way to identify contributing causes without indicting an individual or team. The practical value is better information. People are more likely to disclose uncertainty, confusing tools, near misses, and unsafe workarounds when the review is designed to learn.

Look beyond the final action

The last visible action before an outage attracts attention because it is easy to name. The conditions that prepared the incident may have existed for months.

Review at least five layers.

1. The immediate action

Record what changed and its direct technical effect. Keep the language factual. “The migration command removed the index from the primary table” is more useful than “the engineer made a careless change.”

2. The working context

Document workload, urgency, interruptions, handoffs, communication gaps, and the information available to the operator. A task performed during a major incident has a different context from the same task performed during a planned maintenance window.

3. The tool and interface

Check whether names, colours, prompts, defaults, and navigation made environments easy to confuse. Review whether destructive actions clearly displayed the target and likely impact. Confirm that automation reported what it was about to change before execution.

4. The technical controls

Review permissions, blast-radius limits, deployment stages, change validation, rollback, backups, feature flags, circuit breakers, and rate limits. One mistaken action should not automatically gain the maximum possible impact.

5. The organisational system

Look at staffing, ownership, incentives, deadlines, maintenance backlogs, and how action items are funded. A team may know about an unsafe deployment path for months but lack time or ownership to replace it.

An investigation that covers all five layers usually finds several contributing factors. That is normal for complex systems.

A practical investigation method

Use this sequence during the post-incident review:

1. Build the timeline from evidence. Start with logs, alerts, changes, chat records, and system events. Mark assumptions separately.
2. Describe actions without judgment. State what happened and what the system did in response.
3. Capture the information available at the time. Avoid adding facts that responders learned later.
4. Identify conditions that shaped each decision. Include tooling, workload, permissions, procedures, dependencies, and communication.
5. Find existing safeguards and explain why they did not contain the event. A control may have been missing, bypassed, badly configured, or too slow.
6. Write findings as conditions that can be changed. Each finding should point toward a system, process, or operating decision.
7. Assign action owners and dates. A postmortem without follow-through is only a document.
8. Verify that the change worked. Test the guardrail through a game day, staging exercise, access review, or controlled failure.

Do not force the review to produce one root cause. A single label may fit a simple component failure. Production incidents usually involve a chain of technical and organisational conditions.

Write findings that lead to work

Weak finding:

The engineer accidentally deployed the wrong configuration to production.

Useful finding:

The deployment tool listed staging and production next to each other, used the same confirmation text for both, and allowed the operator’s normal account to deploy directly to production.

The second finding supports specific actions: separate the environments visually, require the tool to display the target, limit production deployment access, and add an approval or progressive-delivery stage.

Another weak finding:

The on-call engineer missed the alert.

Useful finding:

During the incident, the responder received 47 alerts from six services. The customer-impact alert used the same priority and channel as low-impact warnings, and the escalation policy had no second responder after ten minutes.

That finding points to alert reduction, severity rules, routing, and escalation design. “Be more careful” does not.

Controls that reduce the effect of mistakes

Choose controls based on the failure path found in the review. Common options include:

• Separate credentials and accounts for production and non-production work.
• Time-bound privileged access with a clear reason and approval record.
• Dry-run and change-preview modes for infrastructure and database operations.
• Progressive delivery with health checks, pause conditions, and automatic rollback.
• Feature flags that separate deployment from user exposure.
• Limits on batch size, deletion scope, concurrency, and request rate.
• Two-person review for rare changes with a large blast radius.
• Clear environment names and persistent production warnings in operational tools.
• Backups whose restoration time has been tested against the service recovery objective.
• Runbooks exercised during game days and updated from responder feedback.
• Alerts grouped around customer impact instead of every low-level symptom.
• Automated checks for known unsafe combinations before a change starts.

More control is not always better. Extra approval steps can delay urgent recovery and encourage workarounds. Put stronger controls around high-impact, infrequent, and hard-to-reverse actions. Make common, low-risk actions fast and safe. AWS guidance on safe, hands-off deployments is a practical reference for reducing change risk through automation and progressive delivery.

Accountability still matters

Blameless does not mean consequence-free. Engineers remain responsible for following reasonable procedures, escalating uncertainty, reporting near misses, and participating honestly in reviews. Leaders remain responsible for staffing, priorities, access design, maintenance funding, and the quality of operational systems.

Accountability should match the part of the system each person can influence. Telling one operator to pay more attention does little when ten engineers share broad production credentials. Asking the platform owner to improve access boundaries and deployment validation creates an accountable change.

Keep performance management separate from the technical learning review when possible. Combining them discourages people from sharing the details needed to understand the incident.

Check whether the action item changed the risk

Closing a ticket does not prove that an incident is less likely to recur. Review the action against the original failure path.

Useful verification questions include:

• Can another person still make the same mistake with the same access?
• Does the system detect the mistake before customer impact?
• Is the blast radius smaller?
• Can the team reverse the change within the required recovery time?
• Has the new control been tested under realistic conditions?
• Did the change create a slower path that teams are likely to bypass?
• Who owns the control after the incident project closes?

Track repeat incidents and near misses by contributing condition, not only by service name. Repeated findings such as broad access, unclear ownership, noisy alerts, or untested rollback may reveal an organisation-wide reliability problem.

Sources and further reading

• Google SRE: Postmortem Culture, Learning from Failure
• UK Health and Safety Executive: Managing human failures
• AWS Builders’ Library: Automating safe, hands-off deployments