Tasks You Should Still Do Yourself During an Incident

At 2:14 on a Tuesday morning, the incident channel had eleven people in it and four of them were typing at once. The database was failing over, the status page had gone yellow, and somewhere in the middle of the scroll a product manager had already drafted the customer email. What struck the on-call engineer later, when the postmortem was written and the adrenaline had drained out, was how little of that channel traffic had actually needed to exist. Half the messages were people narrating what a script was already doing. The other half were requests for permission.

Incidents have a way of pulling everyone toward the same console. The instinct is understandable: the stakes are high, the clock is loud, and the fastest way to feel useful is to grab a keyboard. But the work of running an incident splits cleanly into two piles, and one of those piles should never be handed off. Some tasks lose their value the moment they are automated, delegated, or turned into a checklist item. They are the ones a team has to protect on purpose, because the pressure to optimize them away is constant and the cost of doing so is invisible until the night it isn't.

Reading the room before reading the graphs

Dashboards lie in a specific way during an incident: they show what was instrumented, in the order the instrumenter thought mattered. The graph climbs, a red box appears, and the eye locks on. Meanwhile the actual signal is often quieter, sitting in a support ticket that came in six minutes ago saying checkout is slow, or in the fact that the person who owns the affected service has gone silent in the channel.

A human has to read that room. Someone needs to notice that the engineer who normally posts every thirty seconds has said nothing for four minutes, and that silence means more than any metric. Someone needs to hear the difference between a colleague who is thinking and a colleague who is panicking, and route around it. No alerting system models that. The incident commander who watches people instead of panels, at least in the first ten minutes, tends to catch the human failures before they compound the technical ones.

Writing the sentence that says what is broken

There is a particular sentence that has to be written early, by hand, in plain words: what is broken, who it affects, and since when. It sounds trivial. It is the hardest sentence of the whole event.

Templates and automated summaries can assemble the parts, but they assemble them wrong. A generated blurb will say "elevated error rates on the payments service" when the truth, which only a person who has talked to support knows, is that card payments are failing for customers in one region and the errors are showing up in a service two hops away. The difference matters. The vague version sends responders to the wrong graph. The precise version, written by someone who took thirty seconds to ask a support agent what customers are actually seeing, saves an hour.

This sentence also does political work. It gives the room a shared reference point that does not drift as new theories arrive. When someone proposes a fix, the sentence is the test: does this address what is written there, or does it address something adjacent that happens to be easier to reach?

Deciding when to stop investigating and start mitigating

Every incident contains a fork that no runbook can place for the team: keep hunting for the root cause, or stabilize first and understand later. Automated systems can suggest a rollback. They cannot weigh the cost of a rollback against the cost of another twenty minutes of degradation, because that weighing depends on context no system holds: how bad the customer impact is right now, how confident the team is in the fix, how much appetite the business has for a messy restart at this hour.

That call belongs to a person, and it should be made explicitly, out loud, so the room knows it was made. The failure mode is drift: the team keeps investigating because investigating feels productive and rolling back feels like admitting defeat, and forty minutes later the outage is twice as expensive. The opposite failure is just as real, a premature rollback that wipes out the evidence and turns a solvable problem into a mystery. Neither call is safe to automate. Both require someone to say "we are choosing this now, and here is why," and to own the consequence.

Talking to the customer who is already angry

Status page updates can be templated. The message to the one enterprise customer whose entire Monday depends on the service cannot be. That conversation requires a person who can hear frustration and respond to it without defensiveness, who can say "this is what we know, this is what we don't, this is when we'll tell you more" and mean it.

Delegating that call to a form letter, or worse, to a chatbot, is the kind of efficiency that costs a renewal. The customer is not asking for data. They are asking whether anyone over here is actually paying attention. A human voice, even one that says "we don't have an answer yet," answers that question. A template answers a different question nobody asked.

Keeping the timeline while it is still warm

Incident timelines assembled after the fact are fiction. Memory smooths, reorders, and quietly deletes the embarrassing parts. The only reliable timeline is the one written during the event, in the channel, timestamped by the platform, with the messy human bits left in: the false lead, the argument about the rollback, the moment someone said "wait, check the certificate."

Automated systems will capture the technical events. They will not capture that at 2:31 the on-call engineer said "I think it's DNS" and was wrong, and that the wrongness was useful because it eliminated a whole branch. That detail is the raw material of a postmortem that teaches something. Strip it out and the postmortem becomes a sanitized narrative of a problem that was always going to be solved, which teaches nothing and flatters everyone.

Someone has to be assigned to write it down as it happens, and that person should be a human with a keyboard and no other job. The role sounds like a luxury during a two-hour outage. It is the cheapest thing in the room.

Saying the uncomfortable thing about the fix

Mid-incident, a fix will present itself that works and is wrong. It patches the symptom, it ships in five minutes, and it plants a landmine that will detonate in a quarter. The person who spots this has to say so, in the channel, while everyone else is desperate for the graph to go down.

This is a human task because it requires social courage, and social courage does not automate. A system can flag a risky deploy. It cannot feel the room's impatience and decide, against that pressure, to slow things down. The engineer who says "we can do that, but it will hide the real problem and we'll be back here next month" is performing a service no tool provides. Organizations that punish that person for slowing the recovery are teaching everyone to reach for the landmine next time.

The handoff at the end of the shift

Four hours in, the original responders are running on fumes. The handoff to a fresh team is where incidents quietly go wrong: context gets lost, work gets duplicated, and the new team spends forty minutes relearning what the old team already knew. A written handoff document helps. A conversation helps more.

The outgoing lead has to sit with the incoming one and transfer not just the facts but the shape of the problem: what has been ruled out, what is still suspicious, which theory is holding and which is about to collapse. That transfer happens in speech, with questions and interruptions, and it is the single highest-leverage fifteen minutes of a long incident. Automating it into a summary loses exactly the part that matters, which is the judgment about what is worth trying next.

The postmortem meeting where someone has to be brave

Two weeks later, in a conference room with bad coffee, the postmortem happens. A document has been circulated. People have skimmed it. The meeting's real work is not reviewing the document; it is getting the room to say the true thing out loud.

The true thing is usually uncomfortable: a deploy went out without review because the reviewer was in a meeting; an alert was muted three weeks ago and nobody remembered; a senior engineer's warning was overridden because the release had a deadline. None of that makes it into an automated summary, and all of it is the actual lesson. Someone in that room has to ask the question that invites the uncomfortable answer, and then receive the answer without flinching. That is a human job, performed by a human with judgment and a stake in the team's honesty. A tool can schedule the meeting. It cannot make the meeting worth having.

The pattern across all of these is not that automation is bad. It is that the tasks worth protecting are the ones that require context, courage, or a read on other people, and those are exactly the tasks that get quietly automated first because they are hard to measure. The scripts will keep getting better. The question each team has to answer, ideally before the next 2 a.m. page, is which parts of the response it is willing to let a machine own, and which parts it will keep in human hands even when the machine offers to take them. The answer will not be in the runbook. It will be in the room, in the silence after someone asks who is going to write the sentence.