In most incidents, the technology has a path to resolution pretty quickly. What makes them hard is the chaos around them.

Nobody is sure who is in charge, and five people have SSH’d into the same server. The product manager is asking for updates in one Slack channel, the CEO is asking in another, and your on-call engineer is trying to answer both while also debugging.

I think of that chaos as a design failure, and it is one that firefighters solved fifty years ago.

#The fires that changed everything

On September 26, 1970, a fire broke out near Laguna Beach, California. Within days, 773 separate fires were burning across Southern California at the same time. By the time they were contained, 16 firefighters were dead, more than 700 structures had burned, and over 500,000 acres had been destroyed. The damage came to more than $230 million.

The investigation afterward found that the size of the fires wasn’t what overwhelmed the firefighters. It was that the agencies responding couldn’t work together.

Dozens of departments from different cities, counties, and federal jurisdictions showed up to fight the same fires with radios that couldn’t talk to each other. They used different terms for the same commands, so one agency’s “evacuation” meant something different to another. There was no unified command, which meant multiple incident commanders were giving conflicting orders to the same people. Some areas had twenty firefighters on a single burning structure while others had no one.

The firefighters were doing their jobs. The system around them was failing them.

#FIRESCOPE and the invention of ICS

The next year, the U.S. Forest Service, the California Division of Forestry, and several Southern California fire departments started a joint effort called FIRESCOPE (Firefighting Resources of California Organized for Potential Emergencies). Their job was to design a command system that would actually work at scale, across agency lines, in the middle of a major incident.

What they built over the next several years became the Incident Command System.

They designed it directly from what had failed in 1970. Radio interoperability was a technical problem that the right equipment could solve, but the coordination failures were structural. They came from ambiguity about who was in charge, what terms meant, what role each person was supposed to play, and how decisions would be made and communicated.

ICS dealt with that by standardizing anything that couldn’t be left to interpretation. There is one Incident Commander per incident. There is a fixed set of functional roles (Operations, Planning, Logistics, Finance/Administration) that mean the same thing no matter which agency you came from, and everyone uses common terminology. Span of control is defined, so no commander oversees more than seven people, and three to five is ideal. The structure also scales: small incidents only use what they need, and large incidents expand the same hierarchy without adding new concepts.

The system was piloted in California through the mid-1970s, refined through real incidents, and adopted as a national standard in the 1980s. By the 1990s it was how wildfire response worked across the American West.

#From wildfire to everywhere

The September 11 attacks exposed the same kinds of coordination failures at a national scale: federal agencies with overlapping jurisdictions, local and federal responders working under different command structures, and communication breakdowns that looked a lot like what FIRESCOPE had studied after 1970.

The Department of Homeland Security’s answer was to make ICS universal. In 2004, FEMA established the National Incident Management System, with ICS at its core, and Hurricane Katrina in 2005 showed again why it mattered. Emergency responders across the United States, from rural volunteer fire departments to the National Guard, now train in ICS. The principles that came out of the 1970 California fires now run hurricane response, mass casualty events, chemical spills, and search and rescue across the country.

ICS spread into other fields too. Hospitals use a version called HICS (the Hospital Incident Command System) for mass casualty events and internal disasters, utilities use it for grid restoration after major outages, FEMA’s Community Emergency Response Teams teach civilians the basics, and the Red Cross runs on it.

A framework built to keep firefighters from dying in California wildfires ended up as the model for emergency response in America.

#How it got into tech

This is the part I find most interesting, because most of the engineering world uses it without knowing where it came from.

Google’s Site Reliability Engineering book, published in 2016, has a chapter on managing incidents. It describes an “Incident Command” role, a single coordinator who manages communication and hands off the technical work, and it explicitly credits ICS as the model. The term traveled a lot further than that credit did.

In the years after, Atlassian, PagerDuty, AWS, and dozens of other companies published their own incident response frameworks. “Incident Commander,” “Comms Lead,” “Operations Lead,” and “Scribe” are now standard vocabulary in engineering orgs everywhere. The ideas behind them (unified command, defined roles, explicit handoffs, span of control) are ICS adapted for Slack and PagerDuty.

Most engineers using these roles have no idea they are running a process designed for fighting wildfires, but the Incident Commander slot on your on-call schedule traces straight back to Laguna Beach in 1970.

I think that history is worth knowing because once you understand why ICS was designed the way it was, its rules are a lot easier to follow and a lot harder to dismiss.

The people who built ICS were responding to deaths. The rules they wrote down came out of failure and were refined over decades of high-stakes use. “One Incident Commander” is a hard rule because organizations that treated it as a suggestion had people die.

The stakes in tech are lower, but the failures look the same, and the same fixes still work.

#In high school, I learned this without knowing it

I was part of the San Jose Fire Explorer Program, which was run through the fire department. We trained in structure search, hose work, and ladder operations with real drills at a real burn building.

We trained under ICS without me really understanding what ICS was. There was always a clear IC, roles were named and assigned before a drill started, communication went through set channels, and handoffs followed a format.

So when I started managing production incidents years later, ICS didn’t feel like a new framework I had to learn. It was the obvious answer to a problem I had already seen solved. The chaos I kept running into in engineering orgs, with five people on the same SSH session, parallel Slack threads, and nobody clearly owning anything, looked exactly like what FIRESCOPE analyzed in 1971.

#The four things that actually matter

If you only take a few things from this, take these:

One Incident Commander. There is always exactly one person with decision-making authority. It isn’t a committee or whoever speaks up first. It is one named person who is accountable.

Clear roles, so people know what they are supposed to do and, just as important, what they aren’t. Scope creep in an incident is as dangerous as scope creep in a project.

Defined communication channels, with one place for incident updates and one place for stakeholder communication instead of twelve parallel threads.

Explicit handoffs, so that when someone steps out, what they know doesn’t leave with them.

#The IC’s job is to direct, not do

This is the hardest part for engineers who are used to being the strongest technical person in the room. When something is broken, the instinct is to get your hands on the keyboard, pull up the logs, and start fixing it.

When you are the IC, you have to resist that. The moment you go heads-down on a technical problem, you have stopped being the IC, and nobody is running the incident anymore.

The IC’s job is to keep the big picture: assign who does what, make decisions when the team is stuck, manage communication, and set priorities. Someone else does the hands-on work.

#The command structure scales with the incident

For small incidents, the IC handles everything. As incidents grow, with more responders, more stakeholders, and a longer duration, the IC starts delegating:

• Communications Lead: owns all messaging, including internal updates, the status page, and customer-facing communication. They draft and the IC approves.
• Operations Lead: does the hands-on technical mitigation. Your strongest technical responder goes here and executes the plan.
• Planning Lead: tracks the timeline, documents decisions and next steps, and thinks two moves ahead. If the Ops Lead is focused on right now, the Planning Lead is focused on what comes next.
• Logistics Lead: makes sure everyone has what they need, like access, tooling, and staffing. If someone needs a database credential reset at 3am and can’t reach anyone, that is a logistics failure.

This maps directly to the ICS sections: Operations, Planning, Logistics, and Finance/Administration. The tech version usually drops Finance and combines a few functions, but it is the same structure firefighters have been using for fifty years.

#Severity is based on customer impact, not technical complexity

A control plane in a degraded state with no customer-facing impact isn’t a SEV1. A login page returning errors for 10% of users is a SEV1, even if the root cause is a two-line config change.

Severity is always about customer and business impact, and it can change as you learn more, so start where you are and adjust. The exact thresholds matter less than having them defined before an incident starts, because the worst time to debate severity is in the middle of one.

#Sustainable on-call is a reliability problem

We have all heard “we should take care of our engineers.” It is true, but it doesn’t always move the needle in planning conversations, so I like to frame it differently.

A tired engineer makes worse decisions. They miss things, communicate poorly, stay heads-down when they should be coordinating, and don’t hand off when they should. When they burn out completely, you lose knowledge that is almost impossible to replace. That makes responder health a reliability problem as much as a people problem.

No one should serve as Incident Commander for more than two hours without a rotation plan. Two hours of sustained, high-stakes coordination is exhausting. For any incident running longer than that, the IC should be planning their handoff before the two-hour mark, not once they are already burnt out.

ICS has known this since the 1970s. Incident command for extended operations includes mandatory rest cycles and shift rotation. Whoever set up your on-call rotation didn’t invent that idea. They inherited it from people who learned it fighting wildfires.

#The handoff is the reliability practice

Most teams treat the role handoff as an interruption to the real work, something to get through quickly so the next person can start fixing things. I think that is backwards.

A good handoff keeps what the team knows inside the incident as people cycle out. A bad handoff, or no handoff at all, is how important context gets lost and how incidents end up taking twice as long.

Every handoff should include:

• Current status and severity
• What’s been tried and what worked or didn’t
• Active mitigation steps
• Known risks and open questions
• Outstanding action items

Write it down and announce it in the incident channel instead of passing it along privately in a DM.

#Treat every incident like an incident

The ICS structure only works if you use it every time. SEV2 bug? Declare an IC and log it. SEV3? Same thing.

The process should feel boring on the small ones, because boring means it has become muscle memory. Then when the SEV1 hits at 2am, nobody is reading a runbook because everyone already knows their position.

Tabletop exercises and fire drills help, but nothing builds the reflex like real reps. The teams that handle major incidents well aren’t necessarily calmer under pressure. They have run the process on so many small incidents that the roles kick in automatically.

That is how ICS was meant to be trained, too. You practice on the small fires so the big ones don’t require improvising.

#Blameless postmortems, or you’re wasting the incident

Every incident that reaches SEV2 or higher should get a postmortem, with no exceptions.

It also has to actually be blameless. That means more than “we won’t fire anyone over this.” The analysis should focus on system failures and process gaps instead of individual performance, because the moment people think they will be blamed, they stop being honest and you lose the ability to learn from it.

A good postmortem produces specific follow-up items. “We should improve monitoring” isn’t one. “We need an alert on X metric with a threshold of Y by this date, owned by this person” is.

By the time you write the postmortem, the cost of the incident has already been paid. The downtime happened, the engineers got paged, and customers were affected. The postmortem is how you get something back for that. Organizations that take it seriously see fewer incidents, resolve them faster, and get more resilient over time, while the ones that skip it end up paying the same cost again next quarter.

The next time you assign an Incident Commander, write up a timeline, or hand off to the next shift, remember that you are using a system built to keep firefighters alive. It has been tested in conditions that make a database outage look pretty small, and it works.