Why Is the Problem Still There After All the Firefighting?

The server's temporary directory reaches 95 percent capacity, and an alert fires.

An engineer logs in, searches through the filesystem, deletes some logs, and watches usage fall. The chat channel goes quiet. A few days later, the same alert returns, and the team repeats the same steps.

Anyone who has worked around software systems recognizes the scene. Restoring the service has to come first. But if the response ends with deleting the logs every time, the team is not becoming more efficient. It is only becoming more practiced at fighting the same fire.

Firefighting is highly visible.

The alert has a timestamp. So do the first reply, the recovery, and the messages from whoever stayed up until midnight. All of that fits neatly into a weekly report. Removing an obsolete job, adding a capacity guardrail, or automating a repeated recovery step is quieter. It can be hard to prove which future incident never happened because of that work.

The measurement system then begins to shape people's choices. Clear the red light in front of you first. Investigate why it keeps returning when things are less busy.

But “less busy” rarely arrives. A problem that has not been designed out of the system keeps manufacturing the next emergency.

The easy mistake is to treat response time and problem quality as the same metric. A ten-minute recovery shows that the on-call engineer reacted quickly. If the same class of incident returns every week, it also shows that the system is still producing trouble on schedule.

The first number is easy to count. The second requires a team to connect events over time. When only the first is rewarded, everyone becomes better at those ten minutes and finds it harder to spend half a day preventing the next alert.

Fast firefighting proves that this incident ended quickly. It does not prove that the number of problems is falling.

Google's SRE book calls this kind of work toil: operational work that is manual, repetitive, automatable, interrupt-driven, and leaves no lasting improvement in the system.

Google gives its SRE teams a specific target. Operational work should stay below 50 percent of an engineer's time, while at least half should be reserved for engineering that reduces future toil or improves the service.

That number is not a universal rule for every company. Its value is that it forces a team to keep two kinds of busyness on separate ledgers.

One kind restores normal service today. The other makes an alert less likely to fire next week.

The first is necessary. If the second disappears for too long, the team becomes trapped by its own diligence. As the service grows, the manual workload grows with it. New hires inherit a larger set of buttons to press and have no time to improve the system.

A one-off failure may deserve nothing more than a manual response. The warning sign is repetition: the same directory fills again, the same service needs another restart, or the same data needs another manual correction. The second occurrence is still an incident. By the third, the response has become part of the process design.

I would rather look at three numbers.

First, how long did this incident take to recover? Second, how many times did the same class of incident return in the next reporting period? Third, how many manual steps are still required to handle it?

Return to the temporary directory at 95 percent. Deleting logs improves the first number. Disabling useless logs and setting a rotation limit reduces the second. Turning capacity checks and cleanup into an auditable automated job finally lowers the third.

These three numbers do not require every incident to become a major engineering project. They force the team to say whether it merely restored service or left a durable improvement behind. Both matter, but the speed of the first cannot substitute for the result of the second.

Looking only at the first number creates more heroes. Looking at all three shows whether the system is actually improving.

Real efficiency is not fighting the same fire more impressively. It is preventing one recurrence.

I am vulnerable to the satisfaction of “fixed immediately,” too.

The alert clears, the ticket closes, and another box gets checked. The feedback is fast and certain. Spending half a day tracing a root cause, changing a monitor, or removing a piece of historical baggage can look as though nothing happened at all.

That is the uncomfortable reversal. A team may want to solve the underlying problem, yet the organization has reserved applause for firefighting and left no visible place for the fire that never started.

If someone fights the same fire beautifully every week, the problem is not only that person. The system is using their sense of responsibility as a patch. The more reliable that patch becomes, the easier it is to ignore the hole underneath it.

The next red light still needs someone to respond.

After it goes dark, however, the real result is whether that same alert appears for a third time.