This Pipeline Was 'Verified' for Two Days — It Never Actually Ran

These days, a lot of solo developers, small teams — even me — are using AI to squeeze work that used to take several people down to one or two.

I've been doing the same thing this year: writing code, testing, shipping. I used to think about hiring someone to share the load. Now an agent mostly picks it up. The efficiency gain is real.

But there's something I only figured out by tripping over it myself: the part of the work that got more efficient is exactly the part that used to be someone else catching your mistakes.

I run a small scheduled job that delivers new content into my personal knowledge base. The idea is simple: check every day for anything new that needs to go in, move it, log it — so I don't have to do it by hand.

It ran smoothly for a while. The log printed "delivered: 0/0" every day, looking perfectly fine. I relaxed for two days straight.

Then one Sunday, doing my usual check on the knowledge base, I hit the batch where there actually were 27 items to deliver — and for the first time, the job didn't sail through. It crashed. I stared at the error for a second, and my first thought was: "wait, wasn't this already working?"

Tracing it back, I found the problem: the check for "is there anything to deliver" ran before the code that read the config path. So whenever the pending count was zero, the function never reached the line that could actually fail. That line read a Windows-style path from config — and on a Linux container, expanduser just couldn't resolve it, silently returning an empty directory.

So "verified" never meant the pipeline was verified. It meant one narrow branch — "does nothing break when there's no data" — had been tested. And that branch was never going to break.

For two days the log kept saying "fine," because it had never once touched real input. That wasn't me being careless. The "fine" I was watching simply never passed through the part that could go wrong.

That got me thinking about something bigger. It's not just my one small automation setup — a lot of small teams are compressing toward "fewer people, more AI."

Growth used to mean hiring. Now the instinct is: don't hire if you don't have to. Work that used to take several people working together gets handed to one or two people plus a stack of AI agents. Things move faster. Fewer meetings, fewer rounds of back-and-forth before a decision gets made.

But in the old setup, even without anyone whose actual job was "catch the mistakes," there was usually a colleague who'd glance at the log in passing and say, "hey, this number looks off, are you sure?" That free layer of error-catching came from having enough people around — not from anyone's sense of duty.

The half of the work that got more efficient is exactly the half that used to be someone glancing over your shoulder.

Fewer people means that layer of protection is gone too. When something breaks, it's not that no one's there to stop it — it's that no one even knows there's something to stop.

Afterward I gave myself two hard rules. First: in a "degrade, don't block" design, the fallback path can't also be the loosest possible default — otherwise, the moment a dependency fails, it disguises itself as a normal result.

Second: verifying a data-moving pipeline means actually moving data through it once. A dry run that succeeds proves nothing. Edge cases like a zero count tend to be exactly the input that skips past the one line that could fail.

Next time I stand up something similar, I'm adding one more step: don't just check whether the program returns 200 — go back and confirm the data actually landed where it was supposed to.

I've only tested these two rules on my own small system, a few hundred records at most. Whether this "catch your own mistakes" approach holds up once the data or the team gets several times bigger, I haven't run into that yet. I honestly don't know.

Now, before I turn on any new automation pipeline, I force myself to run it against real data first — not just check that the log says "passed."

Whether that guardrail keeps working, I'm not sure. The real test is how long it takes next time something quietly slips through — two days, or two months. That's the thing I'll be watching.