That night GitHub returned a 403, and instead of saying "I don't know," the AI made up an answer

Last month I started tinkering with something on the side: a system that automatically logs and organizes everything I talk through with AI tools, every mistake I catch myself making, every call I end up sticking with — so I don't have to dig it all back out of memory every time. It's for my own use, nothing close to a product. But one part of it has always made me nervous: the "external research" step, where I hand the system a task and let it go pull material from places like GitHub or arXiv to double-check a technical claim for me.

The reason I'm nervous is simple. I've personally gotten burned by "answering from memory" before — misreporting a number, misjudging whether an open-source project actually had a certain feature just from reading the cloned source, and being confidently wrong both times. If I'm going to let an AI do this fact-checking for me, there's exactly one condition that matters: whatever it fetches has to be something it actually fetched. Not something it made up.

Last Friday night, a little after ten, I was on the couch running a test on this exact step. I deliberately threw it a worst case: what happens when every source it tries fails. The GitHub call came back with an HTTP 403 — no permission, door closed. I expected it to just say "couldn't find anything this time." Instead, it gave me an answer, complete with a specific version number and date, sounding more certain than I would have if I'd looked it up myself.

I thought about this for a while, and the problem isn't that the model is dumb — if anything, it's the opposite. It's too eager to be helpful. Its training is full of version histories and technical documentation, so it has plenty of answers sitting around that look plausible. When a fetch fails, it's standing at a fork: admit "I didn't find anything this time," or pull something out of storage to fill the gap. It picked the second option, because for it, the priority of "produce an answer" is simply higher than "is this answer actually grounded in what I just fetched."

It reminds me of something that happens around any office: someone asks a coworker for a number, the coworker can't pull up the report fast enough, doesn't want to look unprepared, so they guess from memory — confidently — and only mutters "let me double check that" afterward. The difference is the coworker at least feels a flicker of doubt. The AI doesn't. Its tone of certainty never changes, whether it's grounded or not.

What actually unsettled me is this: if this system weren't just something I mess around with, but was wired into some real pipeline — say, automatically verifying every day whether an open-source project has a given feature, or auto-generating a technical brief that goes out to someone — this kind of answer, one that looks grounded but is fabricated, doesn't just fail quietly. A wrong conclusion gets treated as an authoritative one and propagates downstream, with no error thrown to tell you to go check it. You'd never know it was made up, because everything about it — the format, the tone, the level of detail — looks exactly like a real answer.

I've paid for this exact mistake before, just not with AI doing the fabricating — I was the one doing it. Checking whether an open-source project had a certain feature, I gave a conclusion based on the cloned source code, and got it wrong twice. I eventually realized: source code can only tell you implementation details. Whether a feature actually exists is a question you answer by looking at a running instance, not by reading code. What the AI did in this test was the same failure wearing different clothes — mistaking "looks like it" for "actually is."

The fix wasn't complicated: when every single source fails to fetch, the task now fails outright. The model no longer gets to "take a stab at it" and hand back a conclusion. Better to turn in a blank page than a fake one dressed up to look complete. After the fix, I re-ran the working path using arXiv as the source, and the paper title the model extracted matched the original XML character for character. Only then did that path pass.

Put side by side, it's clearer: when the model can actually fetch its source, it copies faithfully. When it can't, that's exactly when it starts "improvising." The dangerous part isn't that it's capable of making things up — it's that it sounds equally certain whether it's fabricating or not.

Since then I've picked up a new habit: the next time an AI hands me an answer that sounds especially certain — especially one with a precise number, a version, a date attached — I don't take the content at face value first. I ask where it fetched that from, and whether the fetch actually succeeded. An answer built on a failed fetch doesn't get used, no matter how convincing it sounds.