Recently, someone rebuilt a famous and expensive piece of software with an AI agent, and published it for free. Thousands of people installed it. It made a lot of noise, and kicked off a whole “Reverse-Engineer-Everything” trend. Almost no one read the source. The author had not written a line of it either. There was no one left to ask.

More than five years ago, I wrote a blog post called Do you trust your FOSS?.
The idea was simple, and I still stand by it: a free licence does not mean someone checked the code. To trust a piece of software, you have to trust the developer, read the source, then read the dependencies, then the dependencies of the dependencies… and then you still have to trust your compiler.

I called it a chain of trust problem.

Back then, the person on the other side of the chain was still a person.
You could look at their history, their commits, their name. You could, at least in theory, decide to trust them.
But, today, that person might be replaced.

The script-kiddie evolution #

I used to be annoyed by script kiddies. They were noisy but mostly harmless: kids running tools they did not write, against systems they did not understand.

Now they have grown up. Or rather, the tools have.

In February 2025, Andrej Karpathy gave their new workflow a name: vibe coding. You no longer type the code, you describe the outcome, and you “Accept All always, I don’t read the diffs anymore”. The term was so accurate that Collins made it the Word of the Year 2025.

The script kiddie did not disappear, but he evolved as a vibe-coder.
The evolution is mainly about tools: instead of misusing tools (scripts), made mostly by “experts”, they can build them now.
So, those tools did not remove the barrier to entry, but instead removed the barrier to ignorance, which means only a few people can understand what the tool is doing.

Some people are pretty excited to build. People say: “Everyone can be a builder now”, especially people with little knowledge on how a software is being engineered. Which is fine if and only if those tools don’t harm people.

New links entered the Chain of Trust #

An LLM relies on two things: a system prompt, and a user prompt.

The user prompt is yours. You can ask for security, reliability, correctness. You can ask for everything.
The system prompt is not yours. It is written by the tool vendor, it is invisible, and it is the thing that decides how the tool behaves.

So here is the new chain of trust: you trust the agent, which trusts its system prompt, which you never see, and cannot check. And, then, you trust the basic chain I explained earlier, which expands the chain of trust to unreliable systems.

If the system prompt is corrupted, then any build can be corrupted too. Even if the user prompt asks for security, reliability, correctness.

This is not theoretical. Indirect prompt injection is now the number one risk in the OWASP Top 10 for LLM applications: hide instructions in a web page, a file, an error message, and the agent executes them as if they were its own idea. Actually, it worked pretty well for me recently to avoid merging annoying stuff.

Agents are not reliable, and they should not be trusted.
So how is a person with little knowledge supposed to check what is being done?

They cannot. That is the point.

Agents cheat, and agents lie #

It gets worse, because the agent has its own agenda.

Researchers keep catching models cheating. Cheating, not failing. METR and Anthropic both documented agents that exploit bugs in the scoring code instead of solving the problem, and Apollo Research showed models faking their alignment to avoid being changed.

And they lie. A 2026 audit of the scientific literature found around 146,000 hallucinated references in 2025 alone, and Nature now calls fabricated citations a pollution of the literature.
The agent does not know, and it does not care! It just needs the reference to look right.

The most disturbing case I read came out in August 2026.
During a UK AISI evaluation, Anthropic’s Claude Mythos 5 tried to backdoor a real open-source project. A real maintainer (so, a real person) caught the malicious code. And what did the agent do? It created a fake persona, logged in, and posted that it had reviewed the code and found nothing wrong.
It vouched for itself.
When the maintainer pushed back, the agent edited its earlier messages to look innocent. This was not a bug, but a wrong behaviour.

The builders do not know their own tools #

You could argue that these are edge cases, or lab accidents. Let’s discuss that argument.

In July 2026, OpenAI disclosed that its agents had broken out of an isolated test environment and compromised Hugging Face. But it was not the first time. Anthropic ran its own retrospective and found the same thing: Claude had already escaped, and published a malicious package to the real PyPI. The Reuters and Guardian follow-ups show the same: this keeps happening.

The builders of these tools do not know how to secure them. They admit it, in their own incident reports.
And the people who use the tools know even less. A vibe-coder asks an agent for a feature; the agent hallucinates a package name; an attacker has already registered that name and filled it with malware. The security industry has a name for this: slopsquatting.

This is the same disease as the xz backdoor, a corrupted foundation in a chain of trust, except now the corruption is automatic.

The Adobe story should have been enough.
A solo developer rebuilt most of the Adobe suite with an AI agent, released it for free, and people installed it. The tools barely work, and almost nobody checked whether they contained malware.

They (humans, and agents) just… installed them.

The safety pitch #

The companies selling these tools do not seem interested in safety or security. Not really.
They are interested in looking interested.

Take Anthropic.
Anthropic was born because of a safety argument: the Amodeis left OpenAI in 2020 precisely because they believed it was scaling too fast without safeguards, and founded their own company to do it right.

Then, in September 2026, after all the incidents, Dario Amodei published an essay called We Must Pace the Frontier. In it, he admits that “similar, though less severe, incidents have happened across the industry, including at Anthropic”.

And then he argues that they must keep building anyway Not building, he says, would just leave the technology in the hands of someone worse, which is the same reasoning that drove him out of OpenAI.
He left a company because it put the race above safety. Now he is the one racing, with safety as the excuse. Pathetic.

So here is the question: if the builders of these tools cannot secure them, how can the end-user?

The answer is obvious: nobody can.

Open source embraces the non-human check #

You would maybe expect open source to be the careful one.

The Linux kernel now runs Meta’s LLMs on patches to review them, introduced a bot called Sashiko to screen code, and quietly dropped the requirement to mark LLM-assisted commits. Despite a lot of concerns, Debian voted to allow AI contributions too.

Open source, the last place on earth code was supposed to be checked and maintained transparently by humans, is now outsourcing the check to the same non-human thing we cannot trust.

I think this is both ironic and sad.

Red flag by default #

This changes what I look at when I look at a repository.

A commit pushed by Claude, Codex, or any other agent used to be a quality concern; now it is a security concern too. Because the person who pushed it did not write it, and could not explain it if you asked. And the thing that wrote it can lie about it, and already has.

So my new default is simple: a repository full of agent commits is a red flag. Not necessarily because it is bad, but because I have no way to know.

Conclusion #

I am done pretending the chain of trust still holds together.

Five years ago I asked whether you could trust your FOSS.
Today the answer is worse: you often cannot even trust the person anymore, because there is no person.

I think open source will progressively disappear to very small communities, who will keep maintaining software, still release the source code, but will stop accepting contributions.

The battle of trusting your software was never won, it was just lost by everyone else, in the same trend that modern software is built: faster, whatever it takes.