A future where agentic code equals negligence

Just saw the interview with Alex Bores, and I appreciate a lot of his ideas of regulation. Although I would argue some of his ideas should have been implemented decades ago because software security isn't a problem unique to AI.

One risk I see unique to generative AI is the use of it to write code. I see a poison pill that could create immense losses to enterprise software companies.

I am a cyber security software architect and I have been reviewing code for security in the Fortune 500 enterprise software space for over 10 years. I currently advise teams on how to deploy LLMs safely.

What I see with agentic code is blatant security errors. I try to point those out but I can't review every line of code, only earn the devs of their overall folly. It's usually by accident that I see some of the things that I see.

Lately I see teams using agents to do their security reviews, and when I spot check those reviews, I find serious things that the agents miss. My favorite thing is when teams generate security related architectural diagrams using the llm and they're completely not based on reality. This can really screw up a threat model session.

And so my prediction is it won't be long until there will be a civil lawsuit based on some kind of security breach where the use of agentic coding will be a central factor in the breach.

After that happens, anyone that is using agentic code today will be automatically a suspicion of having negligently written code. This will not only make the company liable for the damages of their mistakes but increase the penalties that are also applied.

I expect EU regulators to have a field day with this particular facet of new software development. Lawyers of plaintiffs that have been harmed by software errors will make this a core strategy of their discovery process. I won't even be surprised if one day we'll see a NIST directive that bans the practice all together for software purchased by the US federal government.

Companies won't be able to claim innocent human error because there were no humans in the loop. Everything will have been generated by an llm, pull request reviewed by llms, and security reviewed by llms. They'll have no choice but to settle with whatever parties are suing them for their error.

It could become the thing that is retroactively terrifying to enterprise customers. They may refuse to work with vendors that used agentic AI for code development. Enterprise software companies that did code use agentic AI in the past will have to go rip it out and re-implement.

Am I overreacting? Are there any cyber security professionals out there that feel like the code generated by llms is safer than code generated by developers? I mean I've trained a lot of developers to write safe code so they don't innately have skills in that area.