[Discuss] Is open source more secure at the current level of AI?

[rrose at pobox.com (Randall Rose)]

Thanks.  From what I've seen of current AI capabilities -- not simply relying 
on public self-promotion by AI companies -- it is believable that there are 
current AI agents that can find exploitable bugs in code, and it is even more 
believable that they soon will be either fairly good or unusually good at 
doing so.

The reason why I said "arguably it creates more risk for open source" is:

(1) When a company develops software as closed source, there is less of an 
attack surface for hostile AI agents to look at the part of the code that is 
closed-source.  We in the FOSS community may not like this fact, but it is 
true, and it becomes more relevant if AI agents come to be better at finding 
exploitable vulnerabilities than most human coders.

(2) If a company that develops closed-source software checks its 
closed-source code for AI-detectable vulnerabilities, I am *not* confident 
that it does this for all open-source code it makes use of.  Does the company 
check for vulnerabilties in gcc's source code, for instance?  Perhaps few 
companies, if any, would, and that might extend to other open-source 
software.  Does the NSA want each big tech company to use AI to thoroughly 
scrutinize all the outside code it uses, or would the NSA sometimes prefer if 
big tech does not do a thorough job on this -- and do we want to rely on 
assuming the NSA is not having a negative effect here?  Some companies might 
check a *portion* of the open-source code they use, some might check none of 
the open-source code at all.

A few other points:

(3) Even if, contrary to what I suggested, we could be confident of the 
flattering conclusion that open source has no additional vulnerabilities 
relative to closed source, it would remain true that, even apart from 
comparison to closed source, open-source software may well now be more 
vulnerable than we thought due to AI.

(4) As I implied, I think it is worth knowing whether all open-source 
projects are doing enough against the risk of AI-detected vulnerabilities.  I 
share your hunch that at least some of them are not.  It is a test of how 
much the open-source movement is still energetic as opposed to 
stuck-in-the-mud to see if they deal with this well.  I think Git, for 
instance, should make it easier to see at a glance whether a project is 
effective in requiring AI testing of vulnerabilities in commits before 
release and how thorough that AI testing is. Distros' repositories should 
have similar policies.

I don't consider any of what I've said to be a proof that closed source is 
better now or in the future, and I'm undecided on that issue.  But I'm honest 
enough to see that open source may now have some new vulnerabilities to 
grapple with, which are worth facing.

On Thu, Apr 9, 2026, at 7:25 PM, Dan Ritter wrote:
> Randall Rose wrote: 
>> In the current state of the art, AI agents like Claude Mythos are good at 
>> finding exploitable bugs in code.  
>
> Objection: Anthropic says this. Pretty much everything Anthropic
> has ever said turns out to be overstated at best.
>
> (Counter-objection: Greg K-H says that LLM-discovered kernel
> bugs are now actually worth investigating.)
>
>> That affects open-source systems differently than closed-source systems, 
>> and arguably it creates more risk for open-source.
>
> I have had visibility into several companies' nominally
> closed-source software and SaaS products, and it is a mistake to
> think that the work that they do is significantly insulated from
> open-source work.
>
> The XKCD about the Internet relying on a small Jenga brick
> developed by one person in Nebraska? Approximately true for
> every large project. 
>
> Don't think of proprietary software as being different from open
> source. Think of proprietary software as being a layer of icing
> on top of a cake made mostly from open source components.
>
>> I suppose we are all biased in the pro-FOSS direction.  But these risks 
>> should be faced.  Are open-source projects doing enough against these 
>> risks?  Are there open-source projects that are so benighted that they 
>> don't even guard against risk (1)?
>
>
> No. Yes.
>
> -dsr-