Post #4156044

@glyph@mastodon.social Lot to process here, and sorry it's taken hours to reply. I consider @trailofbits@infosec.exchange to be the pre-eminent group doing software testing right now. These folks know their business. They have produced a really useful guide on setting up testing harnesses and fuzzers exactly as you're describing. They have also created some of the best LLM skills for leveraging that technology to find issues in source code. A lot of them are hooks into deterministic tools like Semgrep/CodeQL. So when they say we've entered a new era in which vuln finding through LLMs is producing something meaningful, I believe them. But that's appeal to authority. More pertinently, the means to perform the testing you describe are not novel; they're not secrets. And yet for some reason, it has eluded even mature projects. Why? It's worth noting that in many cases, the models are setting up testing suites. It's not like they're just inferring from the source. Some barrier has been broken here. Maybe other conditions could have led to the same outcome, but either way, we have to deal with the fallout. And fallout there will be. For each discovered vulnerability, we should expect a long tail of unpatched systems. That was always true, but at this volume, the required response to stay ahead of exploitability will likely outstrip most orgs' patching capacity. New models of defense are required. If you're asking whether the capacity to find new vulnerabilities is "worth it," I think it's hard to make the case. I deeply agree with you about the disgusting waste of resources on this misadventure; the folly is not in dispute. Nevertheless, the models we now have can perform the task of finding bugs and vulns in our code, and even the open weight models can do so at rates similar to commercial models, on relatively modest compute. As for lasting impact, a "durable residue" on the model weights themselves, I suspect it's too early to tell. I don't know how current efforts will inform the next iteration of model training, and if that even improves output. What is clear to me is that the processing of source code, perhaps the most structured, well-patterned text humans have ever created, is the niche LLMs fit best, and where any hope for revenue lies for an industry burning money like The Joker.