Post #2835618

In the latest episode of “behold the power of Mythos” from The Hacker News - Claude Mythos AI Finds 10,000 High-Severity Flaws in Widely Used Software I distilled it so you don’t have to. Of these vulnerabilities, 6,202 have been classified as high- or critical-severity flaws impacting more than 1,000 open-source projects. That 10,000 count didn’t even survive until paragraph 3. Subsequent analysis of these [6202] vulnerability candidates has identified that 1,726 are valid true positives. Ah fuck. 1726. But wait, a bad infographic has entered the ring! 23,019 potential vulnerability candidates Ok now we’re talking. 1,900 Reviewed by external security firms Wait, what? Why those? Why only those? 1726 confirmed positive You couldn’t even cherry pick the valid ones? 467 reported to maintainers Where did the other 1259 go? Maybe this other part of the flowchart will go better… 1,129 reported direct to maintainers by Anthropic, at their request (May contain false positives) 1129 + 467 = 1596 total reported to maintainers Most of them just spammed at open source maintainers. Right. Maybe Anthropic’s media release has the goods! 1,752 of those high- or critical-rated vulnerabilities have now been carefully assessed by one of six independent security research firms, or in a small number of cases by ourselves Slightly higher than the 1726, and I’m guessing the internal reviews were from scanning Claude’s code, but ok, whatever. Of these, 90.6% (1,587) have proved to be valid true positives, and 62.4% (1,094) were confirmed as either high- or critical-severity 10% of 10000 is still something, right? On maintainers’ request, we sometimes disclose bugs directly, without further assessment. We’ve now reported 1,129 such unvetted bugs, of which Mythos Preview estimated that 175 were high- or critical-severity. I’m sure those maintainers enjoyed that 16% accuracy rate based on Mythos’ own estimations. But wasn’t that 1129 the bulk of your reports? We estimate that we’ve disclosed 530 high- or critical-severity bugs to maintainers so far. There are a further 827 confirmed vulnerabilities (estimated as high- or critical-severity in the same manner) that we’re aiming to disclose as quickly as possible. 530 is only a third of the reports you made to maintainers… 65 of those have been given public advisories The infographic says 88. I’d ask if they were massaging their financials like they massaged 65 advisories, but we know they are. 23,019 potential vulnerability candidates of all severities, 65 advisories. If you printed the code out and drunkenly threw darts at it you’d probably hit the same level of accuracy.