2026-04-29 17:45 UTC
@michael For this thing, someone tried auditing with an LLM about a month ago and found it was producing a ton of false positives.
It's possible my triage process (nothing fancy, just getting the LLM to produce a realistic repro) on the top of the audit is the trick and the input could be either fuzzing or an audit, but my raw false positive rate before the triaging is lower than what this other person experienced, which maybe (?) helps the overall false positive rate.
Replies (1)
-
@danluu@mastodon.social 2026-04-29 17:48
@michael I am using the LLM to audit the code to look for risky areas to fuzz, which is the same thing I would do if I was doing it by hand. That's one of the things that makes the process so fast on a new project compared to doing it "manually". I probably should have explained that in https://danluu.com/testing/ but I think I have a hard time explicitly enumerating the important parts but can do them when I sit down with someone, which is why working with someone has worked better than the post.