2026-04-29 06:05 UTC
Replies (4)
-
@danluu@mastodon.social 2026-04-29 06:11
I'm not sure what to do about this. With LLMs, it's easier than ever to test things and the time/cost it takes to get a piece of software to a particular quality bar has gone way down and, simultaneously, software quality seems to be getting much worse. I've sat down with a couple people and wrote a fuzzer with them (pre-LLM) in 15-30 minutes and converted them for life, but I haven't figured out a framing that actually works when I write it as a blog post.
-
@jpf@mastodon.social 2026-04-29 06:08
@danluu Timothy Fitz had a very similar experience after writing this blog post: http://timothyfitz.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/ - e.g. "Sure it works for IMVU, but it'll never work for me"
-
@raphael@mastodon.sdf.org 2026-04-29 09:27
@danluu what does fuzzing/random testing look like for business logic? It feels like a lot of the examples are of the “thing crashes” variety but with CRUD apps where we tend to be reading data, transforming a bit, then throwing it back into the DB, it feels like the failure states are way less obvious Have you had success in that space?
-
@nesevis@hachyderm.io 2026-04-29 13:34
@danluu Fascinating read, but there are a fair few dead links in there now (partition testing, IBM’s SixthSense). I’m working on a PBT library that combines random sampling with pairwise/tway covering arrays of boundary/«interesting» values and optionally partition testing through specified «directions» (predicates that must be hit and are AND’ed with the property when reducing, each of which get a biased generator). It’s an incredibly interesting space, and I think a large part of the reason random testing hasn’t been adopted more widely is due to a combination of ergonomics and performance. Hypothesis is a great example of getting a lot of the ergonomics right.