Post #1863811

2026-04-29 14:03 UTC

Something I believe but never managed to put into words is that software/RTL tests lose some of their value every time you fail them. If you keep trying you will eventually find a hole in the test that lets incorrect code pass. Is there a name for this? This is one of the things that makes me skeptical of LLM-generated code. Thrashing around until the tests pass is more likely to leave you with fundamental correctness issues than if you&#39;d thought about things up front, tried to make the code correct by construction, and then used the tests as a guide to areas you need to review more closely.

Replies (7)

@self@fedi.tali.network 2026-04-29 14:12

@wren6991 unsure if there's a name for this in particular (other than maybe underdetermination?), but it sounds a lot like a consideration in program synthesis (eg CEGIS). though LLM research has largely taken this field over as i understand it :(

Open ##2080114
@whitequark@social.treehouse.systems 2026-04-29 14:13

@wren6991 yeah

Open ##2080115
@abraxas3d@mastodon.radio 2026-04-29 14:19

@wren6991 really good question. Even without LLMs, TDD can give a false sense of security. Because passing tests do not guarantee functional code. It's worse when you succumb to the temptation to choose the test, rather than choose the code. For TDD to work, you need to have some steely nerves (discipline!). Now imagine doing TDD as a disembodied pattern matcher. No discipline. When it accidentally works it seems like magic. Because cathexis. (We will pack bond with toasters and plushies).

Open ##2080116
@tuban_muzuru@beige.party 2026-04-29 14:20

@wren6991 I wrote code for 40 years, C and Java. I came to the LLMs with no preconceptions. Here's the TL;DR for reasonable use of an LLM, doing things you'll like and respect. When you're beginning, you'll want to ask for code. Ask for spec instead. Spec is cheap. The LLM can shit out a mountain of code, never accept that sort of thing. But guided by a well constructed spec, it's writing good code. And that code is about as good as your own code. Given a lousy spec and a handwave of a go-get-'em, your code would be awful, too

Open ##2080117
@penguin42@mastodon.org.uk 2026-04-29 15:06

@wren6991 I think in some cases the LLMs are starting to understand the semantics of what they're asked for; it's not randomly writing stuff until it passes the test. Not as good as a careful attentive experienced programmer maybe; but better than many when well directed.

Open ##2080119
@truh@shark.community 2026-04-29 15:33

@wren6991 Goodhart's law

Open ##2080120
@amelia@gts.spoonstack.org 2026-04-29 15:35

@wren6991 I don't know of a specific name for it in test land, but in ops, it'd be called "alarm fatigue." The clincher isn't a test failure per se, it's the response to test failures. It's a very general psychological phenomenon, because it's a property of biological nervous systems - any repeated stimulus that evokes an "oh no" response will lead, over time, to one of two converged outcomes. One outcome is fairly high-cost and requires a supportive context, but is ideal: a careful, root-cause analysis is performed, the true origin of the failure is identified, and the entire operational context is subject to potential adjustment to systemically preclude further failures in an identical mode. Needless to say, most teams and orgs... aren't getting this right. Far more common is the other outcome: people brand a test (or an entire suite) as "flaky", the collective culture shifts towards ignoring the failure due to irritation, and maybe someone will get bored enough to "solve the problem." But now, that doesn't mean addressing the real issue, it means making the alarm shut up... and before you know it, you can have a full on choir of a dozen fire, tornado, and alien invasion sirens all going off at once and people'll sleep right through it and just keep punching the clock.

Open ##2080128