Post #2831779
2026-04-30 15:35 UTC
@peterhuene@hachyderm.io yup! like this post argues, each time you fail a test and make adjustments, you use up some of its descriptive power. LLMs do this at *scale* making the tests useless.
https://types.pl/@wren6991/116488366700176901
Replies (1)
-
@peterhuene@hachyderm.io 2026-04-30 15:40
@esoterra@hachyderm.io Absolutely. It's such frustrating nonsense. Anthropic's C compiler passed a bunch of the GCC test suites, but had such laughably bad code generation that made some operations ~200000X slower. Writing *performance* tests are really difficult and rarely reflected in something like a regression test suite, even assuming the LLM passed the regression test the "correct" way. This PostgreSQL port will most likely perform like Microsoft Access running on a toaster. And how does one dive into that much slop to improve it post-facto? It would likely be faster to have never used an LLM in the first place to get to a place of _quality_.