Post #2649901

Here's a kind of 101 thing that a lot of people in the world of AI coding are missing, I think. Question: What are the implications of the fact, "All the tests pass?" Answer: It actually depends on how the code was written. Unfortunately, the salience of "all the tests pass" has a lot to do both with the strengths of agentic programming and the weaknesses. If code was written without knowledge of the tests, and then happens to pass the tests, I think I would then be a bit confident that the code is not just passing the tests by coincidence, but is actually correct in a deeper sense that makes it likely to pass future tests that haven't yet been written. When code is written with knowledge of the tests, or using the tests as a scaffold, then we have to be a bit more careful. Both humans and LLMs do this, but LLMs are better than humans at Monkeys Paw–style trickery, where you have something that is just totally wrong but nonetheless passes all the tests. So when a codebase is rewritten agentically, the fact that tests pass doesn't make me very confident, unless a lot of the tests were held to the side and not exposed to the agent. This is just basic experimental science (control!), it's hard to understand why this is not obvious to everyone. Unfortunately, having more tests exposed to the agent increases the quality of the results and the likelihood of correctness. So this is kind of a paradox — but it is a paradox that only applies to people whose method is to interact with the produced code only indirectly. It's easy to break the paradox if you treat the code as your own responsibility and only commit what you understand.