Post #4410011

@stuartl@mastodon.longlandclan.id.au @jimbob@aus.social @foone@digipres.club If I write source code and compile it, the output should be a lowering of the source code. Every behaviour in the running program should be the result of something in the source code if it were executed in the language’s abstract machine. Even things that are undefined behaviour are reflected, though the ‘anything might happen once this becomes reachable’ rule makes it hard to reason about real behaviour. But the really important property is that two compilers for the same language will generate programs that both reflect the execution in the language’s abstract machine. They may have wildly different performance characteristics (one of the problems we hit when FreeBSD switched from gcc to clang, for example, was that their inliners worked in opposite directions and so some code that had been refactored to be fast with GCC was slower with clang and became fast if you undid the refactoring) but their behaviour will be equivalent in the ways that the language specifies. This means keeping the source code makes sense. If you encounter a bug in the binary, there’s a chance it’s a compiler bug, and then a new version of the compiler might fix it. If not can reason about the behaviour in the source language. Debuggers let you map back from program behaviour to the source code. And then, when you fix the source code, you can recompile and be confident that, performance aside, the semantics of the rest of the program have not been altered. The language’s abstract-machine semantics serve as a set of constraints that tightly limit the behaviour of the output, given any input, LLM prompts do not have any of these properties. The same prompt, given to the same LLM, will not give the same output twice. But, worse, it may not even give functionally equivalent outputs. The first time you might get exactly what you wanted, the second time you might get nonsense. This is where the slot-machine analogy comes from and, if the slot machine gives you a jackpot, you absolutely want to save that state and not go back! But, even worse, there is no locality property. If you feed the same prompt to an LLM and control its entropy source to be the same, you can guarantee the same output. But if you change a small thing in the prompt, you may now be exploring a totally different area in the latent space. It may change everything in the output. This means that there’s no equivalent of a debugger, no way of going back and saying ‘generate the same, but fix that bug’. You have to use it in a different way and say ‘read this source code and fix this bug’. And that’s where you start to hit context-window size issues. The more tokens in the context window, the more likely that you’ll get complete gibberish from an LLM. Most vendors have a hard limit to prevent this but if you want them to ingest all of the code in a repo to be able to find and fix a bug, there’s a good chance that it won’t all fit. So none of the analogies really work except a drunk coworker.