Post #2564098

2025-08-08 23:23 UTC

A contact just told me that my old "LLMs generate nonsense code" blog post from 2 years ago is now very outdated with GPT5 because it's so awesome and so helpful. So I asked him to give it a test for me, and asked it my favorite test question based on a use-case I had myself recently: Without adding third-party dependencies, how can I compress a Data stream with zstd in Swift on an iPhone? and here is the answer from ChatGPT 5: https://chatgpt.com/share/68968506-1834-8004-8390-d27f4a00f480 Very confident, very bold, even claims "Works on iOS 16+". Problem with that: Just like any other LLM I've tested that provided similar responses, it is - excuse my language but I need to use it - absolute horseshit. No version of any Apple SDK ever supported or supports ZSTD (see https://developer.apple.com/documentation/compression/compression_algorithm for a real piece of knowledge). It was never there. Not even in private code. Not even as a mention of "things we might do in the future" on some developer event. It fundamentally does not exist. It's completely made up nonsense. This concludes all the testing for GPT5 I have to do. If a tool is able to actively mislead me this easy, which potentially results in me wasting significant amounts of time in trying to make something work that is guaranteed to never work, it's a useless tool. I don't like collaborating with chronic liars who aren't able to openly point out knowledge gaps, so I'm also not interested in burning resources for a LLM that does the same.

Replies (1)

@denschub@mastodon.schub.social 2025-08-09 04:27

Because it already happened: if you read my post and you feel an urge to respond with something along the lines of "it's just a hallucination", "it's just a bug", "it will be better in ChatGPT 6", or anything even close into that direction, please stop. Read this post and the next post, think about them, and if you have a factual argument to respond to me, only then reply. Focus on my factual claims, not on some inaccuracies in my analogy because of course it's not 100% accurate, that's the nature of analogies. ChatGPT making up ZSTD compression in the Compression framework is not a bug. It's not even a weird edge-case. ChatGPT is doing exactly what it is designed to do. Let me try to explain. If we grossly oversimplify what an LLM is, it's "just a statistical model" that generates "language" based on a chain of "what is most likely to follow the previous phrase". "language" can be anything: it can be human language, a fictional language, but it also can be code or even genetic information. Any kind of textual thing that you can feed large amounts of into a model works. "Not having an answer" is not a possibility in this system - there's always "a most likely response", even if that makes no sense. ChatGPT inventing ZSTD compression in the Compression framework isn't due to a lack of training data. If you request an overview over all compression algorithms supported, it answers correctly with a comprehensive list that does not include ZSTD. So, if you want to anthropomorphize ChatGPT, you could say "it knows that ZSTD isn't supported", but that doesn't matter. LLMs do not possess the ability of logical thinking, deductive reasoning, or anything else. "It knows" that there are a bunch of compression algorithms available, the constants are all called COMPRESSION_[method], so there's a high likelihood of COMPRESSION_ZSTD to be the answer to a user asking for ZSTD compression in Swift. And so it generates that. The only way ChatGPT will stop spreading that nonsense is if there is a significant mass of humans talking online about the lack of ZSTD support. For example a bunch of StackOverflow questions asking "How do I do this?" and people responding "you don't, Apple doesn't support it, you have to use third-party libraries" - or if you have a bunch of white dudes working in tech complaining on social media about Apple not supporting ZSTD in the Compression Framework. My next post will be an attempt at comparing human thinking and LLMs generating text. As mentioned earlier, it's an analogy - and it's not going to be 100% accurate. If you want to reply, focus on the factual claims. If you only want to nit-pick my analogy, I have to assume you're not interested in productive argumentation.

Open ##2879950