Post #2597108

@UlrikeHahn@fediscience.org I just shared this with my lab in our "use of AI" discussion channel, right after this article: https://gowers.wordpress.com/2026/05/08/a-recent-experience-with-chatgpt-5-5-pro/ Relating to our recent discussion, I'm starting to think "unreasonable reasoning" is actually a pretty good descriptor. If you point then in the right direction and keep them in an abstract domain, they can chain together ideas and do useful work. But at the slightest provocation, they will also happily go down an elaborate garden path of utter fantasy, unintentionally confabulating whatever one might expect to see. The juxtaposition is wild, but I think it shows that what we casually call "reasoning" is a few things mixed together. LLMs have some, but not all, of these faculties, which explains why we have such a hard time assessing them and come to such radically different conclusions. We need to stop conflating these things in our own minds to better appreciate what LLMs can and can't do.