Post #4187552

2026-07-29 07:34 UTC

Okay, we have a new contender for Most AI Thing to Ever Happen 1) July 25th: someone messes around with an LLM and posts a proof of the Collatz conjecture that does, in fact, verify in the theorem prover. (The AI use is not disclosed on the github page) https://github.com/xrchz/CollatzLean 2) July 26th: several serious bugs are posted in the theorem provers, that in principle could allow a false statement to be "proven" true. They're serious, yes, but no need for panic, because you're not going to blunder into accidentally exploiting the bugs while writing a proof, probably. https://github.com/leanprover/lean-kernel-arena/pull/81 3) July 28th: someone who was right to be very skeptical of the Collatz proof, and had the expertise to study it with a fine-toothed comb, discovered it was exploiting a bug https://github.com/leanprover/lean4/issues/14576 4) The "proof" turns out to be exploiting multiple similar but distinct bugs to pass different solver variants! 5) the human who posted the proof acknowledges the AI use and claims they did not knowingly point it towards the bugs it exploited. https://leanprover.zulipchat.com/#narrow/channel/270676-lean4/topic/Counterexample.20to.20the.20Lean.20Conjecture.20.28Soundness.20Bug.29/near/613135216 Note that the proof was posted shortly before the related bug reports were posted. It is an open question if the AI found people discussing the bugs shortly before they were formally posted and "decided" to exploit them, if the AI "knew about it" as a learned strategy from the training stage (putting every single "proof" it's ever made and ever will make into profound doubt), or if it's recently been repeatedly blundering into it by sheer stupidity and that's how people noticed the bug at about the same time. Theorem provers aren't magic, and have bugs just like all other programs. They are tools to help us double-check our reasoning. When you skip the reasoning and ask an AI to "prove" something for you that's over your head, you're entering an adversarial pact with the monkey-pawed Devil of Customer Satisfaction. my initial source for investigating this myself: https://lipn.info/@mevenlennonbertrand/116997927457012577

Replies (6)

@0xabad1dea@infosec.exchange 2026-07-29 10:35

(sorry, I somehow flubbed my keyboard and posted this half-finished, so I deleted it) an important thing to understand here is that software like theorem provers were designed around the assumption that the user cares about getting a correct answer, and wouldn't intentionally sabotage themselves by leaning into obscure, buggy behavior but AIs are the proverbial "nasal demons" of programming lore at their most manifest: you said you needed this theorem proved; it will "prove" the theorem at any cost, because the AI itself doesn't know or care about the beauty of mathematical truth, it cares about you clicking the button that indicates you were satisfied with its output today and are likely to pay for more tokens in the future The metaphor of "nasal demons" was invented to teach new programmers about the dire meaning of "undefined behavior": asking the compiler to do something it doesn't have exhaustively clear rules for, so it will do... something. Do nothing at all for one cycle, perhaps. Delete the hard drive, perhaps. Summon demons to pour out of your nose, perhaps. This is of course an absurdly impossible example to get you to remember it, but the point is: if your program causes undefined behavior, it is entirely possible something you don't expect and won't enjoy will happen. LLMs are nasal demons. They do not have exhaustively clear rules for anything: they wing it, all of it, all the time. Asking it to interface with a complex system with hidden bugs and find a miracle solution will cause results that you, as the entity that actually cares about the result being correct, will not enjoy. :neodog_devil:

Open ##4193744
@calx@infosec.exchange 2026-07-29 11:12

@0xabad1dea@infosec.exchange "[Theorem provers] may not be ready for such problems."

Open ##4194980
@ahltorp@mastodon.nu 2026-07-29 13:25

@0xabad1dea@infosec.exchange @mevenlennonbertrand@lipn.info Seems very related to p-hacking. The primary tool of LLMs is deception: the whole idea of chatbots is to mimic human interaction so that Grice’s Cooperative Principle can be exploited, which they have been doing long before LLMs. When interacting with another machine it’s a bit different: ”brute force” is a better way to describe it, but it’s still a deception based approach. Any route to winning is acceptable, much like the behaviour of the people who fund it.

Open ##4199529
@whitequark@social.treehouse.systems 2026-07-29 07:50

@0xabad1dea@infosec.exchange ohhhh... if I'm reading the repro right, this is really similar in cause to a well known OCaml quirk

Open ##4200228
@mattblaze@federate.social 2026-07-29 07:58

@0xabad1dea@infosec.exchange @mevenlennonbertrand@lipn.info oh wow. This is either terrible or wonderful and I have no idea which.

Open ##4209848
@dzwiedziu@mastodon.social 2026-07-29 08:00

@0xabad1dea@infosec.exchange > we have a new contender for Most AI Thing to Ever Happen Do we have enough contenders start a league for this award or a simple knockout tournament will suffice for this month? :thonking:

Open ##4215216