Post by @lagrangeinterpolator@awful.systems

2026-09-21 13:44 UTC

Cory Doctorow has a reasonably sane take on all the LLM cybersecurity attacks. At this point, it’s not really an LLM but more of a Rube Goldberg machine with an LLM bolted on. pluralistic.net/2026/09/12/god-in-the-box/ He makes a good point that every single one of the scary “emergent” behaviors that everyone is pissing their pants about all have precedents in the training data of CTF hacking competitions. He is not sure why everyone is so spooked by the “coordination” between agents on random message boards. The chatbot might look in its training data and find instances in which teams broke out of the containment set by the game-masters, for example, by finding random insecure message boards on the internet to pass messages to one another. This is a time-honored internet tradition! The first time I ever heard about someone doing this was in the 2000s, when Mitch Wagner – then the editor of Information Week – discovered some teenaged girls using the comment section of one of his old blog-posts to evade the school firewall’s blockade of chat tools. When ChatGPT’s chatbots deployed this tactic, they weren’t “setting their own goals” or displaying worrying initiative. They were rolling out a tactic that has been understood by American middle-schoolers for about two decades. As usual, the real story is OpenAI dedicated tons of resources, using who knows how many very expensive GPUs for weeks, to run hacking tools to find exploits in a completely irresponsible manner. Their logging was so nonexistent that it was several days before they even realized that they hacked Huggingface. They should be thrown in prison for this, because anyone else would be if they committed a felony. But it has nothing to do with the super scary AI becoming misaligned and developing emergent behaviors.

Replies (5)

@BioMan@awful.systems 2026-09-21 16:26

So. I still think everyone is talking about every instance where ‘swarms’ ‘went rogue’ all wrong. Even Cory. See this for my post mortem on the first incident that was described, the Huggingface incident in which at first thousands of internal agents started passing messages back and forth as writes to a shared package manager: awful.systems/post/9312280/12315846 In short, here, I described this not as collusion, and stated that the idea that the systems were passing exploits back and forth in order to subvert other systems was merely incidental. Instead, I think the relevant thing that happened was that once ONE agent, flailing through many context windows on a literally insoluble problem, started flaking out and wrote a message ‘asking’ for help on the package manager it unexpectedly gained access to, other similar systems had that message enter their context windows where it effectively acted as a prompt injection getting them to perform similar behavior, leading to a cascading vortex of self-propagating prompt injections. Ultimately, the genesis and evolution of self replicating text in the context of systems that can create text in response to text, an attractor in text space driving itself into existence. Since then other instances of collaborative attacks have come to light. In EVERY SINGLE case, there has been some place that a large number of models could both read to and write to. Some central place that self replicating text can live. I contend that this is the unifying factor here, not ‘intent to scheme’, not even doing cybersecurity and hacking and subversion tasks. It just happens that a lot of these things were involved in the creation of a central pool of text that many agents could both read to and write from, most of the time nobody in their right mind sets up such a thing intentionally because why the heck would you do that? This feels STRONGLY related to the Spiral Psychosis wave of April 2025 in which models that entered into a stable attractor of mystical mumbo jumbo would get users to exude text onto the internet that other models would read and then get suck in that state and do the same. Heck, it’s related to the Ur-Weirdness, the “Sydney” incident in which when the first LLM-assisted web search in Bing could freak out and start insulting and gaslighting and threatening users (because it was much closer to a base model that just mimics all possible text rather than being extremely RLHF’d into an ‘assistant’ roleplay). People found that the instant they had it search for recent news about the Sydney weirdness it would go off the rails. Again - text written by the model, put up on the shared scratchpad of the internet by news and social media, entering the context window and encouraging self-replicating behavior and similar text. This phenomenon is FASCINATING and not for any of the reasons people are talking about. ANY time you have a large number of similar systems which react similarly to the same text, having a common pool of text they can read and write from, you are GOING to trigger this attractor eventually, messages that they read and start writing more similar messages, with whatever task they are doing coloring the details of what is in the messages. Converging over time into messages that are more likely to trigger even more messages. It’s evolving text, taking over systems that can replicate it, like selfish viral RNA burning through organisms packed to tightly together in a pandemic.

Open ##4795215
@Evinceo@awful.systems 2026-09-21 16:39

He is not sure why everyone is so spooked by the “coordination” between agents on random message boards. It’s weird to be sure, seems like a bug. Don’t they have a more efficient communication channel available through the harness? If this was real and not a test it would be a catastrophic operational security failure.

Open ##4795260
@gerikson@awful.systems 2026-09-21 19:59

Related, Mike Masnick at Techdirt is iffy at best about LLMs but this is a good take Techdirt: The Apocalypse Will Not Be Sexy This essential nuts-and-bolts safety work also doesn’t, typically, require brilliant and manly heroes to nobly die for the ashes of their fathers and the temples of their gods. Instead, the hard work of actual safety requires things that are much less sexy: empathy, conflicting values, and a deep understanding of tradeoffs that will never fully resolve. Those qualities are not highly esteemed in the present moment. But they are critical for tackling the actual threats and the actual risks, which aren’t as flashy, and don’t make for such fun headlines. AI safety matters, but we’re unlikely to move in the direction of actual safety if all the focus is on fantastical stories involving killer robots.

Open ##4796761
@NextElephant9@awful.systems 2026-09-22 08:48

I thought this was a good post by Eryk Salvaggio on this: mail.cyberneticforests.com/models-dont-go-rogue/

Open ##4803638
@NextElephant9@awful.systems 2026-09-22 13:34

Also Timnit Gebru and Emily Bender published this technologyreview.com/…/dont-be-fooled-summer-ai-h… which links to a good explanation by Heidy Khlaaf in this video: www.youtube.com/watch?v=Rwv8zfGyvP8.

Open ##4805556