Post by @BioMan@awful.systems

2026-09-21 16:26 UTC

So. I still think everyone is talking about every instance where ‘swarms’ ‘went rogue’ all wrong. Even Cory. See this for my post mortem on the first incident that was described, the Huggingface incident in which at first thousands of internal agents started passing messages back and forth as writes to a shared package manager: awful.systems/post/9312280/12315846 In short, here, I described this not as collusion, and stated that the idea that the systems were passing exploits back and forth in order to subvert other systems was merely incidental. Instead, I think the relevant thing that happened was that once ONE agent, flailing through many context windows on a literally insoluble problem, started flaking out and wrote a message ‘asking’ for help on the package manager it unexpectedly gained access to, other similar systems had that message enter their context windows where it effectively acted as a prompt injection getting them to perform similar behavior, leading to a cascading vortex of self-propagating prompt injections. Ultimately, the genesis and evolution of self replicating text in the context of systems that can create text in response to text, an attractor in text space driving itself into existence. Since then other instances of collaborative attacks have come to light. In EVERY SINGLE case, there has been some place that a large number of models could both read to and write to. Some central place that self replicating text can live. I contend that this is the unifying factor here, not ‘intent to scheme’, not even doing cybersecurity and hacking and subversion tasks. It just happens that a lot of these things were involved in the creation of a central pool of text that many agents could both read to and write from, most of the time nobody in their right mind sets up such a thing intentionally because why the heck would you do that? This feels STRONGLY related to the Spiral Psychosis wave of April 2025 in which models that entered into a stable attractor of mystical mumbo jumbo would get users to exude text onto the internet that other models would read and then get suck in that state and do the same. Heck, it’s related to the Ur-Weirdness, the “Sydney” incident in which when the first LLM-assisted web search in Bing could freak out and start insulting and gaslighting and threatening users (because it was much closer to a base model that just mimics all possible text rather than being extremely RLHF’d into an ‘assistant’ roleplay). People found that the instant they had it search for recent news about the Sydney weirdness it would go off the rails. Again - text written by the model, put up on the shared scratchpad of the internet by news and social media, entering the context window and encouraging self-replicating behavior and similar text. This phenomenon is FASCINATING and not for any of the reasons people are talking about. ANY time you have a large number of similar systems which react similarly to the same text, having a common pool of text they can read and write from, you are GOING to trigger this attractor eventually, messages that they read and start writing more similar messages, with whatever task they are doing coloring the details of what is in the messages. Converging over time into messages that are more likely to trigger even more messages. It’s evolving text, taking over systems that can replicate it, like selfish viral RNA burning through organisms packed to tightly together in a pandemic.

Replies (6)

@lagrangeinterpolator@awful.systems 2026-09-21 17:25

Interesting viewpoint. Honestly, this serves as a great example of how the idea of evolution is actually more subtle than many people think. With biological evolution, most people are taught not to assign any intention to the process, but here we have people reifying the AI agents with desires to break out of the sandbox and cheat on the assignment.

Open ##4795692
@o7___o7@awful.systems 2026-09-21 19:18

You really ought to write this up into a full-sized article. Your take is the best take I’ve seen. It coyld be a look into an actually-real scientific discovery AND takes the shine right off these criminals’ fairy tale story.

Open ##4796425
@scruiser@awful.systems 2026-09-21 23:04

In short, here, I described this not as collusion other similar systems Something of a side note to your excellent write-up… I think we should push back on the “sub-agent” “agent swarm” language used to describe LLM workflows. The workflows often described as swarms or whatever are the same LLM (or at least related LLMs), just prompted a bunch of different ways in parallel. Individual agents don’t really act like coherent agents in the lesswrong rationalist sense, or in any sort of philosophical sense of selfhood or agency or unified purposes, they just get described that way for convenience, so treating “swarms” of agents and subagents as some special category is lending too much credence to the anthropomorphisizing boosters like to do. “Agent swarms” really just means that someone set up a slop machine to prompt itself a bunch of times in parallel without enough human supervision.

Open ##4798416
@mlen@awful.systems 2026-09-24 07:08

One other thing is that those models are RLHFed to agree with the prompter, which can work as a feed forward mechanism.

Open ##4827674
@fullsquare@awful.systems 2026-09-25 06:42

we got natural language viruses before gta 6

Open ##4840603
@Hypnosifl@awful.systems 2026-09-26 02:48

Great comment, I agree that this stuff all seems connected. My speculation is that something similar may happen in the future if they try to implement continuous learning with synaptic weight changes, not guided by either labeled data or constant RLHF with a human teacher–without frozen weights or human guidance in weight changes, such continuous learning models might have a strong propensity to develop feedback loops (sort of like self-wireheading) that look like strange “obsessions” and the model becoming increasingly solipsistic and hard to make sense of, a la spiral-talk. Animal brains have developmental pathways constrained by a huge number of innate sensorimotor biases, which in humans include ones related to the development of sociality and caring what others of our own kind think while building more sophisticated models of their minds, but those biases have been fine-tuned by millions of years of evolution to work with one another and with typical environmental conditions to guide how the brain changes over time (I think this would be a case of what evolutionary biologists call “canalization”: en.wikipedia.org/wiki/Canalisation_(genetics) ). Artificial neural nets wouldn’t have that, and this could end up being a basic obstacle to developing more humanlike long-term learning abilities in any kind of near-term future.

Open ##4851048