Post by @lagrangeinterpolator@awful.systems

2026-09-19 15:42 UTC

Sorry for the X screenshot but XCancel is shut down from legal pressure. alt text Post by Evan Hubinger on Sep 8: > Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to. Quote tweet by Jasmine Sun on Sep 9: > I hear roughly 3 reasons people keep working at AI labs despite believing in ~10% extinction risk: > > 1) Techno-determinism: Someone will build ASI no matter what, and I can do it better & more safely than China/OpenAI/etc > > 2) Consequentialism: ASI might kill us, but it also might produce utopia/immortality/superabundance, so it’s a +EV bet > > 3) Self-interest: I am personally having fun & getting rich working on cool tech with friends. I don’t think about the macro stuff. > > Notably, none of this is “I’m hyping up the risk for marketing reasons.” People believe what they say, while being capable of a lot of internal dissonance / compartmentalization / self-justification. > > (Personally, I think the public is better served by AI researchers talking about & creating consensus for the specific safety solutions — regulatory or otherwise — they want vs. vagueposting about extinction.) Normally I’d sneer at their 10% chance of extinction claims, but now that I think about it, if you take their claims seriously at face value, you realize just how these people are monsters. We rightly condemn brutal dictators who commit genocide, but even their crimes would pale in comparison to anything with a 10% chance of exterminating all of humanity. These “reasons” are laughably indefensible given the supposed stakes here. The first is already terrible, and they just get worse. The only moral thing to do here is to immediately stop working for the AI companies, and do whatever you can to stop anyone else. It’s not okay to piss in a pool just because you suspect other people are also doing it. And what makes you think that somehow you are uniquely positioned to succeed in making it safe? This is the Sam-Bankman Fried defense (complete with a reference to “EV” (expected value)). When you decide to gamble with the lives of everyone in the entire world, did you first ask them if they are okay with you gambling with their lives? Sure, I might die, but if the ball lands on red, things will be great! How moral does this sound? What the actual fuck? Imagine if Hermann Göring defended himself at Nuremberg by saying, “At least I had fun and got rich and worked on ‘cool tech’ with friends.” Now, I think that these people sincerely believe that AI has a 10% chance of wiping out humanity, because their minds have been poisoned by Rationalist ideology. Their Rationality is so effective that after Bernie Sanders took their premises at face value, they were shocked when Bernie immediately came to the conclusion of locking these people up for 20 years. There are two specific aspects of Rationalism that cause this mysterious blind spot. First, they believe that superintelligence is inevitable, which conveniently removes responsibility from the people actively trying to bring it about. Second, they treat everyone outside the Rationalist sphere as NPCs who dutifully follow the directions on their script. The rule of law does not even occur to them as they treat the extinction of humanity as just another dinner table conversation starter.

Replies (8)

@Architeuthis@awful.systems 2026-09-19 15:55

What the actual fuck? Imagine if Hermann Göring defended himself at Nuremberg by saying, “At least I had fun and got rich and worked on ‘cool tech’ with friends.” Once the rockets are up, who cares where they come down, that’s not my department says Wernher von Braun.

Open ##4764501
@Evinceo@awful.systems 2026-09-19 16:20

At what point does doing something you think will result in the end of the world become attempted murder?

Open ##4764766
@o7___o7@awful.systems 2026-09-19 16:39

Ya see, we’re being charitable when we call them grifters instead of genocidaires.

Open ##4765076
@istewart@awful.systems 2026-09-19 18:52

(complete with a reference to “EV” (expected value)) In this day and age, if I see someone talking about major life decisions in terms of +/-EV, I turn on my heel and walk away rapidly. It was kind of cute when this jargon was entering wide use during the online poker bubble, but that was 20 years ago. What the actual fuck? This third category is the Patrick Bateman types who would have mainly been confined to finance at the turn of the century. They probably picked San Francisco over New York in part because they were worried the New Yorkers might make them wear a suit. There are two specific aspects of Rationalism that cause this mysterious blind spot. First, they believe that superintelligence is inevitable, which conveniently removes responsibility from the people actively trying to bring it about. Second, they treat everyone outside the Rationalist sphere as NPCs who dutifully follow the directions on their script. I’ve been thinking more about this lately, and my metamodel of the rationalist world model (if I may get away with such a pretense) is that they subtly believe that the future is already something crystallized, even if they don’t consciously realize they hold this belief. It is not really a probabilistic philosophy, no matter how much they assert that. Intelligence, to them, is a function that can more and more precisely apprehend the future as it escalates to greater values along the g-scale, and the singularity will manifest an intelligence that can do so with practically absolute precision (hence its prophesied ability to outcompete and replace humans). When I’m talking to people who haven’t encountered this stuff as closely, I half-jokingly describe the reflexive assignment of probability estimates as a devotional practice akin to prayer or meditation… but it really is! It’s a subtle invocation of the AGI/ASI god, in that I make this imprecise estimate now, to demonstrate my belief in a precisely quantifiable future; but come the singularity, God will relieve my feeble human brain from having to commit this lamentable imprecision. In that reckoning, those who are not even willing to venture an imprecise esimate must have no intelligence worth noting.

Open ##4767179
@BlueMonday1984@awful.systems 2026-09-19 21:00

Reading this, I get the feeling writers are gonna get a lot of mileage out of the rats and their ideology. These people have spent years attempting to create something they believe represents an existential threat to humanity (causing untold real-world harm in the process), and they’ve remained fully convinced that they’re the good guys throughout every single step of this.

Open ##4768719
@lurker@awful.systems 2026-09-19 21:39

All 3 of these reasons make them monsters. 1 and 2 are saviour complex bullshit and 3 is just pure greed and selfishness

Open ##4769110
@YourNetworkIsHaunted@awful.systems 2026-09-19 22:43

I find the techno-determinism specifically fascinating because it’s so self-perpetuating. Like, everyone believes they can do a better job than everyone else at building the right robot God, and every person working on it is, according to their own beliefs, accelerating the problem they’re worrying about. It speaks to how aggressively they see themselves as special, like they’re protagonists and everyone else is NPCs. I can’t think of any other way to justify the pattern of “this is a terrible idea that will kill us all except when I personally do it.” Which XKCD comic is it that has the subway full of people pitying everyone else for being mindless sheep? Mix that with the Arrested Development gif of Tobias contemplating an open marriage.

Open ##4770074
@maol@awful.systems 2026-09-20 00:49

#3 suggests that people in tech have gotten way wayyy too comfortable with saying “hey, I know that company’s evil, but it’s a really good job” about companies that make software for arms manufacturers or ICE or surveillance.

Open ##4771952