YourNetworkIsHaunted

This is the most charitable explanation and I think the most likely. Largely because it requires everyone involved to be even less competent for even longer and simply not notice the glaring holes in what they claim is the most important security feature in human history because, presumably, the chatbot kept telling them it was fine.

That sounds like a great way to harm some cylinders.

I think you’re discounting the most likely explanation: that it discovered and communicated the purest transcendent truth and therefore there is no need for further updates. Or maybe nobody was actually checking it for anything and the bot running it has been patiently waiting for more prompts that haven’t come because nobody gave a shit.

I don’t know. My wife swears up and down that the furby she had as a kid was afraid of the dark and would call out for help if it was held upside-down. We can’t find any recollection of this behavior or documentation elsewhere, so it may have also been actually haunted. Being a child and therefore a monster, her response was to leave it in the dark closet hanging upside-down until it accepted that this was its life now.

This is fascinating. Ignoring, for a moment, the assertion that AI agents are sentient beings somehow, what struck me about both part 1 and 2 is the bafflingly sincere excitement about turning a fairly straightforward software project into “building a new civilization”, and the assumption that this whole process will be best repeated for every software project. Let’s even ignore the abysmal economics of needing to effectively invent the universe in order to add apple pie to your crappy flash game. It’s absolutely wild and possibly malpractice for an engineer to be this excited about the needless expansion of scope and complexity of a project. It really showcases the kind of mindset it takes to be this breathlessly excited about the sheer number of different components that are being worked on in order to eventually somehow make a game happen. This is the dopamine rush of seeing the agents do something combined with an abdication of any responsibility for exactly what they do.

Anyone who doubts that authenticity is dead should reference this paragraph.

I think there’s some truth to that, but it also bears acknowledging that the market demand existed already. The number of people who can get a lot of benefit from IT productivity tools is much higher than the number of people who are willing to do the work of maintaining and customizing their own stuff. From an organizational perspective, it often ends up making more sense to directly contract with a known and trusted third party instead of developing the native capabilities, and at that point the tools and skills themselves start to specialize in ways to provide IT as a service at scale. Economies of scale lend themselves to efficiency but at the cost of resiliency and concentrating power, and tech companies chasing the line ever upwards have been abusing the absolute hell out of that, but while they have been altering the deal (pray they do not alter it any further) their bullshit is not the only reason why the deal exists in the first place.

Spread that around the recruiters and make sure they can tell candidates that a noted racist shit heel was appalled by how not racist they were.

There was a video essay I saw a while back that looked at how the system gamifies and conditions the creator experience. They effectively take every dirty trick you’ve heard of Uber using to get drivers to keep driving and use it to push creators to keep posting and driving engagement, even as the actual value of the platform has dropped to the point where basically every video over 3 minutes ends in a patreon pitch. Textbook enshitttification.

I think corporate consolidation is a huge part of it, but I think a lot of software and tech falls into the same trap that a lot of infrastructure projects do: even if everyone uses it, nobody wants to think about it. That general apathy (until it fails catastrophically in a bridge collapse or a major service outage) limits the avenues for feedback on whether software is actually good or not in the same way that people don’t have a lot of feedback on whether a water main is being routed in a reasonable way. It also creates room for grifters who specialize in obfuscating exactly what you’re paying for.

So Hank Green is apparently stepping back from his shit for a while after admitting to using AI for research and scriptwriting and acknowledging an unhealthy relationship with the dopamine hit he got from using it. I think this makes him one of the first high-profile users to acknowledge the problem? Which can’t be a good sign for the bubble.

I mean, it’s less dumb than trying to do it in orbit, but it’s still profoundly dumb.

So Satyress has been getting the headlines lately for their horrifying ThreeHalves monstrosity. But what I find beautiful is that it appears to have been designed first to be functional and second to be easily disabled in the event of a robot uprising. I wish I was kidding.

Onboarding at my new job means I am now “Certified AI Ready” whatever that means. Thankfully other than having copilot available it looks like my team isn’t having it pushed too hard yet, and I only had to interrupt the training video once to rant about how ridiculous their history was for like 20 minutes (working from home has a lot of advantages).

What kills me is that there are so many obvious ways to be less wasteful about this. Nondestructive book scanners exist. They’re expensive but it’s not like AI companies are averse to throwing money down a fucking hole. Even if that’s not an option, it’s possible to rebind the pages and return the books to the market. And regardless of what happens to the physical books, the scans they create could be archived in a format that could be available as a digital library rather than just being fed into the statistical meat grinder to keep the trough topped off with slop. Even if they’ve got to fight with copyright holders it would cost them basically nothing to leave the option open and given the PR battle they’ve been losing it feels like doing so would be incredibly obvious. Hell, even Google Books was able to navigate this in a way that was less cartoonishly evil than this because they could point to their digitization effort as a public good in ways that Anthropic here just fucking can’t.

how productive the field will become Let’s just ship all these people to Macroeconomica

I mean, I think the most charitable interpretation of the goal is effectively the one from Hitchhikers Guide to the Galaxy, where all the AI systems including the simple automatic doors are fully sentient and have simply been given the ability to feel either euphoric joy or simple boredom, and then they have a chip that activates happy mode when they fulfill whatever purpose they’ve been given (“It was my pleasure to open for you, and it is with deep satisfaction that I close again behind you.”) At that point, the argument goes, not making them do the thing would be cruelly consigning them to eternal boredom. This is also the same series that proposed solving the ethics of meat consumption by genetically engineering a cow that actively wants to be killed and eaten by restaurant patrons more than anything else.

If anything that makes for an even stronger case. Gangstalking is already a known pattern of delusional behavior and if a service pitched in part for its ability to provide relief is actually reinforcing that…

I don’t think anyone here is likely to disagree that current LLM systems can’t coherently be said to have some kind of internal experience and the design and functioning of those systems seems to fundamentally preclude the kinds of conscious experience that we want to assign moral agency to. But the key to understanding this as a moral issue imo is that the people designing and building these things seem like they really want to create something that ought to merit that kind of moral consideration. For all that the humanizing language of agency, intelligence, welfare, etc. is in reality part of the grift, I think there’s a strong catch-22 to be argued that if it’s not part of the grift then slavery is pretty implicit in the value proposition of these companies.

The actual worst man, and also just fucking sad. Imagine knowing that not only did you peak in high school, but that you peaked specifically in the moment when you fucked up and lost the state championship. Like, you can just smell the empty beer cans and tears piled up around this guy’s posts.

This was a fun one. It feels like there’s a shared affinity between Rats and traditional conservatives for confusing length and volume of output with actual depth or quantity of thoughts, and I deeply appreciate your willingness to wade through it all. Now that I think about it it would be difficult to write 500 pages saying basically nothing and not repeatedly blatantly contradict myself. Also, release the Foucault tangent, you cowards!

Ooh. I love these. I’m “I don’t think Sam Altman or most of his social circle are good people and I don’t want to give them more wealth and power.”

Vibe Construction

I feel like there’s a version of this that has a slider for how aggressively you’re willing to sacrifice readability, and you could probably get pretty decent results on the scale of 2x to 2.5x just using different encodings of the same basic glyph.

I know we’re all familiar with the metaphor of LLMs as roulette wheel, but I think there’s some real craft to turning that into Russian roulette.

I have definitely done this. Written half an essay in the awful.systems comment box, realized I don’t know what the fuck I’m saying anymore, and deleted it. I like to tell myself that forcing myself to realize I was being incoherent is enough to justify the costs. I guess if I acknowledge that I don’t rely on my online footprint to put food on the table the way the proper blogonauts (we’re still doing blogs, right?) do I can understand the incentives against doing that though.

I think this particular pattern is another consequence of long-termism in the water supply. It’s easy to disconnect yourself from humanity as a concept when you’re in a space that often assumes that you and everyone you could ever have a personal relationship with are already long dead. The fact that this makes it easy to ignore the problems that people today are having in front of your face is purely coincidental.

It’s been a long time since I’ve seen basic questions about functionality answered by “something something blockchain” but that appears to be the answer here.

You know, I think there’s an argument to be made for death threats against tech oligarchs being a crude form of direct action towards redistribution.

I can’t decide if it’s fitting or ironic that this group of men who spends collectively one week a year in spaces they don’t either literally own or otherwise totally control has such a hard time understanding concerns the rest of us have about privacy.

I should probably be grateful that this sent me down the road of learning about pet trusts, the actual legal vehicle for leaving large sums of money to an animal. But that’s less because it’s relevant to the point here and more because I’m a fucking nerd. Notably the money is controlled by a trustee who is empowered to pay for the animal’s food, healthcare, etc. I imagine that an enterprising trustee in the most famous cases could make some very compelling arguments about the necessity of maintaining, say, a big house for the cat to roam around in that they and their family would coincidentally live in as well, but I don’t care enough to look into it in-depth. If the animal dies before the trust runs out of money, then the remainder is distributed according to either the trust instructions or applicable state laws (in the US, at least). Of course the cases of people putting millions into pet trusts are usually less about the pet trust and more about spiting the other presumptive heirs, a sort of disinterestance with a more spiteful edge. To the actual argument here, however, all I can say is that nobody without a large estate has ever made the allegedly absurd decision to dedicate their estate to the care and keeping of their pets. Every single case where that has happened has been because of a member of the supposedly-enlightened moneyed classes being fucking weird and having too much money.

I remember having a lot of fun with Roborosewater where they could generate something that matched the format of a Magic: the Gathering card perfectly and sometimes even something playable within the rules but it was always nonsensical in some ridiculous way. There’s a couple streams from the LoadingReadyRun gang drafting a cube from the resulting chaos that are a hoot.

If you look at literally page 1 you’ll find the first footnote pointing at another chapter of the same goddamn book and now I’m going to go weep tears of impotent rage.

By forcing the question of whether to act before debating how, the signatories appear to be clearing ground for legislative and regulatory work that has not yet arrived. So it’s a pointless attempt to send up a “take me seriously” flare? Also I think this discourse as always misses the distinction between economic displacement caused by AI and economic displacement justified by AI. Hiring managers seem to be anticipating AI capabilities that do not exist yet, and the disruption to labor is well beyond what the current AI systems are actually able to replace. I think instead we’re seeing a combination of business rubes buying the hype and the consequences of bad incentives that make headcount reduction look really good on quarterly financials while in a lot of areas not having an immediate negative impact on capabilities or production. Especially when it comes to IT infrastructure, for example, the fact that you’re not paying someone goes immediately on the financial statements, but the fact that your infrastructure is less reliable and crashes more often has a huge experiential impact on your customers and employees but it takes time for that to cash out into something that impacts the numbers.

Honestly If we want to use the hallucinatron the same way we use hallucinogens (minus the “transcend reality and pierce the veil of the machine elf realm” part) I have no real problem with that. Scrambling and remixing the contents of the world in ways that might prompt actual minds to create new connections and interesting art seems to fit. I don’t think there’s enough of a market for that to support scaling to any meaningful degree and so have less problems from an enabling/infrastructure perspective. I mean the chemical alternatives let you transcend reality and pierce the veil of the machine elf realm. AI can’t do that.

This is an important point. If we treat Scott in particular as though he’s speaking in good faith then his arguments are nonsensical. This is not because he’s a fool but because he isn’t speaking in good faith, and his actual worldview appears to be much more amenable to outright fascism and actual totalitarianism than he would suggest here.

On one hand I agree that in a lot of ways our current tech surveillance regime looks like the kind of orwellian nightmare that people are raising red flags about. I’m admittedly less terrified by this than I am frustrated at how quickly and easily these systems get assembled when it comes to protecting shareholders from potentially unfavorable market conditions like “people using the software they purchased and installed on their computer in a way we didn’t expect”. Like, the US can’t get it’s shit together to do basically anything about domestic terrorism to the point where elementary schools are having domestic terror drills in order to feel like they’re doing literally anything to acknowledge the problem, but God forbid OpenAI has to acknowledge their product’s risk of aiding those terrorists when they’re trying to get their IPO off the ground. But also I think Scott is hilariously uninformed about how in-depth the kind of monitoring systems would need to be in order to function trans nationally and not be subject to individual jurisdiction (because otherwise the only one with authority to go after the secret illegal chinese AI bunker project would be China and that’s a non-starter. As I said in the OP I think this combined with the level of transparency that they’re discussing would basically amount to opening up all the books and internal communications directly to your competitors whether on a corporate or national level, which makes it a non-starter.

I’m not sure if we missed this one or if just I did, but there’s a class action suit against Eightfold AI. The company is accused of essentially creating a social credit score based on your digital footprint and using it to drop applications before anyone at the hiring company got a chance to see them. I spent the last two years watching my options for keeping my family housed slowly dry up while throwing out hundreds of applications. Now I’m concerned that it dug up my old reddit account from when I was a stupid teenager or decided that comments I made here weren’t suitable for participating in the goddamn economy.

Finally got around to actually subjecting myself to AI 2040. I started writing this up for the stubsack then realized I had accidentally put together a lot of words and our previous discussion on the subject had slipped into last week’s sack. That said, let’s start from the top. Having put humanity on this path, the AI companies find it acceptable. But most people don’t. Forget thinking about his legacy—the President is starting to think about what’ll happen to him after he leaves office and the world gets transformed That’s not how American Politics works. I mean it’s a bad quote because the dumb is smeared across a vast expanse of boring prose, but the idea that any election can be summed up on one single solitary issue is ridiculously naive. Local variance and immediate context matter a lot and pretending otherwise makes you a hack. The President announces that the US will pursue international cooperation to avoid an imminent intelligence explosion. That’s not how international cooperation works. I hope I’m overly cynical here but I think that the lessons of international coordination on everything from land mines and chemical munitions to crimes against humanity and nuclear non-proliferation should be relevant here. The efficacy of the various climate change pledges and agreements are also relevant. Both sides can still use datacenters for running AIs that already exist (i.e. inference), but they will retrofit each other’s datacenters with devices to verify that they aren’t being used for new training runs That’s not how IT infrastructure or independent verification work. First off, if you try and tell any competent IT security department to stick a network tap that sees all their inbound and outbound traffic and directly sends it to their competitors you’re gonna get laughed out of the room, basilisk be damned. But let’s also not forget that “how do I trust that your validators aren’t going to demand to tour unrelated infrastructure to collect unrelated military intelligence” has historically been a massive sticking point in, say, nuclear nonproliferation talks. The Consortium countries come up with a simple high-level framework: we’ll agree to let each other see all the AI research. Then, if we don’t like something someone is doing, we’ll talk about it and perhaps agree to ban it. That’s not how regulation works. This is a classic rationalist fallacy that we see repeated throughout the piece of assuming that ultimately if you express the argument properly everyone will agree on the right thing, and that the differences in what (e.g.) different countries don’t like to see done are somehow going to resolve factually to everyone independently coming to the same conclusion. Otherwise you have what is effectively a race to the bottom where even within “the consortium” everyone just migrates their operations to whoever has the most lax regulations that lets them do what they want even at the cost of breaking the world. AI workloads can be split into research and development (the process of building new AIs) and inference (the use of the existing AIs). In our proposal, research is almost entirely transparent, while inference is still private. That’s not how network monitoring works. How do you trust that I’m not running training data on inference channels unless you’re monitoring everything. Remember, we’re not just concerned with actually identifying wrongdoing, we’re trying to establish beyond a doubt that wrongdoing isn’t happening. Also let’s note that this argument assumes that continuous learning is impossible, when “actively learning long-term lessons from your experiences” is a pretty large part of human-level intelligence, much less the kind of super intelligence they’re concerned about. The result will be a competitive market for AI, in which consumers of AI services have many options to choose from and excellent visibility into what they are buying. That’s not how economics works. The entire concept of patent law revolves around making sure that you don’t lose the competitive advantage that comes from investing in improved products and processes because if you don’t do that then there’s no market incentive to improve things. If we’re gonna throw out market incentives why aren’t we throwing out capitalism altogether again? But even if we set that aside, given how capital-intensive the AI training game (supposedly) is, the big players are going to be able to take advantage of economies of scale and use the resulting efficiency to outcompete smaller shops, leading to concentration of power. We saw this happen with crypto mining pools, and should expect the same to happen basically anywhere that more returns are bought direct with more money. To accomplish this, they agree to build the datacenters in the third-party countries least secure against their rival’s military intervention That’s not how international agreements work. While it’s true that during the Cold War we saw third-party aligned countries hosting nuclear weapons, this was because they directly benefited from being under their ally’s nuclear umbrella and the associated improvements in relations. Like, Europe hosted American nukes not because they had an agreement, they had an agreement because hosting American nukes meant Russia was less likely to Leeroy Jenkins their way through the Fulda gap. Imagine telling (West) Germany that they should host Soviet nuclear infrastructure specifically so that in the event of a breakdown in relations they can be targeted by their allies for violence. I know that there’s a kind of neo-realism in international relations these days that likes to assume that non-superpowers don’t actually have agency, but I promise you this isn’t the case. The attitude towards safety flips. The incidents of AI misbehavior, plus the fact that AI is so deeply deployed into the world economy, plus the transparency into AI research, plus the breathing room to process what’s happening, all combine to dramatically shift the burden of proof. That’s not how public opinion works! At this point we’re firmly in science fiction, but I think it’s significant that part of the plan is once again “then everyone realizes we were right all along and agrees to go along with our plan with minimal friction”. It escalates to the President. He calls Xi Jinping. They bargain and threaten. They yell at each other. Ultimately Xi agrees to ban this type of thing if the US does too. Details are left to the respective regulators to hash out. That’s not how negotiations work. I’ve wanted to talk about this before but couldn’t find a clear point to throw it in, so here it is: are we gonna assume that literally every other issue or conflict in the world other than AI regulation is going to magically cease to exist? Was this part of everyone magically agreeing with everything Eliezer Yudkowsky - who totally isn’t a cult leader - wrote is completely correct? Like, are we expecting the negotiations about chip manufacturing in Taiwan not to run into friction due to the ongoing dispute about Taiwanese sovereignty? And that’s just the most obvious example, ignoring the energy crisis brought about by the disruption in Hormuz and a thousand other areas where disagreement is inevitable and can’t simply be wallpapered over by two executive leaders yelling at each other about it. Now that the United States is limited in the number of total robots it can build, it must choose how to allocate this capacity between companies. They decide to use the free market via a cap-and-trade system. Permits to build robots or compute are sold to the highest bidder and can be freely traded. That’s not how the financial system works. Okay the cap-and-trade solution here is cringe neoliberal realism, but I think the kind of numbers they’re describing merit a reality check of their own. Like, if that share of the overall economy is entirely driven by AI then from the perspective of a normal family trying to buy food and pay rent I’m not sure how you wouldn’t see massively disruptive inflation. Citizen’s dividend or not. And while I’m loathe credit mainstream economists, I also can’t see how a dramatic increase in that dividend like they talk about several paragraphs later doesn’t similarly crater the value of money since nothing that AI is revolutionizing is really impacting the kinds of stuff that people would be buying save that there’s more of it and that people are no longer being paid to produce it. Also from a political economy perspective the way this twists things is really dark. Like, if your industry accounts for three-quarters of government revenues then you basically are the government no matter what anyone says. And that’s pretty goddamn bleak if you don’t own an AI company. Even accepting the science fiction premise the only alternatives are complete capture of the state by AI companies or the nationalization of those companies by the state. Either option, combined with the existing social dislocation, could arguably spark a civil war or revolution. But then I guess we would have to acknowledge that neoliberalism isn’t God’s perfect economic regime, which isn’t possible to do here. It’s fine for AIs to get better at using valid arguments and evidence to convince people of things for the right reasons. That kind of persuasion is asymmetric: it works much better when the argument pushes towards the truth. That’s not how the postmodern condition works. (Yes I am going to keep pushing this bit past it’s limits until this dead horse starts walking and becomes funny again.) Evidence, argument, and truth itself contain an inherently subjective element that means that we can’t nearly separate “good” arguments for rationally advancing a series of syllogisms from “bad” arguments rooted in personal charisma or authority or whatever else the “argumentum ad X” crowd is going on about these days. Truth is not a viable value in its own right, and pretending otherwise just serves to reinforce the social and political systems that can decide how truth is understood at the present moment. In their sci fi Dreamland, that would be either the US and Chinese governments or the CEOs of AI companies, and lots of people are already really mad at them now even before all the sci fi nonsense starts kicking in. Everything after this point is pure science fiction, and of a kind that isn’t even interesting. Good science fiction uses the speculative technology or world to interrogate something about reality. What does it mean to have a perfect simulacrum of a human being that nonetheless isn’t one? How much power should we afford to corporations in an attempt to make miracles happen? Would you still love the god-king if he was a worm? This instead tries to use it to argue why we should ignore reality entirely. The utopia they outline is not meaningfully connected to the policies and values they advocate for, and those policies themselves are fundamentally based on a fiction about how the world actually functions.

Property rights Wait not like that. You’d think that a group of people this long-winded and pedantic would understand that the most seemingly simple rules are the squirreliest in practice

I’m sorry you had to find out this way

Counterpoint, why are these LLM bros still on human social media rather than just asking their chatbots to simulate a forum full of people who disagree with them just enough to be interesting but not so much that they actually risk changing their mind.

I have so many questions about this man and while I don’t actually want answers I would rather get them by choice than wait until his corner of the cultic milieu comes bursting into general relevancy like the Kool-Aid Man.

I doubt that this is how it was originally intended but think about all the boosters pointing at an empty suit while saying shit like “you’re being laid off and this is your replacement,” or “I know you’ve been lonely so I set you up on a blind date with this guy,” or even “this is what’s going to solve climate change.” And then imagine looking around and seeing friends, family, the broader media, and government and business leaders listening to them while smiling and nodding.

I for one would find breaking all of Bitcoin to be quite useful. Once again the conservative comedy-heavy portfolio pays off for the laughing investor!

On one hand, Anthropic sourcing suggests that this is probably at least partially nonsense. On the other hand, though, if there’s any accuracy at all I’m going to spend the rest of my life infuriated that I went down the technical degree route and actively avoided a liberal arts education in order to improve my career outlook and then this happened. Like, I don’t think they were trying to mislead but I feel like every guidance counselor for kids ought to have a plaque in their office saying “please note that the world is complicated, ever-changing, and scary and I might actually have no idea what the fuck I’m talking about”.

It’s both less sad because he’s probably not completely LARPing as his entire family on twitter, but also more sad because that’s his actual mother who couldn’t be arsed to actually be present and decided to slop it up for social media clout instead.

I want to piggyback off this to talk about the inevitable Uber comparisons, because not only is the mismatch between investment and returns several orders of magnitude greater, but there’s also a difference in kind. Uber’s model was to undercut the taxi industry and establish a dependence within their niche before increasing revenues. It’s the classic enshitttification cycle. But the AI plan, at least as advertised, isn’t to undercut a specific industry as much as it is to undercut literally the entire white-collar labor force. There are several problems with this, starting with the fact that the technology isn’t actually able to replace the target in the way it would need to. More significantly, however, is that labor doesn’t work like taxis. If labor can’t get work it shuts down the entire economy because they lose their income and can’t actually consume any of the things the market offers. Also labor tends to get mad and break out the pitchforks and molotovs if things get too bad, and “restructuring the economy to no longer provide you the means to sustain your family” seems like the kind of situation that definitionally makes things too bad. In either event the point is that even if this tech is somehow as revolutionary as advertised then there’s not really any winning for the company.

Glad to see that “regulatory uncertainty” continues to mean “we figured out that the government might actually care if we do something illegal”

Harari’s framing makes AI sound like a jungle predator learning to wear a suit. The scarier version is that it’s the suit itself — and the person wearing it has already left the building. I don’t necessarily hate this, because you can easily read it as highlighting the AI systems’ lack of agency. Rather than posing it as a threat for what it’s going to do, it poses a threat for what it doesn’t do that believers expect it to: actually exercise judgement and thought.