Remote
Talin
@talin@mastodon.social
mastodon
4.7.0-beta.1
I'm not a mad scientist. I'm a mad natural philosopher.
0
Followers
0
Following
16
Posts
Joined July 26, 2018
Occupation:
Software Engineer
Posts
Replying to
@talin@mastodon.social
Replying to
@ewhac@mastodon.social
@ewhac LLMs have a lot in common with Lysenkoism:
- Lots of true believers
- Promises of vast economic benefits, so long as everyone gets on the bandwagon
- Powerful institutions shoving the concept down the throats of the common people
- Skeptics are demonized
Replying to
@ewhac@mastodon.social
@ewhac Code is not an asset; code is a liability. Being able to spew out code faster means that you gain liabilities at an increased rate.
I recently heard someone refer to the process of writing code as "enbugging".
Replying to
@talin@mastodon.social
@cstross@wandering.shop Also, if someone were to write such a book I would absolutely buy it.
There are vast troves of transgender fan-fiction online, a lot of it bad, and a surprising amount of MtF fiction (on places like nifty and fictionmania) is apparently written by subs (not judging!) for whom becoming a woman is the ultimate humiliation fantasy -- and has little relation to the experience of actual transgender people.
I'd pay to read something that was actually good.
Replying to
@cstross__dup_483@wandering.shop
@cstross@wandering.shop A modern version of this tale would have to take into account that the protagonist would be used as a political football by many different factions, both pro- and anti-transgender.
As a general rule, transgender is hard to write about because even mundane details can read as porn. Although some authors have done it (Delaney, Banks, etc.)
Replying to
@talin@mastodon.social
@cstross@wandering.shop As LLMs improve, they will get better at chasing their fitness function, that is, at convincing people that what they say is true. This is what LLMs have in common with YGBM.
Those who know history know that the invention of cheap printing sparked of centuries of bloody religious conflict. In this regard, LLMs are like "hold my beer".
Replying to
@talin@mastodon.social
@cstross@wandering.shop Now, we know from the earliest days of machine learning that algorithms are capable of exploiting any loophole in their fitness function with seemingly godlike competence.
Unfortunately, we can't make a fitness function for intelligence, truthfulness, or integrity. What we can do, at great expense, is make a fitness function for plausibility.
When an LLM goes through its reinforcement learning, it's behavior is rewarded based on whether some human reviewer believed the result.
Replying to
@talin@mastodon.social
@cstross@wandering.shop In the book _Rainbows End_, Vinge proposes a form of mind control called "YGBM" technology, which stands for "You Gotta Believe Me", a way to get affected individuals to accept arbitrary propositions as truth.
What this has in common with LLMs is technologically enhanced plausibility.
Replying to
@talin@mastodon.social
@cstross@wandering.shop Because so many people boosted my post, I want to explore this topic in more detail.
First, a thought experiment: suppose we had cheaply scalable mind control, even if partially effective (LLMs are not mind control but they touch some of the same wires). Democracy would end, since the controllers could sway elections at will. The so-called free market would instantly become a command economy. Science, and any other activity requiring independent thought, would be a dead letter.
Replying to
@talin@mastodon.social
@cstross@wandering.shop Put another way: LLMs have revealed a zero-day exploit in human consciousness and culture: if you can manufacture plausibility at scale, you can bypass all of the accumulated wisdom of centuries of skepticism and critical thinking. Any fact-using profession is potentially vulnerable to this attack.
Replying to
@cstross__dup_483@wandering.shop
@cstross@wandering.shop LLMs are replacing every aspect of civilization - institutions, process knowledge, organizational competence - with civilization fan-fiction.
Replying to
@bendelarre@mastodon.social
@bendelarre I tend to think of AI as the new court astrologers - entities designed to tell the powerful what they want to hear. Court astrologers weren't stupid, they had math, even Galileo did it, and he's a hero of science. This doesn't make them any less wrong.
Consider also that "plausibility" is what reinforcement learning trains AIs to do. Answers are evaluated not on truth or usefulness, but whether a human reviewer believed it. This makes the errors ever so much harder to spot.