Post #3885098

2026-07-17 12:23 UTC

People who are enthralled by the fantasy of "AI" would have you believe that providing good text to their LLMs will help them be more "accurate" or "smarter" etc. That somehow the issue here is "garbage in garbage out". But that was only ever a one-way implication. High quality data in still gets us garbage out with LLMs used to synthesize text, because in doing so they make papier-mâché of their input. >>

Replies (4)

@emilymbender@dair-community.social 2026-07-17 12:23

Beyond that, I expect all professional organizations to stand for good citational practice and care with information provenance. The use of LLMs (and related tech) ruptures such systems. When the papier-mâché that comes out contains something that someone finds to be a good idea, they are cut off from understanding it in its source context. >>

Open ##3885190
@womble@infosec.exchange 2026-07-17 12:34

@emilymbender@dair-community.social ooh, I haven't seen LLM output referred to as "papier mâché" before. Very evocative of the process that creates it. Just got to remember where the accents go, and then it can become my new go-to.

Open ##3944194
@hexamander@infosec.exchange 2026-07-17 17:49

@emilymbender@dair-community.social This! So much this! I've had some good success by using the metaphor of car manuals. Say you feed an LLM manuals for multiple different makes and models of vehicle, so that it can theoretically support all vehicle maintenance needs. All the information is authoritative and well-sourced. You ask it how to do basic maintenace on a commuter car, and it gives you instructions that mix the needs of your average sedan and a heavy diesel truck. You'd do better just to read the manual for the vehicle you have.

Open ##3944195
@stuartmarks@mastodon.social 2026-07-18 02:00

@emilymbender@dair-community.social The one-way nature of this process reminds me of “Schopenhauer’s Law of Entropy”: If you add a teaspoon of wine to a barrel of sewage, you get a barrel of sewage. If you add a teaspoon of sewage to a barrel of wine, you get a barrel of sewage. (Probably not actually from Schopenhauer)

Open ##3944196