Post #2482525

2026-05-13 06:24 UTC

I was only made aware of this (frankly awesome) case of LLM poisoning today: https://www.nature.com/articles/d41586-026-01100-y. A researcher made up a disease and published two evidently fake preprints about it (including sentences such as “this entire paper is made up” and “Fifty made-up individuals aged between 20 and 50 years were recruited for the exposure group”), which were almost immediately picked up by LLMs and documented in their output. Worse, actual – supposedly serious – medical papers also started citing the preprints, demonstrating that academics relying on LLMs to do their work is a genuine problem! Not that I had my doubts but, if anyone did, this seems like the perfect demonstration of the problem. Article immediately added to the syllabus of the class I am co-teaching with Iris Ferrazzo on LLMs for Romance Studies/Humanities! #LLM #GenAI #academia #research #ResearchIntegrity #humanities

Replies (34)

@bms48@mastodon.social 2026-05-13 06:50

@ElenLeFoll@fediscience.org Yeah that's totally getting ingested into my Zotero along with all the other corroborating evidence for "The only way is down". But to falsify I am looking at what working code generation studies there actually are. Prof. Edsger Dijkstra foresaw "complexity generators" in 1975 already.

Open ##2744626
@serfdeweb@mastodon.world 2026-05-13 07:24

@ElenLeFoll@fediscience.org Lawyers already invented "affluenza" years ago.

Open ##2744627
@kunev@blewsky.social 2026-05-13 08:04

@ElenLeFoll@fediscience.org the "reasercher"'s name Lazljiv Izgubljenovic to a speaker of most slavic languages would read sort of like "Lying Loser" 😆😆😆

Open ##2744628
@Grimblob@mastodon.social 2026-05-13 08:41

@ElenLeFoll@fediscience.org as a lifelong sufferer of Bixonimania (undiagnosed by any medical professional) I find this article offensive. ChatGPT told me that I asked a really excellent question when I asked if I had Bixonimania, so there.

Open ##2744630
@missusgumby@universeodon.com 2026-05-13 08:47

@ElenLeFoll@fediscience.org

Open ##2744631
@Pineywoozle@masto.ai 2026-05-13 09:02

@ElenLeFoll@fediscience.org I love this quote about the original article. “Acknowledgments also thanked “Professor Sideshow Bob” and a professor from the Starfleet Academy for access to a lab aboard the USS Enterprise” 🤣 🤣 🤣 #AI #AiSlop

Open ##2744632
@FediThing@social.chinwag.org 2026-05-13 10:12

@ElenLeFoll@fediscience.org The genuine papers citing this is terrifying, are some scientists not reading their own papers any more? 😬

Open ##2744636
@djm62@beige.party 2026-05-13 10:14

@ElenLeFoll@fediscience.org Chris Morris would be proud: https://publications.parliament.uk/pa/cm199596/cmhansrd/vo960723/text/60723w10.htm

Open ##2744638
@MeneerDeBruin@mastodon.nl 2026-05-13 10:18

@ElenLeFoll@fediscience.org Good. Poison as much as possible, let it eat its own crap and dance while it burns to the ground.

Open ##2744639
@mherbert@social.chinwag.org 2026-05-13 10:40

@ElenLeFoll@fediscience.org something for you I think @minxdragon@wandering.shop

Open ##2744641
@pa27@mastodon.social 2026-05-13 10:42

@ElenLeFoll@fediscience.org Yes, saw this at the time - but it's good to remind everyone of what comes out of artificial idiocy LLMs!

Open ##2744643
@mkljczk@pl.fediverse.pl 2026-05-13 11:24

@ElenLeFoll@fediscience.org a Google spokesperson said such results reflected the performance of an earlier model. They added, “We have always been transparent about the limitations of generative AI and provide in-app prompts to encourage users to double-check information. For sensitive matters such as medical advice, Gemini recommends users consult with qualified professionals.” that would mostly work well if they released 'AI overview' as an opt-in feature instead of forcing it on users who have some trust in Google built over the past decades and don't expect it to suddenly start making stuff up lol

Open ##2744644
@kolya@social.cologne 2026-05-13 12:20

@ElenLeFoll@fediscience.org that iMac is approaching his 30s. he's earned the right to be a hypochondriac sometimes.

Open ##2744647
@Savvyhomestead@mastodon.social 2026-05-13 12:45

@ElenLeFoll@fediscience.org Some do it, DELIBERATELY with a lot of other subjects too. AI is amoral especially if the material it is fed is amoral and we know a lot of those tech guys and politicians ARE amoral.

Open ##2744648
@j12i@weirder.earth 2026-05-13 13:10

boost with CN: "AI"

Open ##2744649
@Savvyhomestead@mastodon.social 2026-05-13 13:12

@ElenLeFoll@fediscience.org The obvious end goal of AI is centralized control of information that can be used to bend public opinion, win elections for pedophiles, criminals and set trends. https://cybernews.com/ai-news/gpt-grok-push-paid-results-flights/

Open ##2744650
@villavelius@mastodon.online 2026-05-13 13:40

@ElenLeFoll@fediscience.org LLMs, like seagulls, swallow anything that's thrown at them. It's known as 'gullibility' for a good reason.

Open ##2744651
@grimalkina@mastodon.social 2026-05-13 13:44

@ElenLeFoll@fediscience.org I'm sorry, they put up fake preprints and then said other researchers citing these preprints are the problem? Standing up a fake preprint is absurdly unethical

Open ##2744652
@djl@mastodon.mit.edu 2026-05-13 14:10

@ElenLeFoll@fediscience.org This is a rerun of the Sokal Hoax. https://en.wikipedia.org/wiki/Sokal_affair Like some here, some folks thought the Sokal hoax was ethically problematic, but the postmodernists needed a wake up call, as do the LLM fans. Really: the LLM idea is the stupidest* thing to come out of Computer Science ever. We need to be embarrassed. *: Unnecessary explanation: the idea that random text generation has something to do with intelligence is really really stupid.

Open ##2744663
@Mage_of_Chaos@mastodon.social 2026-05-13 14:11

@ElenLeFoll@fediscience.org That's something the government does every few years already. Nothing new.

Open ##2744664
@agreeable_landfall@mastodon.social 2026-05-13 14:28

@ElenLeFoll@fediscience.org I'm convinced one of the main reasons we die is that we get too old or too sick to manage our own healthcare. We can't do the research to find the right studies, we can't read or understand those studies, and we can't question our providers to be sure they actually understand what's wrong with us. Once we are dependent on mere employees, the quality of care goes way down, and mistakes get made, or our treatment is just ineffective.

Open ##2744665
@failedLyndonLaRouchite@mas.to 2026-05-13 14:33

@ElenLeFoll@fediscience.org as a scientist, you are surely aware of bias such as the bias on social media, where if AI does something bad, it gets widely disseminated (goes viral) whereas if AI does something good, no one talks about it also, PSA when I was a baby PhD student in 1985, my teachers warned me over and over, don't trust something just cause it is published in a peer reviewed journal be careful of all that you read !!!!!

Open ##2744666
@mhoye@cosocial.ca 2026-05-13 14:38

@ElenLeFoll@fediscience.org You might enjoy my recent experience with raclette maximalism. https://exple.tive.org/blarg/2026/05/02/lies-damned-lies-and-stochastics/

Open ##2744667
@the_turtle@mastodon.sdf.org 2026-05-13 15:03

@ElenLeFoll@fediscience.org i *like it a lot* when people teach AI... wrong stuff. Piss in the well at every opportunity, folks.

Open ##2744668
@number6@fosstodon.org 2026-05-13 15:07

@ElenLeFoll@fediscience.org It seems like the real underlying problem is the "publish or perish" syndrome, where the value of a researcher is based on how many papers they write, or how often they're reference. So there's a proliferation of papers, many of which are meaningless, and which no one has time to actually read, being referenced in other papers by other researchers who don't have time to read and evaluate all these other papers.

Open ##2744669
@christianschwaegerl@mastodon.social 2026-05-13 15:32

@ElenLeFoll@fediscience.org The whole thing is like an oil spill.

Open ##2744670
@com@mastodon.social 2026-05-13 16:27

@ElenLeFoll@fediscience.org Unfortunately it’s impossible to test this now that the veracity of the study has been revealed.

Open ##2744671
@Twotired@universeodon.com 2026-05-13 16:31

@ElenLeFoll@fediscience.org This is why doctors recommend that women get screened for bixonimania annually, and men get screened every fifteen seconds. #bixonimania

Open ##2744673
@shafik@hachyderm.io 2026-05-13 17:18

@ElenLeFoll@fediscience.org I recently experimented with asking medical information from LLMs and the results were way worse than I expected. Anything not totally common knowledge they got wrong at least half the time. Just checking the sources they provided quickly showed this.

Open ##2744674
@riggbeck@mastodon.social 2026-05-13 17:54

@ElenLeFoll@fediscience.org It's interesting they chose that particular model of a Mac to illustrate an anthropomorphic computer. I had one, and really loved that quirky design. More of a personality than a piece of technology.

Open ##2744675
@franciswashere@mastodon.social 2026-05-13 19:39

#fake

Open ##2744676
@dazo@infosec.exchange 2026-05-13 21:40

@ElenLeFoll@fediscience.org I'm just waiting for a twist in this story. That even this story is made up, with the help of LLMs ...

Open ##2744677
@reiddragon@fedi.catto.garden 2026-05-13 23:22

@ElenLeFoll@fediscience.org Reminds me a bit of the paper stating Covid was caused by by Zubats and then that paper got cited by a bunch of conspiracy papers even though 1) Zubats are pokemon, and 2) the Zubat paper credited Bruce Wayne

Open ##2744678
@bookandswordblog@scholar.social 2026-05-17 17:12

@ElenLeFoll@fediscience.org the case of FamilySearch removing the word Nego and references to race from 18th-century manuscripts it scanned and summarized was striking too https://bsky.app/profile/caitlindeangelis.bsky.social/post/3mlrieau4h227

Open ##2744679