Post #3611290
2026-07-05 19:08 UTC
Replies (5)
-
@paco@infosec.exchange 2026-07-05 19:44
@drwho@masto.hackers.town And you’re charged per input token. Once they start charging properly, they can say “security” or “protect the children” or some shit and jack up the size of the system prompt to increase revenue. Much the way google degraded search to shovel more ads out. BuT aNThrOpIC aRE the EtHiCaL AI!
-
@anthropy@mastodon.derg.nz 2026-07-05 23:43
@drwho@masto.hackers.town context is managed a bit more gracefully usually; the LLM is fed the entire context, with one copy of the system prompt, a list of current memories, a list of tools, etc. It's not like your context grows with another copy of the system prompt every time you send an "okay do it" (that would make it go off the rails quickly too), but if the LLM hasn't been touched in a while and isn't cached it might need to re-read the entire context. That said, bad agent frameworks might do it I guess?
-
@drwho@masto.hackers.town 2026-07-05 19:09
Those of you who have every played with Markov Chains know what I'm talking about, and what that implies.
-
@petealexharris@mastodon.scot 2026-07-05 19:44
@drwho@masto.hackers.town Also, you are paying for the tokens used to generate its empty apologies when it does something wrong, and the empty apologies are part of the transcript so you're paying to process them again for the rest of the session. It's like they're just spitting in your face really.
-
@ATurnOfTheNut@mas.to 2026-07-05 19:51
@drwho@masto.hackers.town I also read some fascinating research a few days ago that the models largely ignore the tags meant to help them remember which text is prompt, reasoning or answer. They mostly follow text style despite the tags. Makes them easier to break by including text in your prompt to fake a past chain, since it mostly implicitly trusts its own reasoning as safe. That's a security nightmare and going to result in lots of future fun.