Post #4031692

2026-07-23 08:35 UTC

somehow I only recently learned that LLMs need the whole conversation fed back to them on each prompt so their i/o cost scales as O(n^2), something that would be considered completely unacceptable in almost any other production network-accessible software

Replies (4)

@schrotthaufen@mastodon.social 2026-07-23 08:54

@jcoglan@mastodon.social @buherator@infosec.place This also means you can “gaslight” the LLM by manipulating the conversation before it’s sent back to the LLM

Open ##4031691
@troed@swecyb.com 2026-07-23 10:31

@jcoglan@mastodon.social Sort of, but not really in reality. What happens is that you keep the context cached in VRAM and only transfer what's new. It's like complaining on GIMP needing to have the whole image in RAM to be able to work on it.

Open ##4033379
@t_var_s@phpc.social 2026-07-23 15:12

@jcoglan@mastodon.social There is prompt caching, although I'm not entirely sure how it works. Basically everyone wants to solve that problem, but it's still easier to manage state by just shoving the whole conversation again down that pipe.

Open ##4039095
@jcoglan@mastodon.social 2026-07-23 08:47

they ran into this and then didn't figure out a way to externalise the state in some fixed-size form so the LLM can resume from there. you just have to restart the computer and redo everything in any other program this would be considered a DoS vector and have CVEs issued gosh I wonder why their costs are completely unmanageable

Open ##4039199