Post #4039199

2026-07-23 08:47 UTC

they ran into this and then didn't figure out a way to externalise the state in some fixed-size form so the LLM can resume from there. you just have to restart the computer and redo everything in any other program this would be considered a DoS vector and have CVEs issued gosh I wonder why their costs are completely unmanageable

Replies (5)

@jcoglan@mastodon.social 2026-07-23 08:49

this also makes it hard to take seriously claims that the marginal cost of inference is not a serious contributor to the resource use of data centres. your program has polynomial running cost and people run it for hours on end

Open ##4039198
@gabrielesvelto@mas.to 2026-07-23 13:01

@jcoglan@mastodon.social I think part of the reason is that injecting randomness into each step is crucial to the way they are marketed and to many of their use-cases. Without it you'd get predictable outputs which would destroy the illusion of a "natural" conversation with the user.

Open ##4039268
@henrahmagix@tech.lgbt 2026-07-23 08:56

@jcoglan@mastodon.social Have-You-Tried-Turning-It-Off-And-On-Again as a service

Open ##4505510
@frabcus@mastodon.social 2026-07-23 15:49

@jcoglan@mastodon.social Yes they did - the K/V cache. And prices are cheaper if your subsequent queries with the same prefix are within time that that stays in memory on the server. Agreed it is extremely inefficient, and mostly LLMs are better if you do controlled prompts with JSON output, rather than long conversation threads. But people are pushed into the long threads by the standard harnesses. That is a waste!

Open ##4505512
@aredridel@kolektiva.social 2026-07-23 15:49

@jcoglan@mastodon.social Uhhh... they did. It's called "KV Caching" All of this is both a subject of active research _and_ the constant factor on that O^2 is basically acceptable, and bounded.

Open ##4505513