Post #4039198

2026-07-23 08:49 UTC

this also makes it hard to take seriously claims that the marginal cost of inference is not a serious contributor to the resource use of data centres. your program has polynomial running cost and people run it for hours on end

Replies (2)

@jcoglan@mastodon.social 2026-07-23 08:52

SAMA ET AL: why can't we make a profit with this revolutionary technology ME: your operating cost scales super-linearly with the value it delivers to users SAMA ET AL: better tell the government we have invented the angels from evangelion or something

Open ##4039197
@aredridel@kolektiva.social 2026-07-23 15:52

@jcoglan@mastodon.social Yeah, you've fundamentally misunderstood how 'running for hours on end' works. Compaction and other reductions of the context window happen. Things are delegated to sub-agents with their own context windows. This is not unbounded context growth at O^2, this is pretty tightly managed context windows where window^2 is an acceptable number and GPU does go BRR. Nobody is doing billions squared. They're doing, at most, 1M, and usually more like 100k. All of this is made more efficient by attention architectures, too, which reduce that.

Open ##4505509