Post #3121822

2026-05-05 23:49 UTC

I think we're seeing a lot of optimization right now. The most exciting one I've seen is TurboQuant. Short version, every message you send to a model has context, the entire conversation you've had, instructions, skills, everything. That takes up an _exponential_ amount of ram, and this is what is causing the VRAM/RAM shortage. TurboQuant (and other copycats now) claims that it can reduce that VRAM usage of the context by **20x**. That's absolutely huge, that's 1M context models running on consumer hardware potential huge. Deepseek v4 also boasts some large claims, saying they have a model that does better than Anthropic's or OpenAI, while being 1/10th the size. That also is a huge reduction in compute and VRAM, but I'll be looking for the proof. We've seen other items too, with upgrades in running models, how quickly results are streamed, to me TurboQuant is the most exciting. I think it's good that they're finally looking at optimization. Yes, their cost has been power and compute. NVidia is more than happy to keep things inefficient because they sell GPUs that way. Software companies are doing the opposite now, reducing the compute overhead to start saving them money, which they desperately need to do if this is going to continue. New technology has always been horribly inefficient, it's only once more people see it does it start to get optimized. I think this is what is going to be required to finally push past the horribleness of AI companies, and they need to do this quickly.

Replies (4)

@Zikeji@programming.dev 2026-05-06 02:26

There's also speculative decoding and adjacent techniques getting traction, increasing performance of the models on the same hardware.

Open ##3121840
@brucethemoose@lemmy.world 2026-05-06 13:54

Also, on Deepseek V4… you can run it yourself, for free. There’s no mystery. And there’s tons of benchmarks out there already. It’s indeed very efficient if you’re into long context. But at shorter context lengths, it’s not *too* different than Deepseek’s previous releases (and the flood of MoE models that have come since then).

Open ##3121841
@GamingChairModel@lemmy.world 2026-05-06 00:06

> New technology has always been horribly inefficient, it’s only once more people see it does it start to get optimized. Well, I wonder if the frontier ends up looking like supersonic commercial flight (prohibitively expensive so that there wasn't enough of a market for consumers at the actual cost of providing the service): technology that continues to exist but never really gets used, because the alternatives that aren't as good are still much, much cheaper.

Open ##3121844
@brucethemoose@lemmy.world 2026-05-06 13:48

TurboQuant is total baloney. It’s just KV cache quantization, and we’ve had all sorts of that for ages. *Backends*, not just papers, have had 4-bit cache with hadamard rotation (a major component of TurboQuant), and very low loss, since like 2023. We’ve had proof that Bitnet works for over a year. And no one cares. No one uses that kind of quantization because it reduces batched throughput, just like TurboQuant. Besides, new architectures (like DeepSeek V4) render it obsolete, as they don’t use traditional KV cache anymore. I honestly have no idea how TurboQuant became such a meme, other than major astroturfing. *** ***TL;DR*** All AI news is total bull. It’s chum for investors. You need to look at what the engines, papers and actual LLM weight architectures are doing.

Open ##3121845