Post #3528140
2026-06-25 02:13 UTC
Not anymore. Not with hybrid offloading, where the GPU handles dense tensors and the CPU only runs the sparse MoEs. I’m running a 300B model on a single 3090, and its faster than I can read.
You just need to use the right framework, and the right model.
I’d suggest trying ik_llama.cpp and a MoE like one of these: huggingface.co/models?other=ik_llama.cpp&sort=mod…
And speculative decoding like DFlash or MTP (which you can also get specific models for).
EDIT: Wrong link.
Replies (1)
-
@atzanteol@sh.itjust.works 2026-06-25 03:43
I’ll check that out - speed isn’t my biggest issue so much as coding performance… The qwen 3.5 model I was using can write code, but it’s… Meh? Like sometimes it doesn’t even compile. I did try tweaking llama.cpp to do some cpu offloading and it does seem to allow for much larger contexts at a modest performance loss. I’ll check out larger models.