Post #3528141
2026-06-25 01:58 UTC
64G. But CPU inference is painfully slow.
Replies (1)
-
@brucethemoose@lemmy.world 2026-06-25 02:13
Not anymore. Not with hybrid offloading, where the GPU handles dense tensors and the CPU only runs the sparse MoEs. I’m running a 300B model on a single 3090, and its faster than I can read. You just need to use the right framework, and the right model. I’d suggest trying ik_llama.cpp and a MoE like one of these: huggingface.co/models?other=ik_llama.cpp&sort=mod… And speculative decoding like DFlash or MTP (which you can also get specific models for). EDIT: Wrong link.