Post #1984541

Also, for any interested, desktop inference and quantization is my autistic interest. Ask my anything. I don't like Gemma 4 much so far, but if you want to try it anyway: - On Nvidia with no CPU offloading, watch this PR and run it with TabbyAPI: https://github.com/turboderp-org/exllamav3/pull/185 - *With* CPU offloading, watch this PR and the mainline llama.cpp issues they link. Once Gemma4 inference isn't busted, run it in IK or mainline llama.cpp: https://github.com/ikawrakow/ik_llama.cpp/issues/1572 - If you're on an AMD APU, like a Mini PC server, look at: https://github.com/lemonade-sdk/lemonade - On an AMD or Intel GPU, either use llama.cpp or kobold.cpp with the vulkan backend. - Avoid ollama like it's the plague. - Learn chat templating and play with it in mikupad before you use a "easy" frontend, so you understand what its doing internally (and know when/how it goes wrong): https://github.com/lmg-anon/mikupad *** But TBH I'd point most people to Qwen 3.5/3.6 or Step 3.5 instead. They seem big, but being sparse MoEs, they can run quite quickly on single-GPU desktops: https://huggingface.co/models?other=ik_llama.cpp&sort=modified