Which specs are as low as reasonable possible for local LLM models? Do you recommend some distro in particular?
2026-05-14 21:51 UTC
Replies (8)
-
@meowmeow@quokk.au 2026-05-14 22:09
A budget build is going to run you $4k+ for something like qwen3-coder:30b, and you’ll probably be annoyed at the speed of you’re used to Codex or Claude.
-
@eager_eagle@lemmy.world 2026-05-14 22:52
heavily depends on the model choose the model you want on this website and it’ll give you some specs likely to run it runthisllm.com
-
@Ftumch@lemmy.today 2026-05-15 00:36
If you need or want to run an LLM on limited hardware, you may want to look into so-called bitnets with ternary connections. These should be efficient enough to run an OK LLM on a CPU with 16 GB of ram if not less. Unfortunately they’re barely out of the experimental stage, so you’ll probably have to compile BitNet.cpp yourself or wait a few months until full support lands in Ollama. I haven’t run a bitnet myself yet, so I can’t personally vouch for their effectiveness or usefulness.
-
@chicken@lemmy.dbzer0.com 2026-05-15 07:44
Get a 3090
-
@thingsiplay@lemmy.ml 2026-05-15 08:04
I use local LLM with 8gb VRAM and 32gb system RAM, thanks to Vulkan support. My GPU is a RX 7600. I can run qwen/qwen3.6-35B-A3B-Q4_K_M.gguf and gemma-4-26B-A4B-it-Q4_K_M.gguf in example. It will first fill in the GPU and the rest will use the system RAM instead, which is slower but at least it will fit and run bigger models. I just need to lower the context length, which has a great impact (current custom value is 64k for anyone who wants to know). But this is still highly limited and not competitive at all. I mostly play around with it and occasionally ask a question here or there and that’s it. So if you are serious about your system, you need something faster and with more than just 8gb VRAM.
-
@monovergent@lemmy.ml 2026-05-14 23:55
16 GB VRAM GPU, models stored on SSD, rest of the computer doesn’t have to be crazy. Intel Arc is best bang for the buck at the moment. You can get LLM running on 8 GB cards or even the CPU, but IMO such small models are more novelties than workhorses. I personally use Debian but you’ll be fine as long as your distro’s repo has drivers recent enough for your GPU. For perspective, I’m using such a build to help with boilerplate code, single-use scripts that I don’t have the patience to trial-and-error (like ones that have to deal with directory structures and special characters), getting an idea of what’s what when decompiling and reverse engineering, brainstorming tip-of-the-tongue ideas, and upscaling images.
-
@thedeadwalking4242@lemmy.world 2026-05-15 11:41
apxml.com/tools/vram-calculator
-
@plutopos@lemmy.zip 2026-06-15 08:26
There are no lower or upper bounds for local LLMs. Just get the best GPU/RAM combo you can with your budget. Some laptops also have NPUs that are dedicated specifically to AI iirc