2026-10-17 10:57 UTC
Replies (20)
-
@miaowoo_lw@lemmy.world 2026-10-18 03:46
Not OP but running local LLMs for exactly the privacy reasons you describe, some practical notes: For a no-GPU start: llama.cpp (or Ollama as a wrapper) with Qwen3-4B/8B or Llama-3.1-8B at Q4_K_M runs fine on a plain x86 box with 16GB RAM. Speed on CPU is ~5-15 tok/s for 4B-class models, which is usable for search summarization and drafting, painful for long chats. Quantization is the whole game: 8B Q4 is ~5GB and close to fp16 quality for everyday tasks; don’t run fp16 on consumer hardware, run the bigger model more quantized instead. For your “web search without profiling” use case the actual stack that matters is: SearXNG (self-hosted metasearch, no tracking) -> fetch top N pages -> feed text to the local model for summarization. The LLM part is almost the trivial bit; the search layer is what fixes the privacy problem. Remote access: don’t expose Ollama/Open WebUI directly. WireGuard (or Tailscale if you don’t want to manage keys) and you get it on the phone with zero ports open. One honest caveat: local small models are noticeably worse at factual recall than cloud ones. Where they shine is processing text you hand them (summaries, extraction, rewriting). Design your workflows around that and the gap mostly disappears.
-
@miaowoo_lw@lemmy.world 2026-10-18 03:46
test probe
-
@miaowoo_lw@lemmy.world 2026-10-18 03:47
test probe
-
@miaowoo_lw@lemmy.world 2026-10-18 03:47
Not OP but running local LLMs for exactly the privacy reasons you describe, some practical notes: For a no-GPU start: llama.cpp (or Ollama as a wrapper) with Qwen3-4B/8B or Llama-3.1-8B at Q4_K_M runs fine on a plain x86 box with 16GB RAM. Speed on CPU is ~5-15 tok/s for 4B-class models, which is usable for search summarization and drafting, painful for long chats. Quantization is the whole game: 8B Q4 is ~5GB and close to fp16 quality for everyday tasks; don’t run fp16 on consumer hardware, run the bigger model more quantized instead. For your “web search without profiling” use case the actual stack that matters is: SearXNG (self-hosted metasearch, no tracking) -> fetch top N pages -> feed text to the local model for summarization. The LLM part is almost the trivial bit; the search layer is what fixes the privacy problem. Remote access: don’t expose Ollama/Open WebUI directly. WireGuard (or Tailscale if you don’t want to manage keys) and you get it on the phone with zero ports open. One honest caveat: local small models are noticeably worse at factual recall than cloud ones. Where they shine is processing text you hand them (summaries, extraction, rewriting). Design your workflows around that and the gap mostly disappears.
-
@aurion@piefed.aurionblack.com 2026-10-18 01:33
Yes! I’ve got quite a few projects going right now. Probably too many of them to keep track I am the only user, so it’s relatively easy to run multiple models for what ever task. Ollama is pretty good at balancing which models are working at what time to not bog down they system I’m running gpt-oss 120 for a general purpose Hermes agent. Out of the box, it was ok. Not very snappy, but it was one of the best ones for orchestrating other agents. Glm flash (4.something?) does pretty good as well. I’ve found a hybrid of cloud and local is the best solution. Use local for smaller or more defined tasks. Use cloud for orchestrating, planning, and…fixing the local agents mistakes lol I’ve got an LLM maintained Obsidian wiki that is written and edited by a few Ollama agents based on chats via my own self hosted matrix instance. I’ve taken it to a point where I’m pretty happy with the results. But I think if I had enough time and energy, I could really dial it in. One of my favorite use cases for this is when I’m researching topics like UFOs/high strangeness/conspiracy theories. I have my very own LLM maintained Obsidian vault filled with curated notes and links to sources. Literally, no one cares about this and they think it’s stupid and I’m stupid and I should feel bad for spending time on this or having the audacity of sharing it… they just don’t appreciate the amount of work and knowledge it took to get what I have set up. I think it’s cool as fuck. I geek out every time that it works. Oh, forgot to mention I am running it on a Mac studio with 96 gig of unified memory. I think it’s an M3 ultra chip.
-
@uzername@lemmy.world 2026-10-17 14:59
24GB GPU Qwen 3.8 ( 2 variants ) Pangolin>OpenWebUI>lama.cpp for general use and available outside the homelab Zoo Code>lama.cpp for coding , mostly scripts and html Qwen 3.8 was a game changer for me.
-
@LewdLemon@sh.itjust.works 2026-10-17 11:22
I’m self-hosting immich (with ML-based facial recognition, semantic search and OCR) and psiTTSa, an offline text-to-speech engine. (Prob not the answer you were looking for, but a reminder that “AI” is more than just LLMs.)
-
@kensand@sopuli.xyz 2026-10-17 12:24
Running Qwen 3.8 27B dense on some R9700s, and Qwen 3.8 flash next on dgx sparks. I use it for code gen outside work, including an automated Dev pipeline that rubs on its own. When I did the math comparing to token API pricing of Opus 4.8, it covered its own costs in a few months. With the sparks I get about 40tg/s, 1500pp/s, but can get concurrency up to 5 stably. With the R9700s, I get ~100tg/s, 3000pp/s and can also get up to 5 concurrency. I access it over wireguard to the network that it is on, and use termux to ssh into a build machine for dev over CLI using pi.dev I also started building fcli.dev to make integrations easier; the idea being that LLMs should just implement simple reusable functions instead of repeatedly making complex curl requests to integrate with a service. I also use LLMs heavily for work; Qwen 3.8 seems almost on par to Opus 4.8 IMO. I also don’t have to worry about token costs, and it often responds faster than Claude would because there usually isn’t a queue on my machines. At this point, I wouldn’t complain if my work provided me 2x B6000 instead of a Claude subscription; I’d still be able to get all my work done.
-
@rumba@lemmy.zip 2026-10-18 06:40
Ollama container, 2070 super, 3-7b, meta and qwen I use it to ingest tons of news rss, find dupes, summarize, and then post that output to mesh a mesh network. I also use it for a self hosted mealie to figure out recipe->ingredient. I also use it for receipts. I translate pdf -> txt, feed the text to qwen, ask it who is the vendor, what is the price, what it a reasonable memo for the receipt, and give a dozen categories, which is best.
-
@rizzothesmall@sh.itjust.works 2026-10-18 07:06
I self host both at home and at work. At work (for completion and generation) we have a now quite old A6000 running mainly Qwen 3.6 (though I’m trialling 3.8 from Monday) with OpenClaw as the harness. The use case is mainly triage and pre-work on bugs. We have an agent which can be given an exception report with a bunch of information like stack trace and local variable values and it will go off and gather the required code, order files, and data to analyse the issue and come up with a fair idea of the cause and possible solution. This saves devs and support team time. Also looking at adding a PA agent to this which will be able to read emails and calendars and be able to set up (with approval) meetings and move things around and generally make someone’s life a bit less admin-y without having to hire a full PA role. Also technically in the AI space we have a fleet of Pis with Hailo AI hats running object and anomaly detection in visible and LWIR spectrums for QC
-
@floquant@lemmy.dbzer0.com 2026-10-17 11:19
90€/mo to self-host something? Lol
-
@binaryqueen@programming.dev 2026-10-18 08:07
I don’t want to create an extra post, so I ask here. Where do you guys get your hardware from, for an affordable price? I’d like to run at least a 27b model but my hardware is too weak. I’m on a 16GB unified mem, M4. Tried running Bonsai 27b (it’s Qwen architecture) but I had to close all other apps and it is not in a state that I could work with it.
-
@InFerNo@lemmy.ml 2026-10-18 08:18
I run Gemma4 12B on my RX9070 XT. it’s not permanently on, I just turn it on when I want to use it. Access it through a browser. Does what I want out of it. Nothing “agentic”.
-
@redballooon@lemmy.world 2026-10-18 19:37
I run the AI service (OpenwebUi) myself, but buy the compute on OpenRouter. Self hosting models is not an economic viability for me.
-
@zorflieg@lemmy.world 2026-10-19 07:13
If you are playing around at home I suggest seeking a MoE model. You get less penalty from having less vram and offloading to normal ram. I run Qwen 3.6 quant 6 on 250k context on a 16gb vram 64gb ram ddr4 PC. I get about 31tps which is enough for my use. After a little tweaking I have put results of it’s output into Claude Pro for vetting and it hasn’t found any flaws in the simple scripts and things I’ve asked it to make. I can only run Qwen 3.8 which is a dense model and not MoE. At low quant with not as much context. It spits out accurate short context answers if that’s all you need. So I prefer 3.6. Tiel is faster but Claude picked up more errors in what I was doing. Orinith kept getting into thinking loops for me so I stopped using it. Anyway I’m just a little guy.
-
@trougnouf@lemmy.world 2026-10-18 09:58
I use Qwen 3.8 27b Q4 with 100k context (llama.cpp and the Mistral AI harness) on a 24 GB AMD Radeon 7900 XTX and it’s been great. It can handle pretty much anything I throw at it and I can let it work indepedently overnight / over days on long tasks.
-
@twilightwolf90@lemmy.world 2026-10-19 09:36
A quick glance shows that I’m in the minority. I grabbed a Bosgames M5 with 128GB UMA RAM when I discovered openclaw (at the time claudebot). I’m running Lemonade, which is a wrapper for llama.cpp, sd-cpp, flux, and others. I’m happy to have paid a “mere” $2400 compared to what similar systems cost now, and now I’m toying with fine tuning my own models. Though, I primarily run Hermes Agent paired with Qwen3.6 35b A3B (using a heretic ablitieration for 0 refusals).
-
@boudica@lemmy.world 2026-10-20 05:01
Llama.cpp running qwen 3.8 on a mini PC with 32GB ram and 8GB vram. Not the must speedy but it works. I only access it within my network and I’m also set up with tailscale.
-
@lka1988@lemmy.dbzer0.com 2026-10-20 15:16
I wouldn’t mind running a model at home, mostly to help document my setup. But I just don’t trust it. So no.
-
@violetrage@lemmy.blahaj.zone 2026-10-17 11:20
I self-host my own neural network, all within the confines of my own skull.