Post #1333339

2026-04-17 05:19 UTC

IMHO LLM usage isn't coherent with independence. That being said I wrote quite a bit on self-hosting LLMs. There are quite a few tools available, like ollama itself relying on llama.cpp that can both work locally and provide an API compatible replacement to cloud services. As you suggested though typically at home one doesn't have the hardware, GPUs with 100+GB of VRAM, to run the state of the art. There is a middle ground though between full cloud, API key, closed source vs open source at home on low-end hardware : running STOA open models on cloud. It can be done on any cloud but it's much easier to start with dedicated hardware and tooling, for that HuggingFace is great but there are multiples. TL;DR: closed cloud -> models on clouds -> self-hosted provide a better path to independence, including training.

Replies (1)

@SuspciousCarrot78@lemmy.world 2026-04-17 05:34

Yeah, me too :) https://bobbyllm.github.io/llama-conductor/ https://codeberg.org/BobbyLLM/llama-conductor I'm thinking about coding a >>cloud side car at the moment, with the exact feature you mentioned...but...that's scope creep for what I have in mind. Irrespective of all that, I agree: an open cloud co-op could be a good way to have SOTA (or near SOTA - GLM 5.1 is about as close as we have right now) access for when needed. (Not teaching you to suck eggs, so this comment is for the lay-reader): For coding, you can do some interesting stuff where the cloud model is the "general" and the locally hosted LLM is the "soldier" that does the grunt work. We have some pretty decent, consumer-level-hardware runnable "soldiers" now (I still like Qwen 3 coder)...they just don't quite have the brains to see the full/big picture for coding.

Open ##1333349