Post #3881523

2026-07-17 09:05 UTC

I'm VERY curious (if you can tell me) if and how your company does local-and-smol models - did folks just get a chunk of openrouter (or equivalent) to try all the models or did folks just get a handful of GPUs and llama'd it all into place or... I'm particular interested in regular/small/average companies and their decision making process around this IF you decided to actually use LLMs

Replies (1)

@masek@infosec.exchange 2026-07-17 09:12

@sushee@ohai.social We went for the HALO Strix platform with 128GB RAM (https://minisforumpc.eu/products/minisforum-ms-s1-max-mini-pc). There were two reasons for that decision: We hate it to depend on something that someone else can take away at any momentWe supply customers whose data must never leave their own premise. So if we want to place our product, it must work with a Local LLM. We developed our own product using LiteLLM in between so we can always compare easily what a LocalLLM can give us in comparison to the standard models. We also can compare results with other "sovereign" EU models easily. Unfortunately the pricing went up for our favorite hardware significantly in the last months. End of last year I stumbled upon the HALO Strix platform which turned out to be the sweet spot for bang per $ (at that time). Feel free to ask ...

Open ##3881522