Post #3121849

Okay, I fudged the part about "for free." The problem is DeepSeekv4 is literally in preview, and its architecture is so new that engine support for its weights is poor. *Right this second*, you can either pay a few cents to try it from some API (there are many providers since its open weights), or rent a GPU (or maybe a CPU) instance if you don't trust the public tests, and actually want to test resource usage yourself. Or you can quantize it and self host it. I plan to do so on my 128GB RAM/RTX 3090 desktop, which is a affordable config to rent if you don't have a desktop like that. But llama.cpp support is a work-in-progress. Same with other backends like Ktransformers. Realistically your options are: - Wait a week, maybe a few weeks, for the llama.cpp/ik_llama.cpp developers to implement to DSV4 architecture. - Try one of the janky GPU/Apple forks availible right now. - Try one of the slightly-less-janky, but slow CPU-only chinese forks. But once its implemented, I'm going to make my own personal IQ3_KS mixed quantization for 128G desktops, and see how it compares to older architectures myself. *** Another confounding factor is, if you're researching "AI farm inference costs," thats very different. Frugal providers like Deepseek use complicated schemes to batch requests over many GPUs, with each taking requests in parallel. In other words, the more GPUs they have, the more *speed per GPU* they can squeeze out. For DeepseekV3, last I heard, Around 300 GPUs or so was an ideal deployment number... And they aren't even going to be using Nvidia GPUs anyway. I believe Deepseek is switching to Huawei for inference. But however you slice it, they're using order of magnitudes fewer resources than Tech Bro providers like OpenAI or Grok. They *have been*, for over a year.