Post #3733564
2026-06-20 23:14 UTC
Replies (1)
-
@swelljoe@mas.to 2026-06-20 23:32
@matt@toot.cafe @Migueldeicaza@mastodon.social @glyph@mastodon.social a 4-bit quantization is 400-500GB on disk, so, you'd need almost a TB of VRAM to run it with a huge context and KV cache, like you'd get when you buy tokens from Z.ai. 1-bit quantization is 228GB on disk, so a 512GB Mac Studio could run it slowly (maybe even a two or three bit quant with smallish context). So, realistically, I don't think most normal folks will self-host GLM 5.2. But, companies might. ASUS just announced a ~$115,000 748GB AI inference machine, which could maybe run the 4-bit with maybe 200k context. A company that cares about privacy might consider that a price worth paying. When memory supply catches up with demand (or OpenAI stops buying all the supply years out into the future), building a monster like that will come back down to earth. You probably could have built something like that 2 years ago for $50,000, which is a no-brainer for a major corporation that doesn't want their data to ever leave their data centers.