Post #3625059

It’s drops off, but not as much as you’d think. MiMo uses 5:1 SWA, so its long-context compute doesn’t increase as catastrophically as older models. That, and most of the “slowness” comes from the MoE layers being on CPU (whereas the attention layers that get heavier at high context are all on the 3090). That’s the beauty of these MoEs: they’re just the right size for the “compute-lite” parts to stay in CPU RAM. I will measure it tomorrow. It is a constant ~9-10TPS for short queries though, and I have it configured to 85K tokens max at the moment. And do you mean context compaction? I don’t automate that; when I use that particular model, I tend to use it in Mikupad, aka “raw” notepad mode, and manipulate the context directly. This is so I can do things like chop out conversations, pick different tokens from the logprobs, or edit its own replies and thinking and continue mid reply. I like manually handling this because, being a local model, prompts are cached. Prompt processing is minimal, and streaming starts quickly, if you preserve most of the previous conversation, which is actually a really nice advantage over APIs.