Post #1242210
2026-04-16 01:08 UTC
Replies (4)
-
@wren6991@types.pl 2026-04-16 01:18
As an aside, MoE models seem dumber than a box of rocks on client hardware and my guess is they're prevalent because they're scaled down versions of frontier models that are divided across GPUs. Speculative decoding with paired large + small dense models seems more interesting for local use. I could imagine a parallel universe where this tech had remained a niche hobby and research interest and I actually... you know. Liked it. Instead of hating it.
-
@wren6991@types.pl 2026-04-17 00:52
@unlambda I don't have any machines that big unfortunately. I can just fit the small IQ3 quantisation of that model but it seems a little the worse for wear (and Q2_K is completely broken).
-
@wren6991@types.pl 2026-04-17 01:26
@unlambda it seems less prone than Qwen3.5-35B-A3B to going into an infinite recursive tailspin in the CoT
-
@wren6991@types.pl 2026-04-17 01:39
@unlambda I haven't dumped out the state but I suspect the sparse routing decision in pure MoE like Qwen can lead to a positive feedback loop when the experts are small. Like you have one expert which basically encodes a prior of "one more level of recursion bro, just one more level" and the routing layer keeps selecting it because that strategy worked in post-training. Bigger models have enough behavioural diversity per expert that they're less prone to that. I'm kind of interested how they fixed that in Qwen3.6 as it seems to be exactly the same architecture just with better post-training. Gemma 4 26B-A4B seems a little less prone and I expect part of the reason is the large shared expert (3x size of routed experts) makes it less prone to simple positive feedback loops through the routing layer. Idk I'm not an expert, just some guy on the internet