Post #1286701

2026-04-16 01:18 UTC

As an aside, MoE models seem dumber than a box of rocks on client hardware and my guess is they're prevalent because they're scaled down versions of frontier models that are divided across GPUs. Speculative decoding with paired large + small dense models seems more interesting for local use. I could imagine a parallel universe where this tech had remained a niche hobby and research interest and I actually... you know. Liked it. Instead of hating it.

Replies (2)

@wren6991@types.pl 2026-04-16 23:13

There is a 20 GB file on my laptop that can just... do this? (see alt text on the first image) I tried a traditional OCR to compare: the result useful but worse than clank-o-vision. I attached the original image in case anyone wants to try their own hand (source: https://dailyportalz.jp/kiji/ji-ga-kitanai-otona) There are mistakes in the clanker's output. If I prompt it with my guess for a correction, it responds with a plausible summary of why the correction makes sense both visually and in context (third image). It doesn't "think" but it regresses its response from the full context of the image, its original output and the correction, which can yield useful synthesis of those three. A couple more asides: * The way these multimodal models work is super weird: they embed the image in the same vector space as text input so it sees it as a string of highly-dimensional "words". Honestly a neat hack that has no right to work as well as it does. * I showed it one of the new scans of the N-1 moon rocket photos (pic 4) and it identified it as an R-36 ICBM, which is *quite different*, so the model is still dumb as dirt and the "blurry JPEG of the internet" analogy still holds.

Open ##1286702
@penguin42@mastodon.org.uk 2026-04-16 01:31

@wren6991 Hmm I've had good luck with MoE - especially since I'm running on CPU; just been playing with the Gemma 4 MoE version (26B-A4B - the unsloth quantised version) and it seems to me the smartest I've run locally and relatively fast.

Open ##1286704