Post #2577129
2026-05-11 15:20 UTC
Slowly working towards the inference engine I want on Apple Silicon:
- MTLSafeTensors: mmap safetensors into MTLBuffers (https://github.com/finnvoor/MTLSafeTensors)
- Popcorn: a collection of optimized Metal kernels (https://github.com/finnvoor/Popcorn)
Replies (2)
-
@finnvoorhees@mastodon.social 2026-05-11 15:21
Using these + swift-huggingface + swift-transformers I put together a POC Gemma 4 inference engine that is honestly not far off from MLX, which is nice given it has pretty much no optimizations or fused kernels (MLX seems to have massive prefill overhead) https://github.com/finnvoor/Popcorn/tree/main/Examples/GemmaCLI
-
@finnvoorhees@mastodon.social 2026-05-13 18:39
@flangg@det.social seems like MLX has some overhead that gets cached, this was after a few dry runs. Not 100% sure why but I think it might JIT compile a bunch of metal kernels that the system caches for a little while