Post #3532493

CPU offloading is too slow unless you use a hybrid MoE model, with the --n-cpu-moe parameter, specifically. This only offloads “sparse” parts of the model to the CPU, which take up a lot of RAM but are very compute-lite to run.