Expert-Aware Quantisation: Near-Q4 Quality at Near-Q2 Size?

“If only a subset of experts really matter for the task, we can keep those at high precision and crush the rest.”

Mixture-of-experts models waste most of their weights on experts a given task never touches, so Alderson’s pitch is simple: keep the experts that matter at high precision and crush the rest down hard. The result is near-Q4 quality crammed into near-Q2 size, which matters a lot when RAM is the bottleneck and disk space is basically free. It’s a cheap trick that works because nobody was using most of those parameters anyway. The bitter lesson keeps proving itself: half of what these models carry around is dead weight.