MoE (Mixture of Experts)

A Mixture of Experts (MoE) model routes each token through only a fraction of its total parameters — a set of specialized 'expert' sub-networks — so a huge model runs at the speed of a much smaller one. This is why a 512GB M3 Ultra can serve the 671-billion-parameter DeepSeek R1 at a usable ~17–18 tok/s: only a slice of those parameters activates per token, so the bandwidth demand is far below the full model size. MoE models still need enough memory to hold all their weights, which is where the 128GB and larger unified boxes earn their keep. Dense models of the same parameter count would run much slower.

MoE is the architecture that plays best to these capacity boxes: you need the RAM to hold the whole model, but bandwidth pressure is lighter than the parameter count suggests. Eyeing very large models? Favor MoE releases — they're what make big-model local inference feel usable.