Quantization

Quantization shrinks a model by storing its weights at lower numerical precision — commonly 4-bit (Q4) instead of the original 16-bit — cutting the memory footprint by roughly 4× with a small, usually acceptable quality loss. It's the technique that makes big models fit these boxes: at ~2 bytes per weight a 70B model is ~140GB in 16-bit, but ~4× smaller (~40GB) at 4-bit, small enough for a 96GB-allocatable Strix Halo box. At the extreme, a 512GB M3 Ultra runs the 671-billion-parameter DeepSeek R1 at 4-bit entirely in memory. Lower bit-widths (Q3, Q2) shrink further but degrade quality faster.

Quantization is why 'how much VRAM do I need' isn't a fixed number — the same model fits or doesn't depending on the quant level you run. Plan around Q4 as the sensible default and check the fit at each precision.