Memory Bandwidth

Memory bandwidth is how fast the processor can read the model's weights out of memory, measured in GB/s, and for local LLM inference it's the single number that best predicts tokens/sec. Every weight is read once per token generated, so bandwidth divided by model size roughly caps your speed. That's why a Strix Halo box at ~215 GB/s real produces single-digit tokens/sec on a dense 70B, while an M3 Ultra at 819 GB/s is several times faster on the same model. Discrete GPUs hit 800–1000 GB/s; the NVIDIA GB10 boxes sit at 273 GB/s despite being CUDA-native.

Bandwidth, not core count or TOPS, is the spec to weigh for generation speed. Compare it directly across boxes: the M4 Max (410–546 GB/s) and M3 Ultra (819 GB/s) beat every AMD/NVIDIA unified box (215–273 GB/s) on throughput, even though those boxes match them on capacity.