Tokens per Second

Tokens per second (tok/s) is the throughput of text generation — how many word-pieces the model emits each second — and it's the number that decides whether local inference feels usable. As a rough rule of thumb one token is about three-quarters of a word, so 10 tok/s reads faster than most people do and single-digit tok/s feels sluggish for chat. On these boxes it's set mostly by memory bandwidth ÷ model size: a dense 70B on a ~215 GB/s Strix Halo box lands in the single digits, while a 512GB M3 Ultra runs DeepSeek R1 671B (a sparse MoE) at ~17–18 tok/s under 200W. Long prompts add prefill latency before the first token appears.

Set your expectation by model size and bandwidth before buying. If you need fast responses on dense 70B models, no unified box matches a discrete GPU; if you mostly run smaller or MoE models, the capacity boxes are plenty.