Prompt Processing (Prefill)

Prompt processing, or prefill, is the work a model does reading your input before it generates the first token — as opposed to the token-by-token decode that follows. It's more compute-bound than the bandwidth-bound decode phase, and it's a known weak spot for unified-memory boxes: the Mac Studio M3 Ultra, for instance, has slow prefill on long contexts versus GPU rigs, despite its 819 GB/s bandwidth. The longer your prompt, the longer you wait for that first token. Discrete GPUs, with far more compute, feel snappier here even when generation speed is similar.

If you paste long documents or use big system prompts, factor prefill latency into 'is it fast enough' — not just tokens/sec. It's why a unified box can feel slower than its bandwidth suggests on long-context work.