Context Window

The context window is how many tokens a model can consider at once — its working memory for the prompt plus everything it has generated so far. Larger contexts let you feed whole documents or long chat histories, but they cost memory: the key-value cache grows with context length and sits in the same unified-memory pool as the weights. On a 96GB-allocatable box, a long context eats into the budget you'd otherwise spend on a bigger model. Long prompts also add prefill time before the first token, which is slower on unified-memory boxes than on discrete GPUs.

When you size a box, the model weights aren't the only thing in memory — a big context window can add many gigabytes of KV cache. If you run long-document workloads, leave headroom above the model's own footprint.