Key takeaway: Temperature scales logits before sampling. It cannot add ideas the model did not already assign probability to — it only flattens or sharpens what is already there.
What the Parameter Does
At every step the model produces a logit for each token in its vocabulary. Softmax converts those into probabilities. Temperature divides the logits before that conversion.
Dividing by a value below 1 amplifies differences, concentrating probability on the top candidates. Dividing by a value above 1 compresses differences, spreading probability toward the tail.
At temperature 0 the model always selects the highest-probability token — greedy decoding, fully deterministic. At temperature 2 the distribution is flat enough that genuinely unlikely tokens get selected regularly, which is why output degrades into incoherence rather than becoming interesting.
The framing as a creativity control is misleading. What actually increases is the chance of sampling tokens the model considered improbable, and improbable tokens are usually improbable because they are wrong.
Temperature Versus Nucleus Sampling
top_p operates differently and often more usefully. It sorts tokens by probability and keeps only the smallest set whose cumulative probability reaches p, then renormalises and samples from that.
| Setting | Behaviour |
|---|---|
| temperature 0 | Deterministic, repetitive on long output |
| temperature 0.3 | Focused, minor variation |
| temperature 0.7 | Balanced, common default |
| temperature 1.2+ | Noticeably degraded coherence |
| top_p 0.9 | Excludes the unlikely tail adaptively |
| top_p 1.0 | No truncation |
The important difference is adaptivity. When the model is confident — the next token is nearly certain — top_p keeps only that token regardless of temperature. When the model is genuinely uncertain across many plausible continuations, top_p admits more of them. Temperature applies the same transformation in both situations, which is why high temperature damages confident predictions most.
Adjusting both simultaneously is the common mistake. Their effects compound in ways that are hard to reason about. Fix one and tune the other.
Choosing by Task
Structured extraction, classification and anything with a single correct answer should use temperature 0. Variation here is pure error, and determinism makes failures reproducible.
Code generation benefits from low but nonzero values — around 0.2 — which allows recovery from an unlucky first token without inviting invention.
Conversational and creative output works well at 0.7 to 1.0 with top_p around 0.9. Above that range the failure mode is not creativity but drift: sentences that start well and lose their thread.
For genuine variety, sampling several completions at a moderate temperature and selecting among them produces better results than one completion at a high temperature. You get diversity across outputs while each individual output remains coherent.
The Determinism Caveat
Temperature 0 is not a guarantee of identical output across calls. Floating-point non-associativity in batched GPU execution means the same prompt can produce different results depending on what else was in the batch. The variation is rare but real, and systems that depend on exact reproducibility need caching rather than a parameter setting.
The Bottom Line
Use temperature 0 for anything with a correct answer, 0.2 for code, and 0.7 with top_p 0.9 for open-ended text. Tune one parameter rather than both, and get diversity by sampling multiple completions instead of by raising temperature.
