Why Identical LLM Prompts Can Produce Different Answers at Temperature Zero

Thinking Machines Lab, founded by former OpenAI CTO Mira Murati, has launched its research blog with a post about nondeterminism in language-model inference. I want to unpack the idea: give a model the same prompt repeatedly, and its responses may differ. That makes reproducing an experiment or debugging a result harder.

Identical responses can be useful for scientific testing and investigation, although variation can be desirable for creative writing. The post separates ordinary sampling from a less obvious problem. Sampling generally selects a token from a probability distribution; choosing the highest-probability token is the greedy, temperature-zero case, not the definition of all sampling.

Even with temperature set to zero, an inference service can produce different answers. A common explanation combines finite-precision floating-point arithmetic with parallel execution. Finite precision means calculations are rounded, and changing the grouping or order of additions can produce different results; rounding itself is not random.

GPUs perform many calculations in parallel, but concurrency alone does not mean every calculation produces a different answer. The post notes that repeatedly multiplying the same matrices in the same setup can give bitwise-identical results. It argues that this simple observation does not explain what changes when a serving system handles different batches.

Think of a batch as a carpool containing several user requests. The number of requests grouped together can vary with load. A different batch size can lead an implementation to organize its numerical work differently; because floating-point operations have finite precision, the resulting tiny differences can affect a token choice and later output.

The proposed approach is to make the relevant operations batch-invariant: a request should yield the same numerical result regardless of the batch size. I use a kitchen analogy: weigh and mix each bowl consistently even when the kitchen is busy. The underlying implementation addresses operations including normalization and matrix multiplication, not simply keeping processing speed constant.

Attention also involves combining information across prior tokens. If work is divided into different chunks, the order of numerical reductions can change. Consistent methods for those reductions can help preserve the result; the post goes into more technical detail and notes a potential performance tradeoff.

The researchers tested Qwen3-235B-A22B-Instruct-2507 with the prompt asking about Richard Feynman, at temperature zero, generating 1,000 completions of 1,000 tokens each. Without their batch-invariant kernels, they report 80 unique completions, with the most common appearing 78 times. With the kernels enabled, all 1,000 completions were identical in that experiment.

That result is encouraging for reproducibility in the tested configuration, not proof that every model, hardware setup or source of variation has been solved. Repeatable results can make benchmarks, debugging and audits easier to interpret. They do not ensure that a response is correct, safe or trustworthy; those qualities still require separate evaluation.