TLDR Tech / Practical
Non-Determinism in Temperature 0 LLM Outputs Explained
Researchers have identified that outputs from large language models at temperature 0 are not consistently deterministic due to floating-point addition non-associativity and non-batch-invariant GPU kernels introducing minor numerical errors. A derived formula shows that the probability of output variation is linked to the proximity of logit values and the level of noise, with a small number of critical tokens being most affected. While batch-invariant kernels can resolve this issue, simulations indicate a negative impact on performance.
Read the full story