TLDR Tech / Practical
Gradient as Covariance in LLM Parameter Updates
This technical analysis explains the sparsity of parameter updates in large language models after training. The author demonstrates that with softmax sampling, the expected policy gradient represents a covariance, and the Fisher information matrix signifies a variance. This relationship establishes a Cauchy-Schwarz bound that restricts updates, and the Fisher matrix is further broken down into token-level variances of logit sensitivities, determining the areas where learning can occur.
Read the full story