TLDR Tech / Opinion
Policy Gradient's Reach in Reinforcement Learning for LLMs
This technical discussion explores the mathematical limitations on how much a single policy gradient update can alter response probabilities in reinforcement learning for language models. By analyzing the covariance and Fisher information representations, the author concludes that policy gradient functions as a similarity-driven amplification. This means rare responses are only adjusted if their score structure aligns with the sampler's current output. Consequently, unlike supervised fine-tuning or direct examples, reinforcement learning alone struggles to introduce new responses that lack existing structural connections.
Read the full story