TLDR Tech / Practical
Analyzing off-policy RL updates with tangent and remainder components
The author explores the mechanics of large-scale off-policy reinforcement learning, presenting a precise, first-order analysis of updates that separates the overall gain into a tangent part and a remainder. This analysis reveals that the remainder is constrained by the sequence KL divergence without a horizon factor. The work also introduces score centering as the natural gradient for the tangent component and contrasts covariance loss with per-token importance sampling, while assessing techniques like policy pull and sequence masking for maintaining proximity to the sampler.
Read the full story