TLDR Tech / AI
SFT, RL, and On-Policy Distillation Through a Distributional Lens
Different post-training methods like SFT, RL, and On-Policy Distillation reshape a model's distribution in distinct ways, impacting performance and risk of catastrophic forgetting. RL updates policies using rewards from the current policy's samples, promoting task performance while minimizing forgetting, unlike SFT, which pulls towards external data, risking existing capabilities. Experiments show On-Policy Distillation can outperform its teachers, suggesting on-policy sampling data crucially preserves capabilities, making it a key ingredient for future algorithm designs.
Read the full story