TLDR Tech / AI
Why do dLLMs tend to collapse in RL
Diffusion Language Models (dLLMs) experience training collapse during Reinforcement Learning because their log-likelihood must be estimated using high-variance Monte Carlo sampling, which creates noisy importance ratios. These noisy ratios induce gradient spikes that push policy drift in a positive feedback loop, a problem that traditional AR methods like conditional clipping fail to solve. The newly proposed StableDRL framework stabilizes the update process by combining unconditional clipping to suppress extreme values with self-normalization tied to the effective information in the batch.
Read the full story