Reward Models Can Be Too Sensitive

2026-06-29

Meta studied how reward models can overreact to equally good responses, leading reinforcement learning toward reward hacking. The paper proposes measuring both discriminative ability and specificity, then using Monte Carlo dropout to cluster rewards into safer discrete signals.