Language model harnesses are compositional generalizers

2026-07-21

Scaling data will remain the biggest driver of progress. The machine that we feed that data into and its inductive biases are what will determine the coefficients of that scaling. Better returns on scaling require compositional generalization. The capacity for compositional generalization seems to largely live in harnesses.