TLDR Tech / AI
The Two MMLU Scores: What a Benchmark Name Does Not Fix
Different implementations, even with the same provider, model family, metric identifier, unit, and benchmark name, can yield varied MMLU scores. This discrepancy arises from differences in the runners, graders, and dataset splits used. The common 'mmlu' label refers to a dataset family rather than a complete, standardized measurement procedure.
Read the full story