TLDR Tech / AI
Auditing the Reliability of Coding Benchmarks
OpenAI examined SWE-Bench Pro's construction, model failures, and task metadata, concluding that roughly 30% of its public tasks were broken. The analysis showed how flawed evaluations could distort assessments of coding ability, safety, and model progress.
Read the full story