TLDR Tech / Dev
Separating signal from noise in coding evaluations
A recent audit by Anthropic showed that approximately 30% of tasks in SWE-Bench Pro, a widely used coding evaluation benchmark, are broken due to issues such as overly strict tests, underspecified prompts, and misleading instructions. The analysis involved a combination of automated screening and human reviews by experienced software engineers. Given the flaws discovered, a recommendation to transition to SWE-Bench Pro has been retracted.
Read the full story