TLDR Tech / AI
SWE-Bench Pro V2
SWE-Bench Pro V2, a new benchmark with 642 tasks from 11 repositories, features corrected errors and optimized evaluations. AI models like OpenAI GPT-5 and Claude Opus 4.1 achieved only about 23% on the public set, demonstrating the increased challenge and realism compared to SWE-Bench Verified. Top models show consistent performance across tasks and languages, while smaller models struggle with complex, multi-file scenarios.
Read the full story