TLDR Tech / News
New Benchmark Tests AI on GitHub Issues Using Visuals
A new benchmark called SWE-bench Multimodal has been developed by academics from Stanford and Princeton universities. It challenges AI systems to fix actual GitHub issues within JavaScript projects, incorporating visual information like screenshots and video recordings. The evaluations demonstrated that providing internet connectivity substantially boosts performance by enabling agents to locate related pull requests, leading the creators to release all test results without network access. Performance scores ranged from 37.50% to 70.21%, with Claude Opus 5.5 achieving the highest score.
Read the full story