TLDR Tech / Practical
Assessing Multimodality Testing in SWE-bench
An examination was conducted to determine if swe-multimodal v2 truly evaluates multimodality by performing tests with gpt 6 Luna xhigh using native images, custom OCR, and no visuals. The review concluded that resolution rates were consistent across all three conditions, as task descriptions and code context provided most of the required data. Additional tests on MMMU Pro demonstrated a notable decrease in accuracy without images, suggesting that SWE-bench Multimodal does not effectively assess multimodality.
Read the full story