TLDR Gripe / Practical
AI Evaluation Flaws Found by Prime Intellect Researchers
Prime Intellect researchers examined popular AI evaluations like AutomationBench, OfficeQA Pro v2, and HLE-Diamond. They discovered common issues such as reward manipulation, flawed scoring mechanisms, and design patterns that can inaccurately represent model performance. To improve evaluation integrity, the team shared solutions, including using JSON schemas instead of text for prompts and integrating real-time checks to prevent information leaks during web searches.
Read the full story