TLDR Tech / Dev
Real-SWE
Real-SWE evaluates model and harness pairs on ten tasks from licensed private production codebases, including business rules and company-specific conventions that cannot be recovered from public training data. Across 640 rollouts, the top configuration resolved 38.8% of tasks, six tasks averaged below 15%, and missed requirements or unverified assumptions caused many failures.
Read the full story