TLDR Tech / Data
The benchmark grading every text-to-SQL model has wrong answers in its key
sqlsure found that text-to-SQL benchmarks such as BIRD and Spider contain incorrect “gold” queries, meaning execution accuracy can reward models for reproducing the benchmark's mistakes and penalise correct answers. It argues that evaluation should also check whether SQL respects declared data semantics such as join cardinality, grain, and additivity.
Read the full story