TLDR Tech / Marketing
Experiment Estimation Study: How Well Can 1,391 Guesses Predict A/B Test Effects?
A/B test impact is often overestimated, making gut-based prioritization unreliable. Across 1,391 predictions, the median error was 2.7 percentage points, and 56% of guesses were too high. Experience had a limited effect, with teams of 10+ years performing similarly to newer practitioners. Familiarity with similar tests improved accuracy, raising directional correctness to 65.1% and reducing median error from 3.2 to 2.5 points. Small groups performed better than individuals, with crowds of five cutting error to 2.3 points, though a consistent positive bias remained.
Read the full story