TLDR Tech / Marketing
How to Avoid Will Rogers' Phenomenon in A/B Testing
Will Rogers' Phenomenon occurs when moving people between groups makes the average performance of both groups improve, even though no one actually improved. In A/B testing, this can happen when users cross a threshold during an experiment and get reclassified from casual to power users, which can make both segments appear to perform better. This can create false conclusions about a feature's impact and mislead marketing or product decisions. Freeze segment definitions before the experiment starts and track segment migration separately to avoid the problem.
Read the full story