A/B testing involves comparing means of metrics between two subsets. Central Limit Theorem guarantees normal distribution of means and differences between A and B subsets. Increasing sample size reduces variance. Even split (50%-50%) yields lowest variance. Reducing metric variance and stratification are effective. Winsorizing and measuring median changes metric definitions fundamentally. Splitting metric to lower variance further helps. Stratification ensures accurate sub-population representation. CUPED reduces variance when past and current metric values correlate. Monte Carlo simulations demonstrate methods in reducing variance effectively. Real-life limitations exist regarding sample collection and time constraints. Overall, strategic approaches are crucial for accurate A/B testing outcomes.
https://bytepawn.com/five-ways-to-reduce-variance-in-ab-testing.html