A/B testing is a great tool for improving your product, but it’s important to avoid common mistakes that can skew your results. One mistake is including users who aren’t affected by the change you’re testing, which can dilute your experiment results. Another mistake is relying too much on aggregate results and missing important insights from subgroups, also known as Simpson’s paradox. It’s also crucial to have a predetermined duration for your experiment and to test it first before rolling it out to all users. Don’t neglect counter metrics, which can measure unintended negative side-effects. Be mindful of seasonality, and always test with a clear hypothesis. Remember, not everything that can be measured matters, and not everything that matters can be measured.
https://posthog.com/blog/ab-testing-mistakes