The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon)
Ronny Kohavi on evaluation goals, user-experience guardrails, later outcomes, experiment suitability, validity warnings and surprising results. Selected third-party transcript coverage excludes sponsor reads, numerical statistical prescriptions, p-value explanations, health examples, vendors and company-effect claims. This conceptual summary is not an experiment-design or statistical analysis guide. Untimed editorial topics are not video chapters.
Executive Summary (TL;DR)
Ronny Kohavi discusses evaluation goals, user-experience guardrails, later outcomes, experiment suitability, validity warnings and surprising results. Selected third-party transcript coverage excludes sponsor reads, numerical statistical prescriptions, p-value explanations, health examples, vendor recommendations and specific company-effect claims. This conceptual summary is not an experiment-design or statistical analysis guide. Untimed editorial topics are not video chapters.
Key Takeaways
Define the evaluation before reading the result
Kohavi asks what the team is optimizing for and calls the combined evaluation an overall evaluation criterion. A result cannot be called good merely because a convenient number went up. Establish the intended outcome and the constraints that make the improvement acceptable.