O'Reilly logo

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

A/B Testing, A Data Science Perspective

Video Description

Deciding whether or not to launch a new product or feature is a resource management bet for any Internet business. Conducting rigorous online A/B tests flattens the risk. Drawing on her experience at Airbnb, data scientist Lisa Qian offers a practical ten-step guide to designing and executing statistically sound A/B tests.

  • Discover best practices for defining test goals and hypotheses
  • Learn to identify controls, treatments, key metrics, and data collection needs
  • Understand the role of appropriate logging in data collection
  • Determine how to frame your tests (size of difference detection, visitor sample size, etc.)
  • Master the importance of testing for systematic biases
  • Run power tests to determine how much data to collect
  • Learn how experimenting on logged out users can introduce bias
  • Understand when cannibalization is an issue and how to deal with it
  • Review accepted A/B testing tools (Google Analytics, Vanity, Unbounce, among others)

Lisa Qian focuses on search and discovery at Airbnb. She has a PhD in Applied Physics from Stanford University.