Ustream ©

Data Driven Product Development: Experimentation and A/B Testing

1,167 Views Length: 100:23 A recurring theme in the last meeting about monetization was the importance of measuring what’s working and what isn’t. So this week we are going to focus on bringing a data driven product development approach to your startup through… A recurring theme in the last meeting about monetization was the importance of measuring what’s working and what isn’t. So this week we are going to focus on bringing a data driven product development approach to your startup through experimentation and A/B testing (aka split testing).

We are delighted to have two amazing groups volunteer to speak:

*
Ronny Kohavi, head of Microsft’s Experimentation Platform (and before that Director of Personalization at Amazon) will share stories of experimentation at MS: lessons learned, cultural challenges, and the ROI of this effort.

*
Sandra Cheng and Victor Chudnovsky from Google’s Website Optimizer team, respectively the Product Manager in charge of product strategy for GWO and a core engineer on the product, will show us how to use GWO to add experimentation to our sites today.

Both of these teams are at the forefront of bringing a more scientific approach to product development. If you’ve ever been interested in learning the why, how, and what about A/B testing, you’re not going to want to miss this meeting.
Online Experimentation at Microsoft

Over the last three years, we built an experimentation platform system (ExP) at Microsoft, capable of running and analyzing controlled experiments on web sites and services. The goal is to accelerate innovation through trustworthy experimentation and to enable a more scientific approach to planning and prioritization of features and designs (Foley, 2008). Along the way, we ran many experiments on over a dozen Microsoft properties and had to tackle both technical and cultural challenges. The talk will focus on problems not commonly addressed in technical papers: cultural challenges, lessons, and the ROI of running controlled experiments.

The talk is based on the paper at http://exp-platform.com/expMicrosoft.aspx , which won 3rd place at the Third Workshop on Data Mining Case Studies and Practice Prize.
About the Speaker

Ronny Kohavi is the General Manage ... More