Conversions: Living a Testing Culture

by November 9, 2017

In his Living a Testing Culture presentation at Google Conversions 2017 in Dublin Ireland, Max van der Heijden talked about the A/B testing culture at Booking.com and lessons he learned working within it. Here's my notes from his talk:

  • Booking.com books 1.5 million room nights per day. That's a lot of opportunity to learn and optimize. More than 40% of all Booking.com sales happen on mobile
  • Booking.com A/B tests everything. If something cannot be A/B tested, Booking.com won't do it. There's more than 1,000 A/B tests running at any time.
  • Teams are made for testing a hypothesis. They're assembled based on what resources they need to vet a hypothesis. They are autonomous, small, multi-disciplinary (designers, developers, product owners, copyrighters, etc.) and have 100% access to as much data as possible.
  • Booking.com's experiment tool allows everyone to see all current experiments to avoid overlaps and conflicts between testing.
  • Predicting customer behavior is hard. Teams at Booking.com adapt often. As a result, teams often dissolve, spin up, and shift around.
  • Failure is OK. Most of your learnings come from tests that don't work. 9/10 tests at Booking.com fail.
  • Everyone needs to to live the testing culture. Ideas can come from anywhere and go all of them go through the same testing process. Everyone is part of this.
  • Data only tells you what is happening. User research tells you why. These two efforts work together to define what experiments to run next and then test them.
  • What worked before might not work now. Always challenge your assumptions.
  • Make sure you can trust your data and tools. Booking.com has a horizontal team dedicated to improving its experiment tool on a regular basis.
  • You can innovate through small steps.
  • Delighting customers doesn't necessarily mean more profit. Short term gains may not align with long term value. You'll need to find a balance.
  • Go where the customers(tests) take you.