Death by detail — when data is dangerous for product teams

We live in a time when building a product is becoming a science. The world is awash with frameworks that promise you the next big product. Build, measure and learn, they say. The marginal cost of running experiments and A/B tests has plummeted to zero, making them a regular tool for product teams.

That’s it, folks. Our work here is done.

Despite their noble intentions, I have seen teams use data in counterproductive ways.

• Analysis paralysis — when a ‘data-driven’ culture becomes an end in itself, it motivates the wrong behavior. Product teams waste time performing endless A/B tests, even when they know the right thing for their customer. Endless testing creates inertia and dilutes the team’s vision, when they’d rather move on to an audacious problem.
• Local maxima or mistaking the forest for the trees — when teams are obsessed with a north-star metric, they go all out in moving the needle by 0.1%. After years of successful obsession with a metric, teams find that they missed the boat on solving a bigger, valuable problem. (Also see: Blockbuster/Netflix, Yammer/Slack)

With self awareness, product leaders can break free of these traps, and use data effectively.

Jeff Bezos offers an eloquent mantra —

“Good inventors and designers deeply understand their customer. They spend tremendous energy developing that intuition. They study and understand many anecdotes rather than only the averages you’ll find on surveys. They live with the design.

I’m not against beta testing or surveys. But you, the product or service owner, must understand the customer, have a vision, and love the offering. Then, beta testing and research can help you find your blind spots. A remarkable customer experience starts with heart, intuition, curiosity, play, guts, taste.

Make no mistake, data is a great leveler and puts the truth above our flawed opinions. However, rather than treat data as objective truth, I encourage teams to use data as an invitation — an invitation to peek under the rocks, explore our curiosity and discover wonderful insights about customers’ lives. Teams need to be obsessed with their customers’ problems — the metrics will follow.

Let the science underpin your art.