Regular readers know how much I care about experimentation in marketing.
Seth Godin is starting to call this "layering" and it's at the heart of my book, Do It Wrong Quickly. But no good name has emerged. I was contacted recently by Me: What was the purpose of the solitaire vs. poker test for the Game Downloads page in Windows Marketplace (shown below)?
RK: The main reason to avoid something new is inertia. Microsoft has developed software for years and many believe they have the art perfected. With services and websites, there is extra information that is readily available: customer interactions. The ability to prototype ideas and see how customers react to them is something that was not available 15 years ago.
Here are some reasons why people avoid experimenting:
- Some believe it threatens their job as decision makers.
- Program managers select the next set of features to develop. Proposing several alternatives and admitting you don't know which is best is hard.
- Editors and designers get paid to select a great design.
- Failures of ideas may hurt image and professional standing. It's easier to declare success when the feature launches.
- We've heard: "We know what to do. It's in our DNA," and "Why don't we just do the right thing?" Me: What are the best ways to overcome cultural resistance?
RK: I'm not sure I know the "best" way, but what we are doing today is based on several efforts:- Raising awareness and educating people. We give internal talks and run classes. We know the message is resonating because we initially had a hard time filling the classes, and now they're booked with waiting lists.
- Showing successes. Running experiments with those that are on board and showing successes will help others see the opportunity. As with every population, we have the early adopters and the skeptics.
- Highlight the limitations. Experimentation is not a panacea for everything, so we should recognize when it is appropriate and highlight the limitations so it's not misused. Me: Can you could explain the advantages of controlled experiments?
RK: When I was director of data mining and personalization at Amazon, the two most successful innovations by my team were not on any road map the year before, and were initially ranked so low by myself and the team that one was given as a ramp-up project to a new employee, and the other given to an intern. The projects generated hundreds of millions of dollars in incremental revenue. Such an observation is very humbling and highlights the biggest advantage of controlled experiments: we can try a lot of things quickly, and let users guide us.
Me: Do you have any one-to-two page case studies of projects you've worked on that you would like to share?
RK: We have severalSuggest a Correction
Found an error or have a suggestion? Let us know and we'll review it.
No comments yet. Be the first to comment!