The Freedom of Fast Iterations: How Netflix Designs a Winning Web Site

“We make a lot of this stuff up as we go along,” the lead designer said. Everyone in the group laughed until he continued, “I’m serious. We don’t assume anything works and we don’t like to make predictions without real-world tests. Predictions color our thinking. So, we continually make this up as we go along, keeping what works and throwing away what doesn’t. We’ve found that about 90% of it doesn’t work.”

We were talking with the design team at Netflix. Netflix.com is one of the most successful web sites in the world: the sole customer interaction point of their home DVD-rental business. Over the past 9 years, the site has grown from nothing to serving almost 6 million customers who use the site to prune their rental queues, rate movies, and handle any billing and transaction issues. The designers of Netflix.com have a smashing success on their hands, but we didn’t find them resting on their laurels. They want to get even better, and for them that means iterate, iterate, iterate.

Netflix isn’t the only company using a fast iterative design approach. Google has also gained attention for their unorthodox design methods, with many people complaining that they have a huge stable of products, but only a few they’ve designed well. Yet, Marissa Mayer, Google’s Vice President for Search Products and User Experience, explains how trying out lots of ideas helped Google design a better toolbar:

“In the case of the Toolbar Beta, several of the key features (custom buttons, shared bookmarks) were prototyped in less than a week. In fact, during the brainstorming phase, we tried out about five times as many key features—many of which we discarded after a week of prototyping. Since only 1 in every 5 to 10 ideas work out, the strategy of constraining how quickly ideas must be proven allows us to try out more ideas faster, increasing our odds of success.”

This “try and see” attitude is taking hold. The designers at Netflix told us they try out many new features with every site iteration to keep pace with the rapidly changing landscape of the Web, as well as their customers’ increasing comfort with the current site. Much of what they do try doesn’t survive to the next iteration.

So how often does Netflix update its site? Every 2 weeks.

Every 2 weeks they make significant changes. They understand that some of the changes will work, but most won’t.

At first, this sounds like a frustrating design constraint. In talking with the team, we realized that it doesn’t frustrate them at all. Instead, it frees them up to be flexible and adaptive, so they can react effectively to customer needs. As a result, they don’t deal with the many “when we redesign” issues that so many of us deal with in the design world. They’re building for the present—all the time.

Besides living in the now of their web site’s design, The Netflix team gets other benefits to fast iterations. Here are a few our research has uncovered:

Benefits of Fast Iterations

• Fail Fast

  A major benefit of fast iteration is you also fail fast. Failing fast means you invest less time in the things that don’t work. If you quickly find out what works and what doesn’t work, then you take action to turn it into something that does work.

  Ironically, teams that fail fast improve as fast, if not faster, than those who try to get it right the first time. The reason is simple: Teams trying to get it right the first time fail as often as everyone else does. However, when they fail, they fail really slowly and struggle to pinpoint problems because they’ve changed so much at once, making it harder to identify solutions.

• More Experimentation

  The faster you fail, the more experimentation you can do. You can try out ideas that might not have a lot of support, but could be potential winners. This allows for an innovative environment.

  Perhaps you’ve heard of Google’s 20% time? They expect their engineers to work 20% of their time on a personal project—an experiment they find personally interesting. This program has the effect of bootstrapping experimentation, so it will happen more often.

• Learn Quickly

  We’ve all had the experience of sitting in meetings arguing whether something will work or not. Usually, both sides just don’t have enough data to go on, and they end up going with their gut or with the loudest arguer (for better or worse). Fast iteration helps solve this problem by giving developers a platform on which they can test quickly, helping to collect data about any outstanding questions instead of resorting to opinionated arguments.

• Provide Continuing Interest

  In addition to improving your design, fast iterations may have a psychological effect on users. Those users who use your app with any frequency will notice the changes, and if the good ones stick, they’ll appreciate your ongoing efforts to improve.

  The best teams not only design the changes, but design the process for introducing the change. They experiment with methods to overcome the users’ natural resistance to change, providing migration paths and clear benefits for each improvement.

• Reduced Risk

  Quickly iterating helps reduce risk during design. If teams can make many small changes instead of a few larger ones, they mitigate risk because they know which changes have what effect. If a design team makes many changes at once, they have a harder time knowing which parts work and which parts don’t. When you make only one or two changes at a time, you know immediately what effect it has. Reducing risk is a valuable outcome of moving to fast iteration.

Side Effects

These benefits don’t come easy. There are significant changes design teams have to make to their core process to iterate quickly. It’s not a switch a team can turn on or off.

• Culture Change

  Most designers are accustomed to long release cycles. Fast iteration and fast evolution of design creates a different kind of design environment. Gone are the grand visions of the redesign, where teams spend months retooling vast areas of the site. Replacing it is the idea that the site is a living, breathing design that needs constant care and attention. The team at Netflix calls themselves “compulsive data wonks”. They rarely dream very far in the future. Instead, they’re concerned with what’s happening right now.

• Design Determinism

  When teams make the switch to fast iteration, it changes the site’s testing methodology. Testing becomes ongoing. After a release, you test for a certain period of time to determine what to keep and what to throw away. Then you start the process over again immediately. And repeat.

  To some designers this sounds overly deterministic: Doesn’t this take the fun out of design? If all the decisions are cut-and-dried, what does that say about creativity? What about longer-term effects? Is it possible that some features take longer to catch on than others, and that an early flop might not mean it’s not a valuable feature? With fast iterations, if the feature doesn’t work now, then it’s not right for the site, no matter how creative it is.

• You’re Either With Us…

  Netflix’s Chief Talent Officer, Patty McCord, told us their process of fast iteration causes uncomfortable situations for some designers. Once, a designer had spent time and energy working on a feature that testing showed didn’t work. When it came time for the team to remove the feature from the site, the designer was distraught. He had become too emotionally invested in his design, and it got in the way of his job. He ended up parting ways with the team and moving on. Unfortunately, the process of fast iteration affected more than the product itself.

Release Early, Release Often

Eric Raymond, in his famous work, The Cathedral and the Bazaar, wrote the words organizations like Netflix live by: release early, release often. At the time, Raymond was referring to software created in the open commons, by thousands of people, with little or no delivery deadline.

This dictum now applies equally well to web-application development. Fast iterations provide the freedom to innovate because teams can test more features. This reduces the risk of each tested feature. However, it takes getting used to and not everyone adjusts so easily. As with most design processes, teams get the most benefit from quick iterations when they fully embrace the process.