BlogmyQuery – BMQ

When Albert Einstein was a professor at Princeton University in the 1940s, there came the time for the final exam of his physics class. His assistants passed the exam forms to the hundreds of students, and the hall was dead silent. One of the assistants suddenly noticed something was wrong.

She approached Einstein and told him that a mistake had been made with the exam form and that the questions were the same as those in the previous year’s exam. Einstein glanced over the exam form and said that it was OK. He explained that physics had changed so much in the last year that the answers to the questions were now different.

The lean startup movement, like Einstein’s physics exam, talks about the same things that UX people have talked about for decades. The difference is that people are now listening. Lean UX is an approach that quickly followed the lean startup movement. It is not a new thing. It’s just a new name for things that were always around. The difference is in the packaging of these ideas.

One other factor that has changed dramatically is the audience. Entrepreneurs and startup founders have always been asking themselves how to develop great products. The answer that UX practitioners, usability professionals and UX researchers have been giving them was too complicated. UX people (me included) have been using disastrous jargon that only we understand. We have been talking about usability tests, personas, field studies and areas of interest in eye-tracking studies.

The lean startup answer to the same question uses plain language that people understand. When I say, “We need to conduct a contextual inquiry,� I usually get a deer-in-the-headlights reaction. When a lean startup person says they are “getting out of the building,� it is a whole different story. We mean the same thing; we use different words.

Does it matter? I think it does. Who would have thought that startup companies would be looking for UX people and UX founders, and would become interested in doing usability testing, iterative design and customer interviews?

This article takes the principles of the lean startup and suggests their UX research equivalents. Hopefully, it sheds some light on why the lean startup concept is so very well accepted in the entrepreneurial world and why startups suddenly want to do UX research and design.

Validated Learning And Usability Testing

The lean startup movement claims that startups exist not just to make stuff, but to learn how to build sustainable businesses. This learning can be validated scientifically by running frequent experiments that enable entrepreneurs to test each element of their vision, as outlined by Eric Ries in his book The Lean Startup. In my interview with Ries (embedded below), the most familiar voice of the lean startup movement, for my book It’s Our Research, he calls for entrepreneurs to double-check their assumptions to verify that they are right. He determines that validated learning exists to help entrepreneurs test which elements of their vision are brilliant and which are crazy.

In the UX world, we call in the product development people to evaluate their design assumptions in usability tests. We urge them to ask users to complete tasks while using the think-aloud protocol and to identify usability problems.

An interview with Eric Ries about getting stakeholder buy-in for UX research and how it relates to the Lean Startup ideas.

When entrepreneurs hear “validated learning,� they can see the benefit. They understand that this concept refers to proving or disproving their assumptions. When they hear “usability testing,� they associate it with a time-consuming, money-eating, academically oriented project.

Validated learning: You believe you’ll find a new continent if you keep sailing west. So, you test your idea and verify the route using scientific methods and measurements.

Build-Measure-Learn And Think-Make-Check

The fundamental activity of a startup is to turn ideas into products, to measure how customers respond and then to learn whether to pivot or persevere. All successful startup processes should be geared to accelerate that feedback loop. As Ries explains, the feedback loop includes three primary activities: build (the product), measure (data) and learn (new ideas).

Eric Ries’s Build-Measure-Learn feedback loop and the Think-Make-Check UX cycle.

The lean UX approach calls for a slightly different cycle: Think-Make-Check. The difference, according to Janice Fraser (cofounder and first CEO of Adaptive Path), is that this latter feedback loop incorporates your own thoughts as a designer, not just ideas learned through measurement. Janice, who now leads LUXr, indicates that the pattern of a lean startup is an endless loop consisting of two steps: Prove-Improve, Prove-Improve, Prove-Improve. This means that you design something, learn about it, make it better, learn again and so on. There is no room for people who are afraid to put their creations on the line for testing. These two feedback loops are very similar and are making a lot of sense to people in both the entrepreneurial and the UX worlds.

Build-Measure-Learn: How do you build the fastest ship? You try to build and test your hypothesis; you measure the result; and then you learn new knowledge that you can bring to your next ship design.

MVP, And “Test Early And Often�

The minimum viable product (MVP), as Ries explains it, is a version of the product that enables a full turn of the Build-Measure-Learn loop with a minimum amount of effort and the least amount of development time. How many times have UX people told their stakeholders that for every dollar spent on solving a problem during product design, $10 would be spent on the same problem during development, and $100 if the problem had to be solved after the product is released?

We’ve known for years that product prototypes are to be evaluated early in the development process (not just prior to launch). We’ve also known that these evaluations are most valuable if they are repeated throughout the process. The MVP is, in fact, an early prototype that serves as a tool to learn and test the team’s assumptions.

MVP: You want to build a huge ship, but instead of building the ship right from the beginning, you start by testing your idea with minimal design to see if it floats.

Pivot And Iterate

To use the analogy of a basketball “pivot,� one foot of a business is always firmly rooted in what the team has learned so far, while the other foot is moving and exploring new ideas for the business. Instead of taking the big risks of developing something huge, lean startups take small steps forward, developing things and pivoting to better directions. This way, if they fail, the fall will be less painful and will allow them to bounce back and continue. On the other hand, if they had climbed a big cliff, the potential fall would be deadly.

This reminds me of why we pitch for an iterative design process or for using the RITE methodology (rapid iterative testing and evaluation). Many product development decision-makers feel that the best time to conduct a usability test is near launch time, when things look good and are “ready� for users to play with. Many UX research practitioners know that when they agree to conduct a usability test right before a product is launched, especially if this is the first usability test for the product, the following is most likely to happen:

1. The study will result in a long list of defects (i.e. usability problems);
2. The product team will be presented with a long list of issues exactly when they are trying to shorten the list of issues;
3. Only the easiest problems to fix will be taken care of;
4. The most important problems will be ignored and the product will be launched;
5. By the time the team is ready to start working on the next version, there’s already a long list of new features to be developed, leaving the usability issues low down on (or off) the priority list.

The solution to all of this is to adopt an iterative design process that involves fast rounds of small-scale usability tests. Jakob Nielsen has been preaching this for years now. And then along comes Eric Ries, who talks in the most natural way about pivoting companies, directions, customer segments and design. People don’t iterate, they pivot.

Pivot: You want to defeat your opponent, but it is difficult to win instantly by launching a full-scale attack in one shot. The proper way would be to advance and attack step by step, always keeping one foot on the ground and ever ready to bounce back in case an attack is not successful.

Customer Development And Fieldwork

The term “customer development� was coined by Stanford University professor Steve Blank, one of the fathers of the lean startup movement. Customer development means developing your own understanding of who your customers are, what they are like and what their needs are. This is done through an approach guided by the mantra “Get out of the building.� This mantra urges entrepreneurs to interview potential customers, to observe them in their own environment and to try to make sense of it. What a revelation to our UX research ears, huh? We UX people have been getting out of the building for a living for decades now. We call it by different names: ethnography, fieldwork, generative research, exploratory research, discovery research, user research, design research. Phew!

Customer development: You want to trade with a country in the Far East. However, when you finally get to talking with the people of the country, you realize that they prefer to trade with your scientific equipment rather than your gold coins.

The Bottom Line

The lean startup movement, like the story of Einstein’s physics exam, talks about the same things that UX people have talked about for decades. The difference is that people are now listening. The lean startup movement, followed by the lean UX approach, did not reveal any new UX concept. Lean startup thought-leaders do a terrific job and do an awesome service to UX people who struggle to get buy-in for design thinking and UX research.

The secret sauce of lean startup people is that they advocate for user experience research and design as one of the primary solutions to their business problems, and they do it using plain language. I highly encourage UX practitioners to closely monitor the developments and thought-leadership in the lean startup world to see how they can use what they learn in their own organizations, “lean� or not.

Learn More About The Lean Startup Movement

Books

• The Lean Startup, Eric Ries (Crown Business: 2011)
• The Startup Owner’s Manual, Steve Blank and Bob Dorf (K&S Ranch: 2012)
• The Entrepreneur’s Guide to Customer Development, Brant Cooper and Patrick Vlaskovits (2010)
• Running Lean, Ash Maurya (O’Reilly Media: 2012)

Videos

• “An Interview With Eric Ries,â€� Tomer Sharon (18:06 minutes)
  About UX research and the lean startup movement.
• “An Interview With Steve Blank,â€� Tomer Sharon (18:01 minutes)
  About UX research and customer development.
• “A Conversation With Lane Halley,â€� Tomer Sharon (19:05 minutes)
  About lean UX.
• “An Interview With Jeff Gothelf,â€� Tomer Sharon (8:21 minutes)
  About lean UX.
• “Crushing the Boulder: User Experience and the Lean Startup,â€� Janice Fraser, MX 2011 conference (40:37 minutes).
• “The Lean Startup,â€� Eric Ries, Authors@Google (58:09 minutes)

Illustrations by Calvin C. Chan, (@calvincchan), UX designer, Hong Kong.

(al)

© Tomer Sharon for Smashing Magazine, 2012.

When Albert Einstein was a professor at Princeton University in the 1940s, there came the time for the final exam of his physics class. His assistants passed the exam forms to the hundreds of students, and the hall was dead silent. One of the assistants suddenly noticed something was wrong.

She approached Einstein and told him that a mistake had been made with the exam form and that the questions were the same as those in the previous year’s exam. Einstein glanced over the exam form and said that it was OK. He explained that physics had changed so much in the last year that the answers to the questions were now different.

The lean startup movement, like Einstein’s physics exam, talks about the same things that UX people have talked about for decades. The difference is that people are now listening. Lean UX is an approach that quickly followed the lean startup movement. It is not a new thing. It’s just a new name for things that were always around. The difference is in the packaging of these ideas.

One other factor that has changed dramatically is the audience. Entrepreneurs and startup founders have always been asking themselves how to develop great products. The answer that UX practitioners, usability professionals and UX researchers have been giving them was too complicated. UX people (me included) have been using disastrous jargon that only we understand. We have been talking about usability tests, personas, field studies and areas of interest in eye-tracking studies.

The lean startup answer to the same question uses plain language that people understand. When I say, “We need to conduct a contextual inquiry,� I usually get a deer-in-the-headlights reaction. When a lean startup person says they are “getting out of the building,� it is a whole different story. We mean the same thing; we use different words.

Does it matter? I think it does. Who would have thought that startup companies would be looking for UX people and UX founders, and would become interested in doing usability testing, iterative design and customer interviews?

This article takes the principles of the lean startup and suggests their UX research equivalents. Hopefully, it sheds some light on why the lean startup concept is so very well accepted in the entrepreneurial world and why startups suddenly want to do UX research and design.

Validated Learning And Usability Testing

The lean startup movement claims that startups exist not just to make stuff, but to learn how to build sustainable businesses. This learning can be validated scientifically by running frequent experiments that enable entrepreneurs to test each element of their vision, as outlined by Eric Ries in his book The Lean Startup. In my interview with Ries (embedded below), the most familiar voice of the lean startup movement, for my book It’s Our Research, he calls for entrepreneurs to double-check their assumptions to verify that they are right. He determines that validated learning exists to help entrepreneurs test which elements of their vision are brilliant and which are crazy.

In the UX world, we call in the product development people to evaluate their design assumptions in usability tests. We urge them to ask users to complete tasks while using the think-aloud protocol and to identify usability problems.

An interview with Eric Ries about getting stakeholder buy-in for UX research and how it relates to the Lean Startup ideas.

When entrepreneurs hear “validated learning,� they can see the benefit. They understand that this concept refers to proving or disproving their assumptions. When they hear “usability testing,� they associate it with a time-consuming, money-eating, academically oriented project.

Validated learning: You believe you’ll find a new continent if you keep sailing west. So, you test your idea and verify the route using scientific methods and measurements.

Build-Measure-Learn And Think-Make-Check

The fundamental activity of a startup is to turn ideas into products, to measure how customers respond and then to learn whether to pivot or persevere. All successful startup processes should be geared to accelerate that feedback loop. As Ries explains, the feedback loop includes three primary activities: build (the product), measure (data) and learn (new ideas).

Eric Ries’s Build-Measure-Learn feedback loop and the Think-Make-Check UX cycle.

The lean UX approach calls for a slightly different cycle: Think-Make-Check. The difference, according to Janice Fraser (cofounder and first CEO of Adaptive Path), is that this latter feedback loop incorporates your own thoughts as a designer, not just ideas learned through measurement. Janice, who now leads LUXr, indicates that the pattern of a lean startup is an endless loop consisting of two steps: Prove-Improve, Prove-Improve, Prove-Improve. This means that you design something, learn about it, make it better, learn again and so on. There is no room for people who are afraid to put their creations on the line for testing. These two feedback loops are very similar and are making a lot of sense to people in both the entrepreneurial and the UX worlds.

Build-Measure-Learn: How do you build the fastest ship? You try to build and test your hypothesis; you measure the result; and then you learn new knowledge that you can bring to your next ship design.

MVP, And “Test Early And Often�

The minimum viable product (MVP), as Ries explains it, is a version of the product that enables a full turn of the Build-Measure-Learn loop with a minimum amount of effort and the least amount of development time. How many times have UX people told their stakeholders that for every dollar spent on solving a problem during product design, $10 would be spent on the same problem during development, and $100 if the problem had to be solved after the product is released?

We’ve known for years that product prototypes are to be evaluated early in the development process (not just prior to launch). We’ve also known that these evaluations are most valuable if they are repeated throughout the process. The MVP is, in fact, an early prototype that serves as a tool to learn and test the team’s assumptions.

MVP: You want to build a huge ship, but instead of building the ship right from the beginning, you start by testing your idea with minimal design to see if it floats.

Pivot And Iterate

To use the analogy of a basketball “pivot,� one foot of a business is always firmly rooted in what the team has learned so far, while the other foot is moving and exploring new ideas for the business. Instead of taking the big risks of developing something huge, lean startups take small steps forward, developing things and pivoting to better directions. This way, if they fail, the fall will be less painful and will allow them to bounce back and continue. On the other hand, if they had climbed a big cliff, the potential fall would be deadly.

This reminds me of why we pitch for an iterative design process or for using the RITE methodology (rapid iterative testing and evaluation). Many product development decision-makers feel that the best time to conduct a usability test is near launch time, when things look good and are “ready� for users to play with. Many UX research practitioners know that when they agree to conduct a usability test right before a product is launched, especially if this is the first usability test for the product, the following is most likely to happen:

1. The study will result in a long list of defects (i.e. usability problems);
2. The product team will be presented with a long list of issues exactly when they are trying to shorten the list of issues;
3. Only the easiest problems to fix will be taken care of;
4. The most important problems will be ignored and the product will be launched;
5. By the time the team is ready to start working on the next version, there’s already a long list of new features to be developed, leaving the usability issues low down on (or off) the priority list.

The solution to all of this is to adopt an iterative design process that involves fast rounds of small-scale usability tests. Jakob Nielsen has been preaching this for years now. And then along comes Eric Ries, who talks in the most natural way about pivoting companies, directions, customer segments and design. People don’t iterate, they pivot.

Pivot: You want to defeat your opponent, but it is difficult to win instantly by launching a full-scale attack in one shot. The proper way would be to advance and attack step by step, always keeping one foot on the ground and ever ready to bounce back in case an attack is not successful.

Customer Development And Fieldwork

The term “customer development� was coined by Stanford University professor Steve Blank, one of the fathers of the lean startup movement. Customer development means developing your own understanding of who your customers are, what they are like and what their needs are. This is done through an approach guided by the mantra “Get out of the building.� This mantra urges entrepreneurs to interview potential customers, to observe them in their own environment and to try to make sense of it. What a revelation to our UX research ears, huh? We UX people have been getting out of the building for a living for decades now. We call it by different names: ethnography, fieldwork, generative research, exploratory research, discovery research, user research, design research. Phew!

Customer development: You want to trade with a country in the Far East. However, when you finally get to talking with the people of the country, you realize that they prefer to trade with your scientific equipment rather than your gold coins.

The Bottom Line

The lean startup movement, like the story of Einstein’s physics exam, talks about the same things that UX people have talked about for decades. The difference is that people are now listening. The lean startup movement, followed by the lean UX approach, did not reveal any new UX concept. Lean startup thought-leaders do a terrific job and do an awesome service to UX people who struggle to get buy-in for design thinking and UX research.

The secret sauce of lean startup people is that they advocate for user experience research and design as one of the primary solutions to their business problems, and they do it using plain language. I highly encourage UX practitioners to closely monitor the developments and thought-leadership in the lean startup world to see how they can use what they learn in their own organizations, “lean� or not.

Learn More About The Lean Startup Movement

Books

• The Lean Startup, Eric Ries (Crown Business: 2011)
• The Startup Owner’s Manual, Steve Blank and Bob Dorf (K&S Ranch: 2012)
• The Entrepreneur’s Guide to Customer Development, Brant Cooper and Patrick Vlaskovits (2010)
• Running Lean, Ash Maurya (O’Reilly Media: 2012)

Videos

• “An Interview With Eric Ries,â€� Tomer Sharon (18:06 minutes)
  About UX research and the lean startup movement.
• “An Interview With Steve Blank,â€� Tomer Sharon (18:01 minutes)
  About UX research and customer development.
• “A Conversation With Lane Halley,â€� Tomer Sharon (19:05 minutes)
  About lean UX.
• “An Interview With Jeff Gothelf,â€� Tomer Sharon (8:21 minutes)
  About lean UX.
• “Crushing the Boulder: User Experience and the Lean Startup,â€� Janice Fraser, MX 2011 conference (40:37 minutes).
• “The Lean Startup,â€� Eric Ries, Authors@Google (58:09 minutes)

Illustrations by Calvin C. Chan, (@calvincchan), UX designer, Hong Kong.

(al)

© Tomer Sharon for Smashing Magazine, 2012.

UX practitioners, both consultants and in house, sometimes conduct research. Be it usability testing or user research with a generative goal, research requires planning. To make sure product managers, developers, marketers and executives (let’s call them stakeholders) act on UX research results, planning must be crystal clear, collaborative, fast and digestible. Long plans or no plans don’t work for people. You must be able to boil a UX research plan down to one page. If you can’t or won’t, then you won’t get buy-in for the research and its results.

This article addresses one key aspect of planning UX research: the one-page plan document. Before we get to that, we’ll briefly discuss the benefits of research planning and identify the audience of a research planning document.

(Image: Patrick Hoesly)

A word about stakeholders. A stakeholder in the UX world is a code name for the people who UX practitioners work with. These are our clients, whether internal or external to our organization. These are people who need to believe in what we do, act on research results, and fund and sponsor future research. We all have a stake in product development. They have a stake in UX research.

The Benefits Of Research Planning

Very generally speaking, UX research can answer two types of questions:

1. What’s useful?
   What do people need? Who is the target audience?
2. What’s usable?
   Does the design work for people, and how it can be improved?

Dozens of research methodologies could be implemented to answer these and more specific questions, and it is up to designers, researchers and their teams to decide what works best for them and when is the right time to answer their questions.

Here are the benefits of planning UX research:

• Get a better feel of stakeholders.
  A written plan helps you identify what works and doesn’t work for people, and what questions they are trying to answer.
• Engage stakeholders.
  A study plan ensures they are properly involved with the study and its results. If there’s no written plan, then there’s a greater chance that stakeholders won’t feel engaged.
• Writing things down helps you.
  When you put things in writing, they look very different than how you imagined them when they were just thoughts in your head. Always have a written study plan, even if you don’t share it with anyone else.

Now, let’s quickly identify the target audience for the research planning document.

Who Are You Planning For? Who Are The Stakeholders?

As with every product or service, the best offering comes from carefully identifying the target audience, their needs and their wants. Different UX research stakeholders are interested in different aspects of a research plan:

• Product managers and software engineers are mostly interested in the study’s goal, research questions and schedule. In some cases, they are also interested in the criteria for participants. These stakeholders are usually interested in goals and questions because these determine the content of the study and its focus. They are interested in the schedule to make sure it enables them to make timely design, business and development decisions. Criteria for participants interest them when the product targets a very specific demographic and they want to make sure participants are representative of that demographic.
• Managers and executives are probably interested in the study’s goal and the overall cost of the study, because they are likely sponsoring the study. Usually, their bandwidth does not allow them more than that.
• You! The plan is mostly for you. As soon as you put your thoughts in writing, something happens, and you find holes in them. These holes help you improve the plan. A written plan also helps you focus and better prepare for the study. The fact of the matter is that if you can’t boil your plan down to a page, you probably don’t really understand it.

Now that we’ve discussed why a planning document is important and who it is for, let’s get to the nitty gritty of the document.

The Plan That Stakeholders Love: The One-Pager

The users of a research plan love brevity and appreciate succinct definitions of what will happen, why, when and with whom. Here are the sections that go in a one-page research plan:

• Title
  The title should combine the thing you’re studying and the methodology; for example, “Monster.com field study� or “XYZ Phone data-entry usability test.� Sometimes mentioning the target audience of the study is also appropriate; for example, “Whitehouse.com news page interviews with senior citizens.�
• Author and stakeholders
  State your full name, title and email address on one line. After you get the stakeholders’ buy-in for the plan, add their details as well — the research belongs to everyone now.
• Date
  Update it whenever the plan is updated.
• Background
  Describe what led to this study. Discuss the recent history of the project. Be brief, no more than five lines.
• Goals
  Briefly state the high-level reason (or reasons) for conducting this study. Try to phrase it in one sentence. If that wouldn’t make sense, create a numbered list of very short goal statements. If you have more than three to four goals, you are either aiming too high (meaning you have too many goals) or repeating yourself.
• Research questions
  These are the specifics, the core of your plan. Provide a numbered list of questions that you plan to answer during the study. It is extremely important that your stakeholders understand that you will not necessarily be asking the study participants these questions. As a rule of thumb, have no more than seven to ten questions, preferably around five. Later on, you will construct your study script to answer these questions. An effective way to think about research questions is to imagine that they are the headings in the study’s summary.
• Methodology
  In an academic environment, this section has one primary goal: to provide as many details as other researchers need in order to repeat the exact same study. In practice, the goal of the methodology section is to briefly inform the stakeholders of what will happen, for how long and where.
• Participants
  Provide a list of the primary characteristics of the people you will be recruiting to participate in the study. Have a good reason for each and every characteristic. If you have two participant groups, describe both groups’ characteristics in lists or in a table. Append a draft form that you’ll use to screen participants.
• Schedule
  Inform stakeholders of at least three important dates: when recruiting starts, when the study will take place, and when they can expect results. Large research projects require more scheduling details. For example, if the study involves travel to another city or country, more dates might be required for on-site preparation and meetings or for analysis workshops.
• Script placeholder
  When a full study script is ready, it will appear under this title. Until then, all you need is a heading with a “TBD� indication.

A Sample UX Research Plan

XYZ Phone Data-Entry Usability Test

By John Smith-Kline, Usability Researcher, jskline@example.com

Stakeholders: Wanda Verdi (PM), Sam Crouch (Lead Engineer)

Last updated: 13 January 2012

Background
Since January 2009, when the XYZ Phone was introduced to the world, particularly after its market release, journalists, bloggers, industry experts, other stakeholders and customers have privately and publicly expressed negative opinions about the XYZ Phone’s keyboard. These views suggest that the keyboard is hard to use and that it imposes a poor experience on customers. Some have claimed this as the main reason why the XYZ Phone will not succeed among business users. Over the years, several improvements have been made to data entry (such as using horizontal keyboards for most features), to no avail.

Goals
Identify the strengths and weaknesses of data entry on the XYZ Phone, and provide opportunities for improvement.

Research questions

1. How do people enter data on the XYZ Phone?
2. What is the learning curve of new XYZ Phone users when they enter data?
3. What are the most common errors users make when entering data?

Methodology
A usability study will be held in our lab with 20 participants. Each participant session will last 60 minutes and will include a short briefing, an interview, a task performance with an XYZ Phone and a debriefing. Among the tasks: enter an email subject heading, compose a long email, check news updates on CNN’s website, create a calendar event and more.

Participants
These are the primary characteristics of the study’s participants:

• Business user,
• Age 22 to 55,
• Never used an XYZ Phone,
• Expressed interest in learning more about or purchasing an XYZ Phone,
• Uses the Web at least 10 hours a week.

[Link to a draft screener]

Schedule

• Recruiting: begins on November 12
• Study day: November 22
• Results delivery: December 2

Script
TBD

Recap

A short plan that you and your stakeholders prepare together is key to a successful start of a UX research project.

• Boil down your collective knowledge, agreements and understanding of what will happen, why, with whom and when.
• Set the right expectations among stakeholders.
• Try to keep the plan to one page.
• Secure buy-in for the UX research by making it a team effort.
• The core of the plan is the list of questions you are trying to answer. Choose the right ones.

Happy planning!

(al) (fi) (il)

© Tomer Sharon for Smashing Magazine, 2012.