Chapter 4. Qualitative Research Is Great...Except When It’s Terrible

In this chapter:

  • Learn when to do qualitative research and when to do quantitative.

  • Understand the best approach for figuring out what features to build next.

  • Learn what type of research will help you predict whether users will buy your product.

I have now spent several chapters telling you that you will burn in hell if you don’t do qualitative research on your users. I’m now going to tell you when that advice is absolute poison. Aren’t you glad you didn’t stop reading after the first few chapters?

The truth is that qualitative research isn’t right for every situation. It’s fantastic for learning very specific types of things, and it is completely useless for other things. That’s OK. The trick is to use it only in the right circumstances.

First, let’s very briefly touch on some of the differences between qualitative and quantitative research. Qualitative research is what we’ve been mainly discussing so far in this book. It typically involves interviewing or watching humans and understanding their behavior. It’s not statistically significant.

Here are a few examples of qualitative research:

  1. Contextual inquiry

  2. Usability studies

  3. Customer development interviews

Quantitative research is about measuring what real people are actually doing with your product. It doesn’t involve speaking with specific humans. It’s about the data in aggregate. It should always be statistically significant.

Here are a few examples of quantitative research:

  1. Funnel analytics

  2. A/B testing

  3. Cohort analysis

I know we haven’t gone into what any of those quantitative research methods are or how you might accomplish them. If you’re interested in learning more about these sorts of analytical tools (and you should be), you may want to check out the book Lean Analytics by Alistair Croll and Ben Yoskovitz (O’Reilly).

But none of that matters unless you understand when you would choose to use a qualitative method and when you would choose to use a quantitative method. Quantitative research tells you what your problem is. Qualitative research tells you why you have that problem.

Now, let’s look at what that means to you when you’re making product decisions.

A One-Variable Change

When you’re trying to decide between qualitative and quantitative testing for any given change or feature, you need to figure out how many variables you’re changing.

Here’s a simple example: You have a product page with a buy button on it. You want to see if the buy button performs better if it’s higher on the page without changing anything else. Which do you do? Qualitative or quantitative?

How could you possibly choose?
Figure 4-1. How could you possibly choose?

That’s right, I said this one was simple. There’s absolutely no reason to qualitatively test this before shipping it. Just get this in front of users and measure their rate of clicking on the button.

The fact is, with a change this small, users in a testing session or a discussion aren’t going to be able to give you any decent information. Honestly, they probably won’t even notice the difference. Qualitative feedback here is not going to be worth the time and money it takes to set up interviews, talk to users, and analyze the data.

More importantly, since you are changing only one variable, if user behavior changes, you already have a really good idea why it changed. It changed because the CTA button was in a better place. There’s nothing mysterious going on here.

There’s an exception! In a few cases, you are going to ship a change that seems incredibly simple, and you are going to see an enormous and surprising change in your metrics (either positive or negative). If this happens, it’s worth running some observational tests with something like UserTesting.com, where you just watch people using the feature both before and after the change to see if anything weird is happening. For example, you may have introduced a bug, or you may have made it so that the button is no longer visible to certain users.

A Multivariable or Flow Change

Another typical design change involves adding an entirely new feature, which may affect many different variables.

Here’s an example: You want to add a feature that allows people to connect with other users of your product. You’ll need to add several new pieces to your interface in order to allow users to do things like find people they know, find other interesting people they don’t know, manage their new connections, and get some value from the connections they’ve made.

Do you know these people?
Figure 4-2. Do you know these people?

Now, you could simply build the feature, ship it, and test to see how it did, much the way you made your single-variable change. The problem is that you’ll have no idea why it succeeded or failed—especially if it failed.

Let’s assume you ship it and find that it hurts retention. You can assume that it was a bad feature choice, but often I find that people don’t use new features not because they hate the concept, but because the features are badly implemented.

The best way to deal with this is to prevent it from happening in the first place. When you’re making large, multivariable changes or really rearranging a process flow for something that already exists on your site, you’ll want to perform qualitative testing before you ever ship the product.

Specifically, the goal here is to do some standard usability testing with interactive prototypes, so that you can learn which bits are confusing (note: Yes, there are confusing bits, trust me!) and fix them before they ever get in front of users.

Sure, you’ll still do an A/B test once you’ve shipped it, but give that new feature the best possible chance to succeed by first making sure you’re not building something impossible to use.

Deciding What to Build Next

Look, whatever you take from this next part, please do not assume that I’m telling you that you should ask your users exactly what they want and then build that. Nobody thinks that’s the right way to build products, and I’m tired of arguing about it with people who don’t get UCD or Lean UX.

However, you can learn a huge amount from both quantitative and qualitative research when you’re deciding what to build next.

Here’s an example: You have a flourishing social commerce product with lots of users doing lots of things, but you also have 15 million ideas for what you should build next. You need to narrow that down a bit.

This shouldn’t take more than 20 or 30 years
Figure 4-3. This shouldn’t take more than 20 or 30 years

The key here is that you want to look at what your users are currently doing with your product and what they aren’t doing with it, and you should do that with both qualitative and quantitative data.

Qualitative Approaches

  • Watch users with your product on a regular basis. See where they struggle, where they seem disappointed, or where they complain that they can’t do what they want. Those will all give you ideas for iterating on current features or adding new ones.

  • Talk to people who have stopped using your product. Find out what they thought they’d be getting when they started using it and why they stopped.

  • Watch new customers with your product and ask them what they expected from the first 15 minutes using the product. If this doesn’t match what your product actually delivers, then either fix the product or fix the first-time user experience so that you’re fulfilling users’ expectations.

Quantitative Approaches

  • Look at the features that are currently getting the most use by the highest-value customers. Try to figure out if there’s a pattern there and then test other features that fit that pattern.

  • Try a “fake” test by adding a button or navigation element that represents the feature you’re thinking of adding, and then measure how many people click on it. Instead of implementing an entire system for making friends on your site, add a button that allows people to Add a Friend, and then let them know that the feature isn’t quite ready yet while you tally up the percentage of people who are pressing the button.

Still Don’t Know Which Approach to Take?

What if your change falls between the cracks here? For example, maybe you’re not making a single-variable change, but it’s not a huge change either. Or maybe you’re making a pretty straightforward visual-design or messaging change that will touch a lot of places in the product but that doesn’t affect the user process too much.

As many rules as we try to make, there will still be judgment calls. The best strategy is to make sure that you’re always keeping track of your metrics and observing people using your product. That way, even if you don’t do exactly the right kind of research at exactly the right time, you’ll be much more likely to catch any problems before they hurt your business.

As you have hopefully figured out, I’m a huge proponent of qualitative user testing. I think it’s wonderful for learning about your users and product.

But it’s not a panacea. The fact is, there are many questions that qualitative testing either doesn’t answer well or for which qualitative testing isn’t the most efficient solution.

Unfortunately, one of the most important questions people want answered isn’t particularly well suited to qualitative testing.

If I Build It, Will They Buy?

I get asked a lot whether users will buy a product if the team adds a specific feature. Sadly, I always have to answer, “I have no idea.”

The problem is, people are terrible at predicting their future behavior. Imagine if somebody were to ask you if you were going to buy a car this year. Now, for some of you, that answer is almost certainly yes, and for others it’s almost certainly no. But for most of us, the answer is, “It depends on the circumstances.”

For some, the addition of a new feature—say, an electric motor—might be the deciding factor, but for many the decision to buy a car depends on a lot of factors, most of which aren’t controlled by the car manufacturer: the economy, whether a current car breaks down, whether we win the lottery or land that job at Goldman Sachs, etc. There are other factors that are under the control of the car company but aren’t related to the proposed feature: Maybe the new electric car is not the right size or isn’t in our price range or isn’t our style.

This is true for smaller purchases, too. Can you absolutely answer whether or not you will eat a cookie this week? Unless you never eat cookies (I’m told these people exist), it’s probably not something you give a lot of thought to. If somebody were to ask you in a user study, your answer would be no better than a guess and would possibly even be biased by the simple act of having the question asked.

Admit it, a cookie sounds kind of good right now, doesn’t it?

There are other reasons why qualitative testing isn’t great at predicting future behavior, but I’m not going to bore you with them. The fact is, it’s simply not the most efficient or effective method for answering the question, “If I build it, will they come?”

What Questions Can Qualitative Research Answer Well?

Qualitative research is phenomenal for telling you whether your users can do X. It tells you whether the feature makes sense to them and whether they can complete a given task successfully.

To a smaller extent, it can even tell you whether they are likely to enjoy performing the task, and it can certainly tell you if they hate it. (Trust me, run a few user tests on a feature they hate. You’ll know.)

This obviously has some effect on whether the user will do X, since he’s a lot more likely to do it if it isn’t annoying or difficult. But it’s really better at predicting the negative case (i.e., the user most likely won’t use this feature in its present iteration) than the positive one.

Sometimes qualitative research can also give you marginally useful feedback if your users are extremely likely or unlikely to make a purchase. For example, if you were to show them an interactive prototype with the new feature built into it, you might be able to make a decent judgment based on their immediate reactions if all of your participants were exceptionally excited or incredibly negative about a particular feature.

Unfortunately, in my experience, this is the exception rather than the rule. It’s rare that a participant in a study sees a new feature and shrieks with delight or recoils in horror. Although, to be fair, I’ve seen both.

What’s the Best Way to Answer This Question?

Luckily, this is a question that can be pretty effectively answered using quantitative data, even before you build a whole new feature. A lot of companies have had quite a bit of success with adding a “fake” feature or doing a landing-page test.

For example, one client who wanted to know the expected purchase conversion rate before it did all the work to integrate purchasing methods and accept credit cards simply added a buy button to each of its product pages. When a customer clicked the button, he was told the feature was not quite ready, and the click was registered so that the company could tell how many people were showing a willingness to buy.

By measuring the number of people who thought they were making a commitment to purchase, the client was able to estimate more effectively the number of people who would actually purchase if given the option.

The upshot is that the only really effective way to tell if users will do something is to set up a test and watch what they actually do, and that requires a more quantitative testing approach.

Are There Other Questions You Can’t Answer Qualitatively?

Yep. Tons of them.

The most important thing to remember when you’re trying to decide whether to go with qualitative or quantitative is to ask yourself whether you want to know what is happening or why that particular thing is happening.

If you want to measure something that exists, like traffic or revenue or how many people click on a particular button, then you want quantitative data. If you want to know why you lose people out of your purchase funnel or why people all leave once they hit a specific page, or why people seem not to click that button, then you need qualitative.

Go Do This Now!

  • Go from quant to qual: Try looking at your funnel metrics to understand where you are having problems, and then run a qualitative study to understand why users are having problems in those places.

  • Go from qual to quant: Try making a change based on your qualitative research learnings and measuring that change with an A/B test.

Get UX for Lean Startups now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.