Making Software

Using Qualitative Methods in Practice

While qualitative methods are usually applied in research settings, they can be incredibly useful in practice. In fact, they are useful in any setting where you don’t know the entire universe of possible answers to a question. And in software engineering, when is that not the case? Software testers might use qualitative methods to answer questions about inefficiencies in the human aspects of a testing procedure; project managers can use qualitative methods to understand how the social dimensions of their team might be impacting productivity. Designers, developers, and requirements engineers can use qualitative methods to get a deeper understanding of the users they serve, ensuring a closer fit between user needs and the feature list. Any time you need to analyze the dynamics between who, what, when, where, how and why, qualitative methods can help.

Using qualitative methods is a lot like being a good detective or journalist: the point is to uncover truth, and tell a story people can trust—but also realize that there are usually many truths and many perspectives. How you go about uncovering these perspectives depends a lot on the situation. In particular, you need an instinct for the social context of what you want to understand, so you know whom you can trust, what their biases are, and how their motives might affect your sleuthing. Only with this knowledge can you decide what combination of direct observation, shadowing, interviews, document analysis, diary studies, or other approaches you might take.

To illustrate, let’s go back to our bug tracker example. One of the most important factors in choosing which methods to use is you, and so there are a number of things to consider:

• Do your employees like you?

• Do they respect you?

• Do they trust you?

If the answer to any of these questions is no, you’re probably not the one to do the sleuthing. Instead, you might need to find a more impartial party. For example, a perfect role for sleuthing is an ombudsperson. Her job is to be a neutral party, a person who can see multiple perspectives to support effective communication and problem solving. If your organization has an ombudsperson, she would be a great candidate for a class on qualitative methods and could play an instrumental role in improving your workplace.

If your employees do like you, the next most important factor is your employees. Are they good communicators? Are they honest? Do they hide things? Are there rival factions on your team? Is there a culture of process improvement, or is the environment rigid and conservative? All of these social factors are going to determine the viability of applying any particular qualitative method. For example, direct observation is out of the question if your employees like to hide things, because they’ll know they’re being observed. Document analysis might work in this case, but what will your team think about their privacy? Interviews work quite well when the interviewer can establish rapport, but otherwise they’re a garbage-in, garbage-out process. The goal of a good qualitative approach to answering a question is to find ways of probing that minimize bias and maximize objectivity.

Regardless of whether you or someone else is doing the sleuthing, another important consideration is how you explain to your team what you’re sleuthing about. In all of the cases I can imagine, keeping the sleuthing a secret is a terrible idea. You’re probably asking a question because you want to solve a problem, and you’re managing a team of problem solvers: get them to help! It’s important to point out to them, however, that your agenda is to understand, not to dictate. It’s also important to say that there’s probably not a single explanation and everybody’s perspective will differ at least a little. Communicating these points creates an expectation that you’ll be seeking your informants’ perspectives and empathizing with them, which makes them more likely to reflect honestly.

Finally, qualitative methods can feel loose and arbitrary at times. How can you really trust the results of a process without structure? The trick is to realize that bias is all around you and embrace it. You as the researcher are biased, your informants are biased, and the different methods you use to understand a context are biased toward revealing certain phenomena. The more you can explicitly identify the bias around you and understand how it affects your interpretations of what you see, the more objective your findings.