In survey design, there is a popular practice of using open-ended questions to add depth to survey responses. It’s a smart move, given the risks posed by quantitative-only results that often fail to probe the why and the how behind participants’ responses (‘the what’).
This mixed approach to question design is the survey’s response to gathering big and thick data, and offers a number of benefits:
Provides context/a backdrop against which to interpret responses to other questions and/or the survey, overall
Differentiates participant perspectives by adding depth and nuance to responses
Explores the meanings participants assign to topics, questions and responses
Unpacks participants’ reported experiences to understand their implications
Invites unexpected ideas, perspectives and feedback not captured by pre-determined response options
Highlights differences in the ways participants interpret topics and questions, and the impact of those differences on responses
Clarifies response patterns observed in quantitative data (i.e., what motivated participants to respond in certain ways)
Identifies important questions and responses – beyond statistical significance
Suffice it to say, open-ended questions can bring significant clarity and meaning to a sea of numbers.
Like all survey questions, open-ended questions are not inherently good (or bad). How these questions are used, the way they are structured, and what they ask all contribute to the quality (and accuracy) of the insights they generate. Once you have established that a survey is an appropriate means for exploring your research questions, there are some key steps you can take to access the power of your thick data.
Start with the end in mind
As with all data collection, survey design, and specifically the creation of qualitative items, starts with where you want to finish: why are you collecting this data? how do you plan to use it?
Starting with how you would like to use the data by considering what you hope the data will ‘say’ serves the dual purpose of:
vetting your choice to survey (is survey the right method?); and,
guiding the structure of your survey to gather useable quantitative and qualitative data that are in conversation with one-another.
As you think about how you plan to use the data, identify what you think you know, and what you know you don’t. Where are you making assumptions? How might you unpack those assumptions by soliciting targeted input (via the survey)?
Engage in some scenario-planning by imagining the kinds of responses you might get – what would these responses ‘tell’ you? Would they fulfill your purpose? Can you use these data in the way you hoped?
Quantitative first!
While the balance of questions in a survey should favour quantitative items, the specific ratio will depend on what you are asking. Given the quantitative questions will be the limiting factor (i.e., they are the most limited in their ability to solicit depth and detect nuance), it is usually a good idea to start with those.
Once you have a set of quantitative questions, use qualitative questions as probes:
Identify where there are gaps in what you can ask using quantitative questions
Look for quantitative responses that might smooth over important differences in perspectives/experiences
Anticipate how different participants might interpret quantitative questions (participants will all interpret quantitative questions differently - there is no such thing as ‘objective’ questions!), and how this might influence the answers you get. Qualitative follow-up questions can help clarify what participants were thinking/how they understood the question, to guide your interpretation of their answers.
It can also be helpful to think about the survey as an interview. If you were asking a participant these questions, where might you compelled to ask follow-up probes like ‘why?’ or ‘can you give me an example?’. These are often good opportunities to incorporate open-ended options.
Be ruthless about when these questions will give you game-changing insights (i.e., need to know vs. nice to know), since following up almost any question with ‘why?’ can be interesting.
Remember…you have to code what you collect
The limitations of quantitative questions can sometimes prompt a reactive “FINE! I’ll make everything qualitative!” response during survey design. In addition to being another example of the binary (either/or) thinking prevalent in quantitative research, this compromises the quality of data collected.
That’s right: using only open-ended questions generates low-quality data. WHAT?
WHY? Because you have to code it. And coding is ambiguous (there’s no single ‘right way’), labour-intensive (you have to FOCUS), and time-consuming (it literally takes forever).
This coding effort is absolutely worthwhile when the responses are going to offer insights that you could not otherwise access, and that speak directly to the purpose of your survey (i.e., why you are collecting data in the first place / what you plan to do with it).
When deciding to code or to quantify consider the kinds of responses you would get to an open-ended question:
Can you anticipate natural categories that you would likely sort responses into? If so, consider using a quantitative question with those categories instead.
Is it the whole answer, or a certain aspect of the response that offers the most insight? If it is a particular aspect of the answer, think about how you might pair a quantitative question with an open-ended follow-up, and limit the qualitative response to what will deliver the most value.
Does your survey include concepts that are typically treated in a certain way that gives rise to assumptions you want to explore? It is often worthwhile to code open-ended responses that are likely to challenge unquestioned ideas by introducing alternative meanings/ways of understanding.
Quantitative and qualitative questions are friends
While you often find people sitting in one camp or the other, the truth is, quantitative and qualitative questions are complements, not opposites. Each has its own strengths, and when used together, your data instantly gain meaning.
To be clear, this means using quantitative and qualitative questions together on purpose – not just including both types in the survey.
Quantitative and qualitative questions are in conversation.
Rather than thinking about what the quantitative questions will ask versus what the qualitative questions will ask, consider what each might contribute to exploring the same questions. How might you capitalize on the benefits of quantitative questions (i.e., easier to analyze data) and use qualitative probes to add depth?
Tip: An item matrix is a great way to check that your questions ‘hang together’ (i.e., the data they collect make sense together, and seem to be about the same things), and that the data generated will offer the insights you hoped, and be useable in the way you planned.