Good or Bad? How to Tell Where You Market Research Stands

Aug 24

A version of this post was included in my super-fun weekly email for marketers with tips, tricks, and stories.

Sign up for emails and never miss out on the fun!

I’m sure you’ve seen it…a garish color combination on a website, a complex banner design, a cluttered interface, or pop-ups that make you want to shut the whole thing down.

Bad design is easy to spot.

The same can’t be said for market research. Unlike design, it’s difficult to distinguish between good market research and bad market research.

So, how can you be confident in the market research and consumer insights you’re discovering about your brand? It’s so critical to make this distinction because if you’re basing your market strategy on poor information, you’re going to be shooting in the dark.

Just recently, I was on a call with a company that told me their health food brand was targeting women over 60. When I heard how they got their data, though, it was clear the data wasn’t valid.

They had surveyed their email list of subscribers. The problem is that the people on this email list…

Bought the product in store
Felt strongly enough about it to figure out how to join the email list
Took the time to read the emails
Took the time to respond to a survey

It’s no surprise that the demographics skewed older, as younger people are less likely to take these steps to join the email list and answer a survey emailed out. That’s unless the brand is socially popular in their crowd AND has limited runs that they fear missing out on.

Basing a marketing strategy off this data would have been disastrous for this company—they literally could have gone out of business.

So today’s post is about how to tell if the data you’re working with is reliable enough that you can confidently base your business decisions around it. Because I want to squeeze the most out of your market research— boosting customer engagement, sales figures, and other forms of ROI from your market research.

This post outlines five elements to look at when determining if you can trust the results of your market research. These are factors to check your market research against when evaluating its quality.

Element #1: Combination of Qual & Quant Research Methods Used

If you’re going to pay attention to ONE thing alone to determine the quality of research, this one’s it.

Many people rely on surveys and data analytics to get to know their consumers. The thing is that these quant metrics only go so far. Quantitative data shows you WHAT your customers are doing, but a good market research study will tell you WHY they’re doing it.

Combining qual and quant market research methods is the best way to ensure success for your project. At Bixa, we often use this “sandwich” approach to primary research. It’s called this because we sandwich a large-scale survey in between rounds of qualitative research:

Qual: We will sequence our approach to start with a qual method - perhaps in-depth interviews or ethnography. From there, we will create hypotheses about the audiences and what meaningful value your brand may bring. By talking to customers or observing behavior first, we will discover exactly what to ask regarding the survey. This way, we won’t be missing any key questions—because one of the worst things to happen in a survey is when you realize after spending a huge investment in it that you asked the wrong questions or missed key questions. Please don’t make this mistake.

Quant: After forming hypotheses about your audience, that’s when we call in quantitative surveys—to understand if these hypotheses scale across a larger population. This extra layer of validation helps us increase confidence in our recommendations.

Qual: Inevitably, we will glean new insights from this survey that we’ll want to dig deeper into. That’s exactly the reason we buffer the end of the survey with a round of in-depth interviews to deep-dive into specific areas we want to know more about. These are more pointed conversations that enable an experienced moderator to go down the rabbit hole with a customer, honing in on specific moments of truth in their experience that will really impact your customer engagement, conversions, etc.

Each step in a good research study is designed to build on each other and get you more new data to give you a clear foundation for any marketing strategy.

Even in usability testing, small-scale iterative testing will get you much more detailed, better data than a huge large-scale study. Check out Jakob Neisen’s research on small-scale iterative testing for the stats here. But, basically, if you do a huge study, you’ll find more of the same thing over and over again. If you do a small study, fix the top issues, do another small study, fix the top issues, and repeat this process—building up to your total large number of participants, you’ll wind up with a far better, more targeted result from the same amount of research.

Element #2: Finding the Right Audience

Finding the right participants is key to getting the right data—a skewed or biased audience can get you results that look good, but that really don’t match your ideal audience segments or ideal customer avatars.

For your recruit, make sure to spend time really looking at the recruiting screener, the questionnaire designed to disqualify audience members who don’t belong in your niche.

For example, if you’re a packaged health food product looking for qualitative recruits, you can ask applicants to provide a photo or video of what packaged health foods are in their pantry so you can see if they buy the type of foods you need. You’ll have a much greater level of confidence that you’re getting potential customers in your niche based on the characteristics of your audience.

The worst mistake you can make is going with a “friends and family” approach to recruiting because you’ll get people who are not valid customer segments.

The audience spread of its research in a quant study also says a lot about the quality of a market research. Unfortunately, market researchers and companies make the mistake of recruiting audiences with an imbalanced spread of the different personas under review. This mistake is rooted in the wrong notion that talking to a nationally representative population is the answer to accurate market research. In reality, the key is to speak to the proper spread of people who drive the buying decision.

Element #3: Quant Sample Size

When validating qualitative hypotheses with a quantitative study, you’ll want to ensure that your sample size is large enough to represent the larger population accurately. That said, oversampling is also a red flag. It’s hard to tell what’s significant if everything is significant. It can also be a waste of your valuable time and budget.

Not sure what sample size to use in your studies? Here’s a handy sample size calculator to help you calculate this.

Element #4: Research Biases

One of the easiest ways to go wrong with research is with bias. This is one of the top reasons many companies hire objective third parties to conduct their studies—because organizations often have preconceived notions of who their audiences are and what they need, and ask leading questions to verify their hunches (versus asking open-ended questions to either validate or reject full-stop).

Many organizations create fictional personas in boardrooms without stepping into the field. Now, this practice is usually well-intentioned. In some cases, these become real in their minds and cause these companies to approach their research audience with ideas already formed about what customers want or need.

These biases could reflect in your research questions, subtly (or not so subtly) leading participants to give answers that confirm your preconceptions. This may be seen as a win for your study, but remember: your market research is only as good as the authenticity and quality of the responses you get from the participants.

For instance, if we assume a tool is confusing to use, we may inadvertently write a question like this:

How confusing is this tool to use?

Very confusing
Confusing
A little confusing
Not at all confusing

A better option would be:

How do you find the functioning of this tool?

Very simple
Simple
Neutral
Confusing
Very confusing

List 3 adjectives that describe your feelings toward this tool

(if you don’t mind coding a qual response!)

Honestly, I could go on for days about bias and how it shows up in studies when we don’t even realize it. But for now, let’s just say that keeping your research objective and open-minded should be a top priority.

Element #5: Recency of Research

We live in a dynamic world. Think of how disruptive the pandemic was to all our lives. As a result, consumer behavior and marketing have witnessed more massive changes over the last 5 years than in the previous 50.

We’ve heard that COVID sped eCommerce up by 3+ years. Think of how much more commonplace it now is to do grocery shopping online, for example.

The corporate work structure transformed from clocking 9 to 5 shifts in an office to working remotely. In the same way, Zoom webinars and Google Meets have taken over boardroom meetings and even full-blown conferences.

Every industry has witnessed dramatic changes like the ones just listed. This implies that consumers’ needs, expectations, and preferences have also shifted dramatically. If an organization’s market research was done too long ago that it doesn’t capture these changes, it’s not just bad market research. It’s dangerous market research that could cost that organization a lot.

The pandemic is not the only reason for new research; our technology is constantly changing, trends are emerging, and new generations with fresh mindsets are being embraced into the workplace.

So before making decisions based on your old research study, make sure no huge life changes, technology changes, regulatory changes, generational changes, political changes, or other macro inputs have substantially impacted your customers since the last time you did research with them.

Conclusion

Although this blog post is not an exhaustive list of elements of good market research, we’ve tried to capture a few key factors that distinguish good research from bad.

Need help knowing if it’s the right time for research, or how to strategically make bad market research good? Engage a research studio like Bixa with years of industry knowledge and experience. Schedule a free consultation session.

Want more of this marketing goodness?

***

If you liked this article, you might like my blog post on Fix THIS Before Spending Another Penny On Marketing Tactics.

Sarah Weise is the CEO of award-winning marketing research agency Bixa and the bestselling author of InstaBrain: The New Rules for Marketing to Generation Z. For 15 years, Sarah has been a guide to hundreds of leading brands, including Google, IBM, Capital One, Mikimoto, PBS, and U.S. Army, to name a few. Sarah helps brands achieve a laser focus on their customers and build experiences that are downright addictive. She lectures at Georgetown University’s McDonough School of Business and speaks at conferences and corporate events worldwide.

Sarah Weise