When it comes to developing a purely representative sample, telephone research is still far better than online research. Numerous studies have shown this from a variety of different perspectives, leaving no real question about the comparative representativeness of the two methods.
Yes, telephone response rates are horrendous. Pew Research Center recently published a study that showed phone response rates are about 9% today, down from 36% in 1997. The Centers for Disease Control now estimates that 41% of Americans don’t have a landline (which is far higher than the dwindling proportion who don’t use the Internet).
I deal with this issue with clients all the time – should we use phone or should we use an online panel in order to get a representative sample of our target audience? My answer is that without question, telephone is the most accurate way of representing your target group…but that doesn’t necessarily mean telephone is the best way.
Quite bluntly, the issue often comes down to one major factor: cost. Typically, online surveys are vastly less expensive than phone surveys. This is true even if panel sample is selected by quality rather than just by who will throw the most warm bodies at a survey for the lowest price. The cost gap is even larger if the phone sample includes cell phones, which are considerably more expensive to dial and tend to provide even lower response rates than landlines. And the gap increases as the incidence of the study decreases.
Purists will cry “cost doesn’t matter – it’s representativeness that matters.” And to some extent, they’re right. But here’s the reality: we all have budgets. And those budgets are all limited.
Let’s say your research department (or your client) has an annual budget of $250,000 for primary research. Let’s further say that your typical study can either be conducted for $25,000 using an online panel, or for $50,000 by phone. You have two choices:
- Conduct five studies by phone and address five key issues for the year
- Conduct ten studies online and address ten key issues for the year
Puts a different spin on the phone-versus-online issue, doesn’t it? Using these figures (which admittedly are made up out of thin air), you can afford to do twice as much research if you do your work online as if you go the purist route and do it by phone.
Of course, not all studies will show this large of a gap between the costs of phone and online. But at the same time, many will show a much larger gap. Not long ago, I handled a project for a client where the incidence was less than 10%, and the sample universe was limited to the 18 – 29 age group. Out of a random-digit-dial sample, that would be about 2% of the national population. I don’t even want to think about the field costs for 1,000 completes by phone.
Now, I expect at least a few sarcastic comments along the lines of “Why don’t you just use a convenience sample at a conference, or whip out DIY, and then you can do ten times as much research for the same price?” I’m not advocating that we globally substitute low price for quality. In fact, I’m not advocating online research at all. There are times it’s worth paying for the highest quality possible, and there are certain approaches and practices that are worthless as research even if they’re free.
All I’m saying is this: cost must be one of the decision factors, because we live in the real world. If all my clients had unlimited budgets, I doubt I’d ever use another online panel. But they don’t.
In this sense, it’s similar to qualitative research. Conducting twenty focus groups is better than conducting eight. But my experience is that, unless you’re examining very different population groups or changing the stimuli from one set of groups to the next based on what you learn, twenty focus groups will provide relatively little additional benefit for more than double the cost. Yes, twenty is better than eight. No, it’s not better enough to warrant more than twice the expense.
The question for most research is not whether phone provides a more accurate, representative sample than online. That one has already been answered. It does. Without question or argument, it does. In fact, since online panels are not a random probability sample, phone isn’t just better, but the only way of providing a truly representative sample.
But the real question is this: is online panel research sufficient for what you need? Is the sacrifice in representativeness worth the ability to spend the savings on other things, such as additional studies, employee education, larger samples, deeper analysis, pre-testing, or conducting qualitative research before you do the quantitative? Can you do things to raise the quality and representativeness of the online research? (The answer to that is almost always “yes.”) At the end of the day, is the gap in statistical representativeness worth the gap in price?
In some cases, there’s no doubt that the answer is yes, absolutely – it’s worth it to spend the money for phone. In other cases, the answer will be no – online is sufficient for what you need. Not because you’re cheap or don’t care about quality, but because like everyone else, you need to take budget and time into consideration.
Until the NYT/CBS announcement, political polling was one of the last bastions of purist thought on methodology. That’s understandable, given that the polling is meant to tease out the likely winners in what are often very close races. If your candidate has 49% of the vote instead of 51%, that’s a pretty big deal. On the other hand, if your advertising awareness or customer satisfaction numbers are 49% instead of 51%, does that really impact your business model?
You need to understand enough about both methodologies to be able to make that judgment call individually for each potential project. Anyone who globally advocates one methodology over the other is missing the boat. It’s not enough just to say “phone is better” or “online is cheaper” – the balance between the advantages of each methodology is critical, as is the appropriate selection for each project.
Also critical is knowing how to get the best out of each methodology – blindly turning a panel project over to the lowest cost provider and running with whatever data results from it makes no more sense than opening the Duluth phone book, dialing a few numbers, and calling that telephone research. Panel isn’t perfect, but there are ways to get better quality data if you know how (hmmm…the wheels are already turning for another blog post).
Purists are wonderful, because their advocacy helps keep us on the straight and narrow, and makes sure we consider the hard questions. But sometimes “purist” must meet “reality” and make some tough decisions. That process appears to be happening at NYT/CBS, and it will be fascinating to see what happens as they experiment with panel sampling.