In my last post, I detailed some of the issues and concerns in online panel research (from the Grey Matter Research report More Dirty Little Secrets of Online Panel Research), including the following:
- Panels that send respondents 50 – 60 survey invitations per month
- Overly long questionnaires and/or questions that are impossible to answer
- Panels that allow respondents to complete survey after survey in one sitting
- Absurdly low incentive amounts
- Surveys that are in the field a mere matter of hours
- Panels that insert their own questions before yours
- Studies that were irrelevant to their target population
- Panels that carry advertising on their website or give respondents sales messages under the heading of a “survey”
There are some really, really bad panels out there. The question is, how can we as researchers navigate these treacherous waters? In short, what can we do to avoid these bad panels and practices?
First and foremost, we must pay attention to the field. Quite frankly, fieldwork is not exciting. One might even say it’s boring (apologies to field specialists out there!). But it is also critically important. It’s much like the chassis in a car – forgotten under the sleek exterior and plush interior, it’s still what holds the entire car together. It’s not enough to hand your project over to a research vendor or a panel company and wait for the resulting data.
Decide what you actually value in a panel. Do you care what the incentive is? Do you care what the field time is? Do you care whether your project is subcontracted to two other panels in order to fill the quotas? Do you care whether or not they use a router? Why or why not? Think strategically on this.
Ask questions of the panel provider. What incentive will you pay respondents? How long will you keep the field open? Will I be able to get the data from the mid-interview terminates and/or the unqualified respondents? Can you complete this entire study off your own panel or will you be subcontracting it? If I’m using profile information from the panel (e.g. I only want married respondents), what proportion of your panel has that information attached and how recent is the information? Make sure you’re comfortable with the answers.
Then set expectations. For instance, Grey Matter Research has clearly communicated to all of our panel vendors that all respondents must come from their own panel (no subcontracting), that they are not allowed to insert their own questions before our survey (no portal use), and what the field time is to be on each study. We review and test all survey programming before the field. We get the data from the unqualifieds and mid-interview terminates, looking for any bias in who was invited to take the study and/or who didn’t make it all the way through. We review the data afterward, looking for speeders and straightliners (even if the panel company promises to do that). If they deviate from our instructions, they don’t work with us any more.
When the survey is in the field, monitor the progress. Be involved. A research vendor can’t effectively put a survey into the field for 12 hours and skim a convenience sample off the panel if you’re monitoring the progress of the study and have agreed that the field time will be four days. Set expectations with the vendor for what you want out of the field management.
Pay attention to research on research. In addition to Grey Matter’s reports on panels, there are other resources out there, such as the Grand Mean Project from Mktg. Inc. Read. Listen. Talk to other researchers. Talk to the panel companies themselves (or make sure your vendor is doing these things). Read panelist comments at websites on which respondents discuss panels and panel members (such as www.surveypolice.com). Learn.
Do the research the right way, regardless of the methodology. A nonsensical question is a nonsensical question, whether it reaches the respondent through an online panel, a mail survey, their smartphone, or a landline phone call. I will never remember exactly how many times I have rented a car in the last 12 months, no matter what you want out of me as a respondent. And I will lose interest after 40 minutes, or after a few lengthy, complex grids.
Finally, do some poking around yourself. I had no idea that any supposed “research” panel would dare to include a bunch of sales messages to panel members, calling each one a “survey,” until our panelists found it through this test. A completely unethical practice, and a clear violation of section B2b of the CASRO Code of Standards (of which the company is a member)…and one you could only learn about if you were a panelist.
If you are on the client side and work with research vendors who acquire panel sample for your studies, are you confident that your vendor takes these steps (and others) to focus on quality? Have you thought strategically about why you would select one panel over another, or about what would disqualify a panel from your consideration?
If you contract with panels directly (either as a client-side research or as a vendor), do you take these steps (and others)? And if you work with a panel broker, are you even aware of where your sample is coming from?
Ultimately, it’s up to the research community to reward good panels with business and suffocate the bad panels by letting their business dry up. Price should not be the only determinant, and panel sample is not a commodity and should not be treated as such.
If you’d like a copy of More Dirty Little Secrets of Online Panel Research, e-mail me at email@example.com.