There’s one area of both qualitative and quantitative research that too often is a black hole. Researchers can hand their precious project over to someone and expect the results back, but feel they don’t need to worry about what goes on in between. Unfortunately, what you don’t know can kill you.
Well, maybe that’s a bit melodramatic – but what you don’t know certainly can kill your project.
I’m talking about The Field. Whether it’s recruiting qualitative respondents, getting panelists to complete your survey online, gathering completed phone interviews, or any other methodology, fieldwork to many researchers is frequently the least exciting part of a project. Vendors are often selected primarily by price, with the attitude that The Field is pretty much a commodity. And The Field is a practice that too often is just handed over to someone else, such as a panel provider or a focus group facility, to be completed while the researcher turns his or her attention to reporting or designing a methodology for the next project.
But what goes on in the field is a critical part of your success as a researcher. There’s very good fieldwork, and very bad, and it’s up to you to know the difference.
For instance, when you contract with a panel provider to have your questionnaire completed, do you know (or care) whether they subcontract part of the sample to another panel? Whether they insert some of their own questions right before yours, potentially biasing your sample? Whether they’re going to put it into the field for just one day, skimming a convenience sample off the top in order to move on to another project? Whether they use a router, or direct invitations specifically to your study?
When you contract with a phone center, do you know whether the work will be done domestically or off-shored to a center in India or Guatemala, where the interviewers will be probing on your open-ends with English as a second or third language? If the focus group facility tells you it’s not possible to fill your quotas, have you explored exactly why?
Keeping close tabs on what happens in the field is not optional to good research. It should be a collaborative process, rather than something you simply hand off. There are some pretty bad phone centers, focus group recruiters, and online panels out there. There are also things you can learn in the field that will make a difference in your project.
Here’s one example. Grey Matter Research conducted a mixed-modality survey about non-profit giving a couple of years ago. In reviewing the data, we found a serious inconsistency: people who said they had not given to any non-profits in the past 12 months often reported having given toward Haiti earthquake relief just a few minutes later in the survey. In trying to figure out what was happening, we spoke with the phone interviewers. They told us respondents were explaining that with the massive Haiti earthquake, they felt they were giving toward a country, so they didn’t consider that to be helping a non-profit. Without that feedback from the field, we would not have been able to figure out a serious anomaly in the data.
Here’s another. Years ago, we had to recruit focus groups in a market we hadn’t used before. We selected a well-known, highly rated facility for the recruiting from our client’s customer list. Unfortunately, they struggled to get it recruited. Rather than simply open up the quotas and qualifications (as the facility repeatedly demanded), we dug into what was going on. Turns out their recruiters were mostly single mothers who dialed while their kids were in school. Since this was a consumer recruit, that meant most people were getting called during weekdays, while they were away from home. We learned a hard lesson about the need to investigate the field practices of a new vendor before handing them a project – even when that vendor has a good reputation.
How many callbacks is the phone center making on each number? Are they using true RDD sample, or listed sample because it’s much more productive (and therefore easier and more profitable)? Did they randomize the list you provided before they started calling, or are they dialing alphabetically because that’s the order of the list you gave them?
How many other questionnaires has your online panel provider offered their panelists today – is yours the fifth one they will complete in the last two hours? How complete and recent is the typical panelist profile – are you relying on outdated or partial information? Is someone cleaning the data you get, looking for straightliners or speeders who completed a 16-minute questionnaire in six minutes?
Is the qualitative recruiter really using your screener as you wrote it, or is that just a suggestion for the recruiters? When respondents ask them what the call is about, do they say, “Oh, we’re recruiting a focus group among people who use hand lotion regularly,” thereby telling respondents just how to qualify for the groups? Are they just sending out an e-mail blast to their database, telling respondents they’re recruiting a group among women with post-nasal drip, or are they actually contacting and screening people in an unbiased manner?
In short, how much attention are you paying to The Field – this rather unglamorous but critically important element of your research? If the answer is not “a lot,” your projects are at risk.