Why do so many organizations ask broadly general questions and then try to turn them into specifics?
This happens constantly in political research. A question might be asked such as, “Do you favor or oppose increasing taxes in order to help reduce the federal deficit?” Although the results might make for some good headlines, this question is so broad as to be meaningless. Increasing which taxes? How much? For which taxpayers? It’s very easy for respondents to toss off an answer such as, “Yes, we need to take difficult measures in order to cut the deficit,” or “No, I don’t support raising taxes.” But give respondents even a shred of detail, and the answers could be entirely different.
Imagine how different the favor/oppose percentages would be to the following options:
- Instituting a national sales tax of 1/3 of 1%
- Instituting a national sales tax of 20%
- Raising income taxes on the wealthiest 1% of Americans
- Raising income taxes on all Americans
- Raising corporate income tax rates
- Raising taxes on tobacco
Yet all of these approaches (and many more) qualify as “increasing taxes.”
I just saw a report from a study claiming that “43 percent of Americans say they would be more likely to buy from a company that ‘manages its business according to Christian principles,’” and only 3 percent were less likely to do so.
What in the world does this even mean? If “Christian principles” means no lying and stealing, it would shock me that only 43% of Americans would prefer to direct their business to that company. If “Christian principles” means hiring only Christians, putting a Bible verse on all products that are sold, and requiring employees to talk to customers about their religious beliefs, there’s no way only 3% of Americans would avoid that type of company. That question is so general as to be meaningless, yet it’s being promoted by the research company and picked up by some media as “news.”
Worse yet, there’s a conclusion to go with the data. “[T]he research shows that the consumer audience is divided between those who favor Christian companies and those who are simply indifferent,” states the report, noting that there’s almost no downside to a company actively promoting Christianity. Let’s find out how well Starbucks or Chevron would do “embracing and promoting” a specific religious stance, and see if there’s no potential downside.
Ask Americans in a survey if they would buy a hybrid vehicle over a gasoline vehicle, and there’s not enough information there for people to answer. Will the hybrid increase gas mileage by 5%, or by 50%? How does the purchase price compare? How about dependability? Styling and comfort? Power? It could be legitimate to ask how likely they are to consider a hybrid based on what’s currently on the market, because respondents have context with which to frame their answers.
Unfortunately, those factors mean more questions, or even detailed techniques such as conjoint. It’s so much quicker and easier just to ask people whether they would want a hybrid version of that current sedan and let them provide a meaningless answer to a meaningless question.
As researchers, aren’t we better than that?