I just had a great research conversation with my car dealer.
Shortly after I took my vehicle in for service, I got a customer satisfaction questionnaire by e-mail. They wanted my ratings in a number of areas (cleanliness of the dealership, wait time, etc.).
I was actually puzzled on how to fill this thing out. The scale was highly unbalanced. It was 1 – 10 scale, but it wasn’t anchored with “poor” and “excellent.” Indeed, the midpoint was actually around a rating of 3…higher on the scale were words such as “excellent” (5), “outstanding” (7), and “exceptional” (10). (I’m going off of memory here, so if you work with Mercedes’ research department, please don’t post a scathing response that I didn’t get the scale exactly right.)
Was I completely satisfied with my interaction with the service department? Absolutely. On a typical 1 – 10 scale, they would have gotten all 9 and 10 ratings. But on this scale? Did they really do anything that was “exceptional”? And “exceptional” compared to what? Compared to other Mercedes dealers? Compared to a Kia or Ford dealer? Compared to my previous experiences with them? Compared to my expectations?
Lacking any context, I decided to use the words of the scale rather than the numbers of the scale to complete the survey. They washed my car, which they do every time, and which most other dealers do every time someone comes in for service. Is that exceptional? No – it’s standard service I can get anywhere. It’s not like they conditioned the seats and scrubbed the engine. So I put down that it was good, but nothing more. Unfortunately, that’s a rating of about 4, which apparently doesn’t look so good for the dealership.
My service rep asked me what they did wrong. I explained that they did nothing at all wrong, and in fact I was quite satisfied with the visit. But they also didn’t really do anything exceptional – “exceptional” meaning (to me) something I wouldn’t normally expect from them or from their competitors; something I don’t see on every visit. How is it exceptional to receive a clean car, just like I would at the Subaru dealer down the street?
Obviously, they intend the scale to be interpreted one way, and I interpreted it in a very different way. I had no context for how to answer the questions. Their perspective is that they want every experience to be exceptional. My perspective is that this is literally impossible – if every experience is exceptional, don’t those eventually become average or typical experiences? Can something be exceptional if they do it every time?
I’ve had this discussion with research clients about expectation-based scales (meets expectations, exceeds expectations, etc.). If I normally wait 20 minutes for service at Denny’s, and on one visit I only wait 15 minutes, their performance that day exceeded my expectations. If at Cracker Barrel I normally receive service immediately, but I’m forced to wait 5 minutes, their performance fails to meet my expectations. Yet it’s still far superior to the 15 minute wait time. On an expectation-based scale, the 5 minute wait would fail to meet expectations, while the 15 minute wait would exceed expectations. Is that really what either restaurant chain would want to measure? If I expect truly horrible food, and I’m surprised that actually I manage to keep it down, that would exceed my expectations – but is that really a good rating for the restaurant?
Using rating scales, we absolutely have to incorporate context. Without it, different respondents will interpret the scale differently (as I did with the car dealer), which leads to research which doesn’t really tell us anything. We have to understand all possible meanings before putting it in front of respondents. Failing that, we end up as my car dealer did – receiving what Mercedes will interpret as very poor ratings when in fact I’m completely satisfied. That type of research doesn’t do anyone any good.