The Five Star Fallacy
Recently, I took my car to the dealership for an oil change. The service was exactly sufficient. The dealership was busy, so the service representatives were rushing around doing their best to get each customer serviced and out as quickly as possible. Despite the frustration that can come with this rushed approach since any questions I had about future services or options for my car were met with brief, pressured responses, the service department still accomplished what I needed them to- my car received fresh oil.
As I was leaving, something disturbing happened. The service representative explained that I would be receiving an emailed survey that would ask me to rate my experience with them. Then, he said it. Something that made my skin crawl: “If you are satisfied with your service today, please give me five stars. Anything less than five stars is basically a fail for me.” As he exited the lobby, he turned back, smiled, and said, “Remember, five stars!” I was stunned and a bit angry. The representative had just committed an egregious violation in the world of data collection. He attempted to manipulate data (and manipulate me!) to get favorable results- also known as the experimenter effect.
The experimenter effect refers to “any influence a researcher may have on the results of his or her research, derived from either interaction with participants or unintentional errors of observation, measurement, analysis, or interpretation.” Although the service representative is not a researcher, the greater corporation is attempting to measure the satisfaction of their customers to inform their practices (i.e., program improvement research). Except, the validity of their findings is severely threatened due to this employee’s attempts to persuade me to give him a perfect score on my customer satisfaction survey. To be clear, the employee is not at fault here. He was simply adhering to his training. There is a bigger issue at hand. Large corporations, like the one here, train their employees to manipulate survey data to get better ratings.
Two possible explanations for this behavior exist: 1) In its quest to have maximally satisfied customers, these companies put so much pressure on employees to get high ratings that somewhere in the management chain, employees are trained to beg for perfect scores; or 2) the companies do not actually care about the satisfaction of their customers and only care about the marketability of having high ratings. My money is on the latter.
Group norms are powerful influencers of individual behavior. This is why many sitcoms in the 90s had laugh tracks- viewers found the content funnier if they heard others laughing. The same applies to product reviews. Consumers are more likely to do business with an organization with higher ratings. And, companies know this, so they do everything in their power to get higher ratings. More stars equal more profits.
On the surface, this relationship seems beneficial for everyone. Consumers want and will pay for the best available products or services, so companies are motivated in the direction of progress. That’s the beauty of capitalism. However, when companies take shortcuts to artificially elevate the desirability of their services, that’s no longer good marketing- that’s manipulation. Take my car dealership experience, for example; I received an email the next day with the following content in the body of the email: “Was your service experience truly exceptional?” with only two answer choices- “yes” or “no.” By clicking “yes,” the embedded link connected me to the review page of the company’s website with five stars already populated! This is further manipulation by reducing the customer’s effort to a single click. Out of curiosity, I went back to the email and clicked “no.” The embedded link connected me with a multi-question survey- something that requires more effort and time, further steering consumers towards a perfect rating.
This is the experimenter effect in its worst form. It’s purposeful manipulation of the consumer to manipulate future consumers. This company, and many like it, are using multiple scientifically proven principles of influence and persuasion to artificially raise their customer satisfaction ratings to earn more business. As a social scientist, I find this infuriating. As an organizational consultant, I also view it as bad practice. It’s effective, sure, but it’s shortsighted.
Outcome measurement is the key to growth. Whether it’s stepping on the scale each week to see if your diet is working or checking to see if your new advertisement has yielded more sales, solid outcome measurement is your compass when it comes to knowing how to adjust your behaviors to maximize growth. When companies attempt to manipulate customer satisfaction surveys, they are no longer measuring outcomes. They are willingly putting on blinders to what their customers really think- a dangerous practice that is almost guaranteed to halt any customer-informed adaptation of products or services.
As an organizational leader, it is not only acceptable but encouraged to chase high ratings. Your ratings are your azimuth check. However, attempting to manipulate data to give the appearance of better service is not only misleading but it’s losing sight of what’s important. Don’t be afraid to seek candid feedback so you know where and how to adjust. As the late Stephen Covey said, “It takes humility to seek feedback. It takes wisdom to understand it, analyze it, and appropriately act on it.” Data manipulation is neither humble nor wise.