Critical Condition
“If you are counting full-time critic jobs at newspapers, you may as well count tombstones.” That was the response of Johanna Keller, director of the Goldring Arts Journalism Program at Syracuse University's S.I. Newhouse School of Public Communications, to a Nieman Reports query about the number of professional critics employed at dailies around the country. The figures on newspaper critics (News flash: they're not good) are one indication of the state of criticism today, but they are not the only one.
Greetings from 50 testing labs humming with spinning washers and dryers; illuminated with newfangled light bulbs and supersized TVs; ambrosial with the aroma of hundreds of just-baked cookies from dozens of ranges and wall ovens; chilly from the steady blast of room air conditioners; striped and splattered from assessments of paints and stains; and buzzing with trained tasters sampling chocolates or beef jerky or sparkling wine. In other words, the Yonkers, New York headquarters of Consumer Reports, which puts to the test more than 3,000 consumer products each year.
I know, I know. That sounds nothing like your newspaper, magazine, Web operation or the spare bedroom where you write your reviews. But an important lesson Consumer Reports has learned from user reviews—namely, how to use them to become stronger—is applicable to a range of professional reviewers, I believe.
Photo courtesy of istock.
You might think that an organization like ours would react in one of two predictable ways to the proliferation of user reviews you can now find for pretty much anything you want to buy:
Dismiss them as trivial and unscientific. After all, we're the organization that spends more than $7 million each year buying not only products—from Audi sports sedans to ZVOX home theater systems—but also making or buying the testing equipment and sourcing ancillary supplies. The latter includes cotton swatches identically stained with chocolate ice cream, grass, sebum, and other blots to assess laundry detergents; Maine Coon cat fur to test the pet-hair pickup claims of vacuums; cherry-pie filling, eggs, and tapioca we use to create the "monster mash" we paint on the innards of ovens to rate their self-cleaning claims, and much more. Compare that level of testing rigor to Concepcionz and her five-star impressions on Amazon.com of her American Standard elongated two-piece toilet: "The product arrived as described, pretty good price and it arrived very safe. … Everything is as described and I love the product."
Fear user reviews as our nemesis, a potential assassin of the professional tester. Let's face it: Free reviews are more appealing than those you have to pay for. (For access to our ratings, you have to buy a magazine subscription or a subscription to our website.) We know that younger consumers, especially, think advice from friends or even strangers is often all they need to make buying decisions. Years from now, as those buyers grow up, will most consumers consider user reviews to be good enough for everything they buy?
Actually, I've come to the conclusion that there's room in the universe, indeed, an important place, for both personal and professional reviews. I don't pretend to understand the fine points of movie, restaurant or theater reviewing. What I know about product reviews, however, suggests that readers will pay for information they consider valuable and that you do better than anyone else. User reviews—what real consumers focus on, gripe or rave about—can help inform that coverage.
Product testing has been the backbone of Consumer Reports since its founding 77 years ago, in 1936. We're a nonprofit group, we buy every product we rate, we take no advertising from manufacturers—our founders wisely believed that our product ratings could be seen as suspect if they were sandwiched between various manufacturers ads—and so subscription sales largely fuel the revenue of our organization. (Grants and donations account for a small percentage of overall revenue.) You could say we were among the first publishers to adequately value our content.
Our immense surveys of readers, the basis of our exclusive brand reliability information, and our ratings of service providers such as hotels and cell phone carriers, are second in size only to the U.S. Census, we believe. So we've actually embraced user reviews for many decades.
In some circles, the rap on Consumer Reports is that we're dream killers. That cherry of a sports car—the one you hope to buy when you finally "arrive"? Consumer Reports says it's unreliable! That pro-style range you have your heart set on, the one the Joneses already bought? Consumer Reports says there are far better and cheaper choices!
Our readers, however, see us differently. We work for a group of consumers—4 million print subscribers and 3.3 million Web subscribers—who some of us describe as "value enthusiasts." Many of them could afford pretty much anything, but they delight in getting a great deal for their money, not overpaying, and not falling for hype or gotchas. They also love to research what they buy.
Photo courtesy of istock.
Here's what we try our best to deliver that individual user reviews can't:
Depth of testing. When we rate a dishwasher, for example, we're comparing it to hundreds of other models we've tested the same way. For an individual user, his reference is typically only the machine he bought versus the one he's replacing. "Dutchie" from Tennessee, another Amazon reviewer, may be heartfelt in his assessment of his Amana ADB1000AWW dishwasher: "I must say truthfully that the appliance is of very good quality and performance."
Oh, Dutchie. Had you read us, you would have seen that this Amana model, while very inexpensive, is incredibly noisy, only fair for cleaning, and rock-bottom overall in our ratings of 207 machines. For a few hundred dollars more, you could have bought a Consumer Reports Best Buy from a more reliable brand. The depth and breadth of our testing, a big differentiator from other product reviewers, is the main reason millions subscribe to Consumer Reports. It's our "gold content." A fair question for reviewers of all stripes is, Why should people read you? What does your audience value that you can do better than anyone else?
Impartiality. I'm sure the motivations of many user reviewers are aboveboard. And the idea of a national network of consumers helping consumers warms the heart of a consumer advocate like me. But you'll never really know the identity, goals or qualifications of individuals penning a review—or whether they actually own the product they're reviewing.
Is there even such a thing as an impartial user review? Your new refrigerator either met or exceeded your expectations (which can be quite low, if the appliance you're replacing is decades old), and so you LOVE it. Or it didn't, in which case you HATE it. By contrast, our main concern is that our tests are fair and repeatable; we're not invested in how any particular model performs.
Takeaway advice. Professional reviews like ours give you guidance. "The new Cuisinart costs more than many toasters, but its solid performance and sleek design might be worth the investment, especially if you plan to use it every day," we noted about the $80 CPT-420 model.
A high price gets you nowhere, however, when it comes to the $107,850 Fiskar Karma luxury sedan, which was "plagued with flaws," according to our review. "Compared with other luxury sedans, its tight confines and limited visibility can make the cabin feel claustrophobic; a lack of conventional buttons and the worst touch-screen system we've seen make the dash controls an ergonomic disaster; and acceleration lacks the oomph you'd expect from a sports car."
If you rely only on user reviews to make your purchases, you're on your own to figure out how to synthesize dozens of discordant comments. Does one super-negative review annul numerous glowing squibs? Do 10 positive reviews mean a product is likely worth buying?
Granted, there's much less to lose when you're seeking counsel from users about everyday products that cost a few bucks, rather than big-ticket appliances, electronics or cars. And for sure, they can be hilarious to read. (The Consumer Reports of the very good Oh Boy Oberto Original beef jerky noted that it's "a tad spicy, with well-blended smoke, brown sugar, garlic, and fruit flavors." Antimattercrusader's Amazon review of the brand's thin style jerky: "Omg … this stuff is an orgasm in a bag.") But even for reviews of small-potatoes products, it can be tedious to wade through scores of comments.
That said, user reviews can be a real boon to professional testers like Consumer Reports, and we encourage readers to share their experiences with us. (You have to be a subscriber to post a review on ConsumerReports.org, and only subscribers can see them. That doesn't eliminate the chances a reviewer isn't who she says she is, but the bar is somewhat higher than with anyone-can-post reviews.)
A key way we've been helped by user reviews is that they flag problems with products our testing didn't uncover. We often can't test models long enough in our labs for durability concerns to emerge.
Moreover, consumers collectively have many more samples of a product than we test, and the size of the group can help unearth problems. That was the case several years ago with the Braun PowerMax MX2050, a blender we rated highly. Then we heard from a dozen readers that the plastic gear-tooth assembly was prone to breaking. So we developed a tougher test and encountered the same problem with the blender model that our readers experienced. Bottom line: The manufacturer gave consumers a free replacement blade and gear assembly and vowed to fix the problem. We kept testing, and a new Braun PowerMax MX2050 became a Consumer Reports Best Buy. So consumers' voices improved the marketplace.
We regularly review our subscribers' reviews. I encourage other reviewers to do the same, not to pander to their readers, but to understand what matters to them and to ensure that you address their questions and concerns.
For example, when there's a significant gap between our overall score for a product and the average score our subscribers gave it in user reviews, we investigate whether our readers are on to a potential problem.
Readers' comments also help us plan tests, so that we're addressing real-world consumer insights and concerns, and making our ratings all the more relevant. In the future, we hope to be able to synthesize the wider world of online user reviews into our product research.
All of which is to say that, yes, I'm banking quite a lot on the ongoing and much needed role of the professional reviewer. Ironically, for the smartest professionals, that role will be cemented in part by user reviews, which they'll use to help define and refine their unique value to readers to better offer reviews that matter.
Kimberly D. Kleman is the editor in chief of Consumer Reports magazine and an adjunct associate professor at the Columbia University Graduate School of Journalism. She lives in Pleasantville, New York, in a home replete with Consumer Reports Best Buy appliances.
I know, I know. That sounds nothing like your newspaper, magazine, Web operation or the spare bedroom where you write your reviews. But an important lesson Consumer Reports has learned from user reviews—namely, how to use them to become stronger—is applicable to a range of professional reviewers, I believe.
Photo courtesy of istock.
You might think that an organization like ours would react in one of two predictable ways to the proliferation of user reviews you can now find for pretty much anything you want to buy:
Dismiss them as trivial and unscientific. After all, we're the organization that spends more than $7 million each year buying not only products—from Audi sports sedans to ZVOX home theater systems—but also making or buying the testing equipment and sourcing ancillary supplies. The latter includes cotton swatches identically stained with chocolate ice cream, grass, sebum, and other blots to assess laundry detergents; Maine Coon cat fur to test the pet-hair pickup claims of vacuums; cherry-pie filling, eggs, and tapioca we use to create the "monster mash" we paint on the innards of ovens to rate their self-cleaning claims, and much more. Compare that level of testing rigor to Concepcionz and her five-star impressions on Amazon.com of her American Standard elongated two-piece toilet: "The product arrived as described, pretty good price and it arrived very safe. … Everything is as described and I love the product."
Fear user reviews as our nemesis, a potential assassin of the professional tester. Let's face it: Free reviews are more appealing than those you have to pay for. (For access to our ratings, you have to buy a magazine subscription or a subscription to our website.) We know that younger consumers, especially, think advice from friends or even strangers is often all they need to make buying decisions. Years from now, as those buyers grow up, will most consumers consider user reviews to be good enough for everything they buy?
Actually, I've come to the conclusion that there's room in the universe, indeed, an important place, for both personal and professional reviews. I don't pretend to understand the fine points of movie, restaurant or theater reviewing. What I know about product reviews, however, suggests that readers will pay for information they consider valuable and that you do better than anyone else. User reviews—what real consumers focus on, gripe or rave about—can help inform that coverage.
Product testing has been the backbone of Consumer Reports since its founding 77 years ago, in 1936. We're a nonprofit group, we buy every product we rate, we take no advertising from manufacturers—our founders wisely believed that our product ratings could be seen as suspect if they were sandwiched between various manufacturers ads—and so subscription sales largely fuel the revenue of our organization. (Grants and donations account for a small percentage of overall revenue.) You could say we were among the first publishers to adequately value our content.
Our immense surveys of readers, the basis of our exclusive brand reliability information, and our ratings of service providers such as hotels and cell phone carriers, are second in size only to the U.S. Census, we believe. So we've actually embraced user reviews for many decades.
In some circles, the rap on Consumer Reports is that we're dream killers. That cherry of a sports car—the one you hope to buy when you finally "arrive"? Consumer Reports says it's unreliable! That pro-style range you have your heart set on, the one the Joneses already bought? Consumer Reports says there are far better and cheaper choices!
Our readers, however, see us differently. We work for a group of consumers—4 million print subscribers and 3.3 million Web subscribers—who some of us describe as "value enthusiasts." Many of them could afford pretty much anything, but they delight in getting a great deal for their money, not overpaying, and not falling for hype or gotchas. They also love to research what they buy.
Photo courtesy of istock.
Here's what we try our best to deliver that individual user reviews can't:
Depth of testing. When we rate a dishwasher, for example, we're comparing it to hundreds of other models we've tested the same way. For an individual user, his reference is typically only the machine he bought versus the one he's replacing. "Dutchie" from Tennessee, another Amazon reviewer, may be heartfelt in his assessment of his Amana ADB1000AWW dishwasher: "I must say truthfully that the appliance is of very good quality and performance."
Oh, Dutchie. Had you read us, you would have seen that this Amana model, while very inexpensive, is incredibly noisy, only fair for cleaning, and rock-bottom overall in our ratings of 207 machines. For a few hundred dollars more, you could have bought a Consumer Reports Best Buy from a more reliable brand. The depth and breadth of our testing, a big differentiator from other product reviewers, is the main reason millions subscribe to Consumer Reports. It's our "gold content." A fair question for reviewers of all stripes is, Why should people read you? What does your audience value that you can do better than anyone else?
Impartiality. I'm sure the motivations of many user reviewers are aboveboard. And the idea of a national network of consumers helping consumers warms the heart of a consumer advocate like me. But you'll never really know the identity, goals or qualifications of individuals penning a review—or whether they actually own the product they're reviewing.
Is there even such a thing as an impartial user review? Your new refrigerator either met or exceeded your expectations (which can be quite low, if the appliance you're replacing is decades old), and so you LOVE it. Or it didn't, in which case you HATE it. By contrast, our main concern is that our tests are fair and repeatable; we're not invested in how any particular model performs.
Takeaway advice. Professional reviews like ours give you guidance. "The new Cuisinart costs more than many toasters, but its solid performance and sleek design might be worth the investment, especially if you plan to use it every day," we noted about the $80 CPT-420 model.
A high price gets you nowhere, however, when it comes to the $107,850 Fiskar Karma luxury sedan, which was "plagued with flaws," according to our review. "Compared with other luxury sedans, its tight confines and limited visibility can make the cabin feel claustrophobic; a lack of conventional buttons and the worst touch-screen system we've seen make the dash controls an ergonomic disaster; and acceleration lacks the oomph you'd expect from a sports car."
If you rely only on user reviews to make your purchases, you're on your own to figure out how to synthesize dozens of discordant comments. Does one super-negative review annul numerous glowing squibs? Do 10 positive reviews mean a product is likely worth buying?
Granted, there's much less to lose when you're seeking counsel from users about everyday products that cost a few bucks, rather than big-ticket appliances, electronics or cars. And for sure, they can be hilarious to read. (The Consumer Reports of the very good Oh Boy Oberto Original beef jerky noted that it's "a tad spicy, with well-blended smoke, brown sugar, garlic, and fruit flavors." Antimattercrusader's Amazon review of the brand's thin style jerky: "Omg … this stuff is an orgasm in a bag.") But even for reviews of small-potatoes products, it can be tedious to wade through scores of comments.
That said, user reviews can be a real boon to professional testers like Consumer Reports, and we encourage readers to share their experiences with us. (You have to be a subscriber to post a review on ConsumerReports.org, and only subscribers can see them. That doesn't eliminate the chances a reviewer isn't who she says she is, but the bar is somewhat higher than with anyone-can-post reviews.)
A key way we've been helped by user reviews is that they flag problems with products our testing didn't uncover. We often can't test models long enough in our labs for durability concerns to emerge.
Moreover, consumers collectively have many more samples of a product than we test, and the size of the group can help unearth problems. That was the case several years ago with the Braun PowerMax MX2050, a blender we rated highly. Then we heard from a dozen readers that the plastic gear-tooth assembly was prone to breaking. So we developed a tougher test and encountered the same problem with the blender model that our readers experienced. Bottom line: The manufacturer gave consumers a free replacement blade and gear assembly and vowed to fix the problem. We kept testing, and a new Braun PowerMax MX2050 became a Consumer Reports Best Buy. So consumers' voices improved the marketplace.
We regularly review our subscribers' reviews. I encourage other reviewers to do the same, not to pander to their readers, but to understand what matters to them and to ensure that you address their questions and concerns.
For example, when there's a significant gap between our overall score for a product and the average score our subscribers gave it in user reviews, we investigate whether our readers are on to a potential problem.
Readers' comments also help us plan tests, so that we're addressing real-world consumer insights and concerns, and making our ratings all the more relevant. In the future, we hope to be able to synthesize the wider world of online user reviews into our product research.
All of which is to say that, yes, I'm banking quite a lot on the ongoing and much needed role of the professional reviewer. Ironically, for the smartest professionals, that role will be cemented in part by user reviews, which they'll use to help define and refine their unique value to readers to better offer reviews that matter.
Kimberly D. Kleman is the editor in chief of Consumer Reports magazine and an adjunct associate professor at the Columbia University Graduate School of Journalism. She lives in Pleasantville, New York, in a home replete with Consumer Reports Best Buy appliances.
