clock menu more-arrow no yes mobile

Filed under:

Inside Consumer Reports

How the magazine obsessively tests snowblowers, cookies, cars, and toilets

Inside Consumer Reports

How the magazine obsessively tests snowblowers, cookies, cars, and toilets

by Joseph Stromberg on January 21, 2015

At the Consumer Reports testing labs in Yonkers, New York, Emilio Gonzalez cooks chili, eggs, soup, oatmeal, and 13 other foods a few times per month. No one eats any of it. Instead, Gonzalez smears carefully measured quantities of the foods on a set of 113 plates, bowls, cutlery, glasses, and other dishware.

An hour later, the dishes are scraped, loaded into a dishwasher in a specific pattern, and allowed to sit overnight. The next morning, they're washed with a set amount of specific detergent, using 120°F water and exactly 120 volts of electricity. Afterward, Gonzalez and other staff scrutinize the dishes to judge how much food is left on them, and a computer system images the bowls to analyze them pixel by pixel.

"All the dishwashers go through this test multiple times," Gonzalez said. "Sometimes it's two times, if they're consistent, but if we see discrepancies, we'll continue testing over and over." They do this for virtually every dishwasher that hits the US market.

At this sprawling testing center — and at an auto track a few hours away in Connecticut — a few hundred Consumer Reports staff test thousands of products annually with this same level of meticulousness. They test treadmills, smartphones, bike helmets, and toasters, using multiple, randomized trials, statistically sound analyses, and other methods from the world of peer-reviewed scientific research. They leave hundreds of light bulbs on indefinitely to see how long they last. They have teams of trained sensory panelists rate yogurt and potato chips as objectively as possible. They flush bowls of plastic marbles down toilets to see how likely they are to clog.

To ensure they're getting the same thing you'd buy at a store, they buy all of these products anonymously, at full retail price. To maintain independence, they don't run any ads in their print magazine or on their website and don't even allow manufacturers to trumpet positive test results in their own ads elsewhere. For many of these products, Consumer Reports is literally the only group testing this thoroughly — and in some cases, they've noticed potentially dangerous defects and alerted manufacturers or regulators to issue recalls.

A comparison of refrigerator reviews by Good Housekeeping and Consumer Reports. (Tyson Whiting)

None of this is cheap. There are hundreds of websites that get free models of every new gadget, test it for a few days, and put up a review. In 2014, meanwhile, Consumer Reports' operating expenses were more than $258 million, mostly paid for by millions of subscriptions to a product that now competes with a sea of free, ad-supported content that has flooded the web. Within the last six months, the company has brought in a new CEO and a new editor-in-chief and launched a magazine redesign, all part of an attempt to bring new, younger readers into the fold — a critical goal after losing money in 2011 and 2012.

Consumer Reports' testing processes, scientific rigor, and commitment to independence are commendable. But it's difficult to take them all in without asking an important question: does Consumer Reports have a future?

Peter Sawchuk may know more about snow blowers than anyone else on Earth. After decades working for a manufacturer, he now spends a few months per year pushing virtually every new model that hits the market back and forth on a driveway behind the Consumer Reports labs.

Since testing goes on in the fall — in order to have reviews and ratings ready in time for the winter buying season — Sawchuk doesn't usually have snow available for testing. But after years of experimentation, he created an effective substitute: clumped, wet sawdust, with just enough water mixed in to mimic the properties of snow. To test each blower, he fills a foot-high trough with sawdust, then pushes the machine through it over and over, completing enough trials to rate each blower's ease of use, its reliability, and most importantly, the distance that it can fling wet snow.

Peter Sawchuck tests a snowblower with wet sawdust. (Corey Olsen)

Apart from snow, there's often another thing missing in the autumn: many of each year's new snow blower models. Sawchuck and other staff travel widely to obtain as many as possible, but in some cases, they reluctantly buy a machine directly from the manufacturer. "When we do that, we're kind of paranoid that we've gotten a gold-plated sample, so we go out and buy a retail model later on and check to see if it's the same," he said. In some cases, they have gotten souped-up machines with slightly more power than standard models.

This is the rationale for one of Consumer Reports' core principles: buying retail whenever possible. (The company takes editorial independence so seriously that, in 2001, it actually turned down a request for a bulk order of 170 magazine subscriptions for employees of a car dealership chain, saying it might be "viewed as a compromise to our objectivity.")

In order to get a particular food product, this sometimes means calling upon a secret shopper halfway across the country. For cars, it means driving up and down the East Coast to find a dealership with the right model, and having a pair of unlisted numbers for dealers to return calls to — so they don't realize they're selling to Consumer Reports. After testing, most of the products are auctioned to employees or sold at 80 percent of retail cost, to account for wear and tear.

This extreme scientific rigor, mixed with amusing attempts to roughly simulate consumer behavior, defines the testing of every kind of product at the labs. Bloomberg Businessweek has called it "scientific torture." To test vacuums, for instance, staff spread cat hair on a strip of carpet, vacuum it, then weigh how much hair ends up in each vacuum's brushes and storage bag. To test frying pans, they use a machine that scrubs them for hours with steel wool until their coating wears off. In Consumer Reports' exercise testing lab, there are a pair of huge steel rollers with rubber pegs on them, designed to rotate atop a running treadmill, simulating six months of use within a week.

Testing some items, especially tech products, requires even more elaborate laboratories. Consumer Reports is home to a pair of anechoic chambers: specially designed rooms, one that blocks out all incoming sound, the other that blocks out all electromagnetic waves.

The soundproof chamber, used to test speakers and microphones, is built with ultra-thick walls of ridged foam, and rests on an independent foundation that sits on a bed of springs, so no vibrations are transferred from the rest of the building. It is entirely silent, in a jarring way: it makes you realize that places that you normally consider quiet have constant background sound. Elias Arias, an audio product tester, said that some people enjoy being inside with the door shut, but it makes others uncomfortable. Inside the chamber, some people can actually hear their own heartbeats.

A refrigerator's ability to maintain a constant temperature is tested with sensors in every drawer. (Corey Olsen)

The room doesn't just block out all noise from the outside, but absorbs all sound waves traveling inside. As a result, people's voices sound startlingly clear and smooth, as though they've been digitally mastered. "What's great about this room is that you're getting my pure voice," Arias said. "It's an audio clean room, which lets us analyze all kinds of audio without contamination."

The other anechoic chamber, with similarly thick foam walls, is designed to block out all radio frequency radiation, like that of cell signals and Wi-Fi networks. "Once the door is closed, there's absolutely no cell signal in here," said Maria Rerecich, who leads the electronics testing team. "That's so we can run tests under controlled conditions — we can make our own signal."

She and other staff test cell phones in here with a device called the "head and torso simulator": a legless mannequin with a blocky head and a spongy, weirdly lifelike ear. With a phone clamped in place next to it, the machine's mouth emits a number of phrases in a clipped British accent ("A large size in stocking is hard to sell"), and a recorder on the other end measures how clearly they came through. In 2010, when the new iPhone 4 was reported to have reception problems when held in a certain way, this room was enormously valuable for testing the phone itself, independent from the network — and confirming that, when held by the lower left corner, a user's hand could indeed interfere with its antenna.

More recently, the scandal that accompanied the debut of the iPhone 6 allowed Consumer Reports staff to conduct some lower-tech tests. Soon after the phone came out, people began tweeting photos of bent phones: the result of the large, thin devices getting bent in half in tight pants pockets when they sat down. In response, Rerecich and others submitted some iPhones and other smartphones to an "instron test compression machine": a device that exerts a specific amount of force on any object. When it pushed on the center of the iPhone with 70 pounds of force, it did bend it, but it also bent the Androids they tested, along with older iPhones. Rerecich's conclusion: the iPhone 6 wasn't unusually fragile. "Anything's going to bend if you try hard enough," she said, gesturing to a table covered with all sorts of bent phones, the result of a full day's testing.

This meticulous approach even defines Consumer Reports' testing of the most subjective product category: food. Amy Keating, who runs the food lab, conducts tests with a team of sensory panelists, who've been trained to evaluate the qualities of, say, a packaged cookie, and compare it to a competitor's cookie as objectively as possible. "The only way to test food is to use a human palate," Keating said. "But test results aren't our preferences — they're the quality of the food itself."

Granola is placed in identical bowls for testing. (Corey Olsen)

The process works like this: Keating and other staff remove the cookies' packaging and serve them on small plates with codes on them. This way, panelists don't know which brand the cookies are, but Keating can later match their grades with their labels. The panelists assemble in a darkened room with its own ventilation system to ensure that incoming scents don't interfere with their tasting. They sit at a counter and are served the food through a small window (in some cases, under a red light, so the color of the food doesn't sway their impressions). Each panelist will taste a small portion of the food, then spit it out and cleanse his or her palate with water and perhaps a bit of an unsalted cracker.

Afterward, panelists will record their assessment in discrete grades: an 8 out of 10 on the sweetness scale means something very specific, determined months or years earlier, during training. If a particular panelist begins consistently coming to findings that differ from the rest of the group, they go through further training, and might even be eventually let go. In this lab, tasting cookies is serious business.

An independent statistical group analyzes the design and results of all tests, looking for data irregularities and ensuring that products are tested in a randomized order. In testing frozen pizzas, for example, Keating conducts multiple trials, with each brand of pizza getting cooked in each different oven. "Our competitors might just sit around with the labeled packages and taste it once," she said. "We take pains to do this over and over and over."

Consumer Reports' first cover story, in May 1936, was a report on the difference between Grade A and Grade B milk. The magazine's publishers — a group of consumer advocates who were fired from an existing group, called Consumers' Research, for trying to unionize — found that the two types of milk were identical, in terms of both taste and nutrition, and that no one should pay the extra three cents per quart for Grade A.

In the decades since, Consumer Reports has continued to serve as a distinct voice of modest, rational frugality. Most magazines have celebrities on their covers and hawk expensive new products on ads within. Consumer Reports explains why premium gas is a waste of money for most cars and puts bacteria-laced chicken breasts on its covers to warn customers. Instead of shilling for corporations, the magazine tries to convince you not to buy things you don't actually need. Personified, it's your penny-pinching uncle, the one who saves up hundreds of cans and bottles for the occasional trip to the recycling plant to get the deposits back.

Smartphones were subjected to precise levels of mechanical stress to see if the iPhone 6 bent more easily. (Corey Olsen)

But the magazine also weighs in on a number of serious policy and health issues. After its Great Depression years as a shoestring operation (it mostly tested cheap products, like cereal, and borrowed cars because it didn't have enough money to buy them), it grew significantly during the 1950s, with its rise coinciding with a broader surge of interest in consumer goods. During this era, it conducted some of the first reporting on the importance of seat belts and the dangers of cigarettes, contributing to the push for government regulation in both areas — the Surgeon General's landmark 1964 smoking report cited Consumer Reports research. The magazine's association with labor leaders, meanwhile, temporarily landed it on a list of "subversive" organizations compiled by the House Un-American Activities Committee.

In the years since, the company has steadily expanded its testing program, and when necessary, used it to agitate for reform. For years, testers found that many SUVs on the market were prone to rolling over during sharp turns. Finally, in 2000, the increasing number of Ford SUVs rolling over caught the attention of Congress, and Consumer Reports' staff testified that a rollover test should be part of the government's safety testing protocol. The test is now a standard part of the National Highway Traffic Safety Administration's assessment of all new cars.

When Consumer Reports finds defects with other sorts of products, they're often reported to the manufacturer or the Consumer Product Safety Commission before publication, in some cases triggering redesigns or recalls. Most cases are resolved amicably, but the company has been sued 17 times in its history by manufacturers, often for libel. As a sign in the main hall of the testing labs proudly proclaims, the company has never lost or settled. (Consumer Reports has made at least one major error: in 2007, it erroneously reported that a number of infant car seats were unsafe, a report it retracted two weeks later, explaining problems with the testing procedure in detail.)

Recently, Consumers Union — since 2012, Consumer Reports' public policy arm — has gotten more and more involved in causes that have less to do with products reviewed. "Frankly, the marketplace isn't built to favor consumers — it's built to favor those who have money to advertise, or get involved in the political process," said Chris Meyer, a vice president for Consumers Union. "So we're all about moving the marketplace in a pro-consumer direction, whether it's in food, or auto safety, or health care, or finance."

In practice, he said, this means supporting policies that will save consumers time or money, or make them healthier or safer. These principles led the organization to support the Affordable Care Act in 2010, as well as financial reform, including the Dodd-Frank Act. Currently, it is supporting measures that would require food companies to label products that contain GMOs, lobbying the FDA to establish a definition for the term "all-natural," and actively opposing the proposed Comcast-Time Warner cable merger. "We believe that if it goes through, it'll reduce consumer choice, raise prices, and produce worse customer service," Meyer said. "So we're in the middle of a pretty spirited fight over it."

The product that Consumer Reports is most closely identified with is the automobile. To test cars, the company has a 327-acre driving center in rural Connecticut, built around a track originally used as a drag-racing strip. "Other automotive magazines will rent out a track for a day," said Mark Rechtin, an auto editor. "No one else has anything like this."

Each of the 20 or so staff members is responsible for buying a few cars per year. The company's goal is to review every model that goes on sale in the United States, besides high-end luxury cars that are above $80,000 or so. To buy them anonymously, staff constantly visit new dealerships. "I've been buying cars for 15 years here," said operations director Jennifer Stockburger, gesturing around a hangar filled with 50 new sedans, minivans, SUVs, and sports cars, "and I've never been to the same dealership twice."

Before each car is formally tested, she and other staff get to enjoy one of the major perks of the job: driving them around town. Every single car gets around 2,000 or miles, both as a breaking-in period and so staff members can take notes on how the cars perform during everyday tasks, like commuting or buying groceries.

Afterward, each car goes through about 50 different formal tests: headlights, noise, transmission, comfort assessments. Some tests are pretty straightforward, and can be done with the car in place — ergonomics experts evaluate the user-friendliness of the displays, for instance, and the interior volume of the trunk and other cargo areas is measured with large telescoping metal-framed boxes or suitcases.

Consumer Reports' auto testing center in Connecticut. (Courtesy of Consumer Reports)

Other tests — such as fuel economy — are a bit more involved. "We don't just take EPA numbers," said Ryan Pszczolkowski, one of the drivers. "We actually put our own fuel meters in on the gas lines and then drive a ‘city' and a ‘highway' course." The highway course is an actual highway (Route 11, a few miles off the premises). The city course involves two drivers each doing a few laps of the track at circumscribed speeds: going 40, then slowing down to 20, then back to 30, then coming to a full stop, for instance.

The most distinctive test, by far, is the avoidance maneuver. In essence, it's meant to simulate how a car might handle if you were traveling at a high speed, then saw an obstacle in the road — say, a child — and had to swerve into the oncoming traffic lane, then swerve back as quickly as possible. There's no actual child, just a narrow lane, defined by cones, with a kink in the middle that forces a driver to swerve left, then right. The faster a car can drive through without hitting a cone or skidding out, the higher its mark for handling.

This test, like all the others, is usually conducted with an air of sober diligence. But when visitors come by the track, Pszczolkowski enjoys using it to demonstrate the handling capabilities of the sports cars on hand — on one recent day, a bright yellow 2014 Corvette Stingray, with 460 horsepower. To see how fast the car could take the turn, he accelerated down the drag strip, approaching the narrow lane of cones at highway speed. After slipping between the cones, he wrenched the speeding car left, then back right, somehow managing to stay in the jagged lane with tires screeching. Afterward, he was nonchalant, but if you're not a professional driver, it was likely the most difficult driving maneuver you've ever seen carried out in your life. "If an average driver tries to make a move like that, he loses control of the car," he said.

In 2011, Consumer Reports lost $3.5 million. In 2012, it lost another $2 million. For decades, the nonprofit had usually generated tens of millions of dollars annually — money it was able to invest in its testing infrastructure — but a declining subscription base and increasing competition from free online content made for big challenges. Consumer Reports, in other words, was dealing with the same crisis facing virtually all of print journalism.

The company board responded by laying off 13 percent of its staff, splitting the company into two separate divisions (Consumer Reports, which publishes the magazine, and Consumers Union, which engages in policy work and had been the magazine's parent company), and eventually replacing its editor-in-chief, Kim Kleman. In the short term, the restructuring — along with the declines in digital and print subscriptions leveling off — has helped, with the company once again turning profits of $4.3 and $7.4 million in 2013 and 2014, respectively. But the long-term question remains: can a company that pays for an incredibly meticulous review process mostly through subscriptions survive in the digital age?

TVs are tested with a standard clip — of two tennis announcers at the US Open. (Corey Olsen)

Like other areas of journalism, the field of product reviewing has matured considerably since the early days of the web. Sites like the The Wirecutter and Outdoor Gear Lab, for instance, have built devoted followings with thorough testing processes and consumer-oriented reviews. These testing methods might not quite match the rigor of Consumer Reports', but they're still very good, and anyone can read them online for free. Full access to the online Consumer Reports ratings database, meanwhile, costs $39 per year. The company currently has roughly three million digital subscribers, as well as four million subscribers to the print magazine.

What can Consumer Reports do to set itself apart in the future, and keep building that digital subscriber base? Ellen Kampinsky, who was hired from Newsweek to replace Kleman as editor-in-chief of Consumer Reports in April 2014, said she wants to focus on showing readers why the magazine is uniquely trustworthy.

"I think native advertising — and the general blurring that's going on across the journalism landscape — is eroding the trust of readers," Kampinsky said. "When you read Consumer Reports, nobody but you and other readers is paying for it. That's something we need to work harder to trumpet to readers." However, while the company doesn't run ads in print or online, like most review sites, it does send readers who want to buy the products rated to Amazon, which provides a fee in exchange. This doesn't make Consumer Reports beholden to any specific manufacturers, but it does mean that it profits when users do more buying.

Perceptions of Consumer Reports online — and whether it carries the same imprimatur of reliability and objectivity as it does in print — are especially important because it has a problem common to many magazines: an aging print subscriber base. When Kampinsky was hired, the average reader was over 60 years old, a number that had been steadily rising for years. The degree of this problem was captured in the very first line of a 2008 Gizmodo article about a visit to the testing labs: "You probably only read Consumer Reports if a) you are at your grandparents' house or b) you are a grandparent yourself."

In response, Kampinsky has led a redesign of the magazine: introducing some new departments, putting more emphasis on photography, and including more reader feedback. "We want it to be more readable," she said. "Let's say you don't want to buy a lawnmower or a car this month — there should still be something there for you."

Strips of fabric stained with blood, cocoa, and other materials are used to test the effectiveness of washing machines. (Corey Olsen)

In September 2014, the company also got a new CEO: Marta Tellado, who came from the Ford Foundation to replace the retiring James Guest. She has broad ambitions for the organization's involvement in public policy. Among other things, she notes, people are increasingly — and justifiably — concerned with privacy threats online. "We're at a time when consumers need a reliable partner more than ever," she said. "I think that right now, the consumer movement is poised to be reintroduced and taken up by a new generation."

The biggest reason to be optimistic about Consumer Reports' future is that, compared to most print magazines, it realized the importance of the web relatively quickly. "They were one of the early leaders on that," said Samir Husni, a magazine analyst and journalism professor at the University of Mississippi. That's allowed the magazine to build a large group of digital subscribers, one that is more than three times as large as the more-heralded New York Times' digital subscriber base and may be bigger than any other outlet's.

Sites like The Wirecutter aside, most product reviews on the internet, especially for things other than tech products, are not particularly high-quality or trustworthy. And while it may be tough to imagine young people subscribing to a review website when there are hundreds out there for free, that may change as more and more millennials enter the car- and refrigerator-buying phases of their lives. The paywall-supported, subscriber-based model probably won't work for most journalistic outlets, but we've seen a bit of evidence that it might work for one or two especially high-prestige publications in any given field — as it's worked, so far, for the New York Times and the Wall Street Journal.

If anyone can survive this way in the competitive field of product reviews, it's Consumer Reports. "Because of the bombardment of information — because everybody and their cousin can now review a product — there's more need than ever for a well-curated, authoritative review that will really tell you what's going on," Husni said. "You need Consumer Reports, or something similar to it, more than ever."

Designer: Tyson Whiting
Photographer: Corey Olsen

Local police should not be your go-to source for iPhone safety news

The Future Perfect 50

Jerry Chow wants to make quantum computers you can actually count on

The Future Perfect 50

Alec Stapp and Caleb Watney have a plan for national progress

View all stories in Technology