Sunday, July 22, 2007

A sideways smack at the focus-group methodology

Focus groups -- the most picked-on market research methodology of them all -- got a sideways smack from James Surowiecki in the New Yorker's July 9 edition. I say "sideways" because the point of the article was not to dump on focus groups, but to tout the much more fashionable prediction markets as a potential method for identifying hits and misses in the consumer market space. The slap at focus groups was just a by-product:
"Prediction markets avoid many of the faults of focus groups, which tend to be dominated by the loudest and most opinionated people, to be driven toward consensus decision, and to discourage disagreement, making them of limited usefulness. ('Seinfeld,' famously, was a complete bust with focus groups.)"
First, surely someone can come up with something newer than Seinfeld about which focus group respondents were wrong. That example is aging.

Second, note that when people complain about focus groups they always complain that they were wrong at predicting a product's success (usually) or failure (occasionally). Yet among professional qualitative researchers, the idea that you might use focus groups to predict *anything* is ludicrous. So, once more, repeat after me: Focus groups help you discover why and how -- not how many. The job of quantifying and predicting falls to quantitative research.

Here's a similar example from another New Yorker writer, Malcolm Gladwell. In his book Blink he tells the story of the Aeron chair: Apparently in the early 1990s, when the concept for the Herman Miller Aeron chair was being researched, focus groups were held with "facility managers and ergonomic experts" -- the target market. These group participants mostly did not like the odd aesthetics of the chair. Gladwell gives the Herman Miller folks a big pat on the back for going forward with the launch anyway. And what happened? Quoting Gladwell:
"In the beginning, not much. The Aeron, after all, was ugly....however, in California and in New York, in the advertising community and in Silicon Valley, it became a kind of cult object....It began to appear in films and television commercials....and by the end of the 1990s....was the best-selling chair in the history of the company."
I'd like to point out here that this is *exactly* what the focus group research indicated. Not predicted -- but indicated. The target market didn't like the chair and wasn't likely to adopt it. That is a finding -- not an implication, insight, or recommendation. (The fact that the focus group did not like Seinfeld was also a finding, not a recommendation.)

A smart qualitative researcher would have been able to dig deep and get a sense for what those findings meant. It's *not* about whether the groups hate the Aeron and Seinfeld -- it's about gaining insights that help Herman Miller put together a product-launch strategy for the Aeron that....about understanding the kinds of reactions Seinfeld elicits from a specific small demographic. The mistake comes with premature closure. The findings are not insights or recommendations. They are not quantified. They are not predictions.

The purpose of focus groups is not to predict or quantify. The purpose is to understand how and why, to get the "lay of the land," to come to an understanding of the opinions of a particular, very small group. There are lots of ways to do this, of course -- observational research, such as traditional ethnography or an online hybrid; one-on-one interviews; interviews and groups online, in virtual worlds; interviews and groups with target markets, with existing customers. Gathering 8 to 12 strangers in a room and asking previously written, non-directed questions of them, leading them into a group discussion, is a perfectly legitimate method -- as long as it's done correctly.

The prediction market, as cool and up-to-the-moment as it is, won't give you the depth of insight and understanding in people's likes, dislikes, and motives that qualitative research will. But I can totally see how it would be way better for predicting who'll win the Oscars, and may also be better at predicting how well a given book will sell (the subject of the Surowiecki article).

I'll give Surowiecki the last word, because despite his sideways slap at focus groups, this indicates that he *does* "get" consumer research -- but like a lot of others, he just can't resist smacking the focus-group methodology around a little.
"The collective intelligence of consumers isn’t perfect—it’s just better than other forecasting tools. The catch is that to get good answers from consumers you need to ask the right kinds of questions; asking the market to predict how many copies a book will sell, which requires predicting how a wide readership will behave, is better than asking the market to predict which manuscript will get a book deal, which requires predicting the decisions of a small number of editors. (The Simon & Schuster experiment with MediaPredict, unfortunately, focusses more on the latter.) And you need a critical mass of people to participate."

No comments: