Do your Analytics Cheat the Truth?

The rise of analytics-informed insight is a welcome development. The disingenuous and deceptive manner in which many of these statistics are presented is not. I’m simultaneously stunned and disappointed by how manipulative these analytics have become at the very highest levels of enterprise oversight. The only thing more surprising is the fact that many senior executives are unwilling to ask simple questions about the analytics they see.

Increasingly, I observe statistical sophisticates indulging in analytic advocacy – that is, their numbers are deployed in the service of influencing arguments rather than identifying underlying dynamics. This is particularly disturbing because while the analytics – in the strictest technical sense – accurately portray a situation, they do so in a way that discourages useful inquiry.

Analytics presentations should always identify the outliers – how they were defined and dealt with, and, most importantly, what the analytics would look like if they didn’t exist. It’s astonishing what you find when you make the outliers as important as the aggregates and averages.

My favourite example of this comes, naturally enough, from Harvard. Few people realise that, in fact, the average net worth of Harvard dropouts vastly exceeds the average net worth of Harvard graduates.

The reason for this is simple. There are of course many more Harvard graduates than Harvard dropouts. But the ranks of dropouts include Bill Gates, Mark Zuckerberg and Polaroid’s Edwin Land, whose combined net worth equals roughly $70 billion. Take that number and divide it by the significantly smaller amount of dropouts, and the result will suggest that the average Harvard dropout is much wealthier than the average Harvard graduate.

This is, of course, ridiculous. Unfortunately, it is no more ridiculous than what one finds in a statistically significant number of analytics-driven boardroom presentations. The misdirection – and mismanagement – associated with outliers is the most common pathology found in otherwise stats-savvy organisations.

Always ask for the outliers. Always request that analysts display what their data would look like with the outliers removed. There are other equally important ways to wring greater utility from aggregated analytics, but it’s best to start with the outliers. Analytics that mishandle outliers are ''outliars.''

(Michael Schrage is a research fellow at the Sloan School's Center for Digital Business at the Massachusetts Institute of Technology. He is the author of "Serious Play'' and the forthcoming "Getting Beyond Ideas.'')

© 2011 Harvard Business Publishing