jump to navigation

We Don’t Understand Our Numbers March 27, 2016

Posted by Peter Varhol in Strategy, Technology and Culture.
Tags:
trackback

I recently bought The Cult of Statistical Significance: How the Standard Error Cost Us Jobs, Justice, and Lives, by Stephen T. Ziliak and Deidre N. McCloskey.

Here’s the gist. Statistics is a great tool for demonstrating that a difference found between two sampling results is “real”.  What do I mean by real?  It means that if I measured the entire population, rather than just took samples, I would know that the results would be different.  Because I sample, I have uncertainty, and statistics provide a way to quantify the level of uncertainty.

How different? Well, that’s the rub.  We make certain assumptions about what we are measuring (normal distribution, binomial distribution), and we attempt to measure how much the data in each group differ from one another, based on the size of our sample.  If the two types of results are “different enough”, based on a combination of mean, variation, and distribution, we can claim that there is a statistically significant difference.  In other words, it there a real difference in this measure between the two groups?

But is the difference important? That’s the question we continually fail to ask.  The book Reclaiming Conversation talks about measurements not as a result, but as the beginning of a narrative.  The numbers are meaningless outside of their context.

Often a statistically significant difference becomes unimportant in a practical sense. In drug studies, for example, the study may be large enough, and the variability low enough, to confirm an improvement with an experimental drug regimen, but from a practical sense, the improvement isn’t large enough to invest to develop.

My sister Karen, a data analyst for a medical center, has pointed out to me that significance can also be in the other direction. She collects data on patient satisfaction, and points out that even minor dissatisfaction can have a large effect across both the patient population and the hospital.

That’s just one reason why the measurement is the beginning of the conversation, rather than the conclusion. The number is not the fait accompli; rather, it is the point at which we know enough about the subject to begin talking intelligently.

Advertisements

Comments»

No comments yet — be the first.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: