##
We Don’t Understand Our Numbers *March 27, 2016*

*Posted by Peter Varhol in Strategy, Technology and Culture.*

Tags: statistics

trackback

Tags: statistics

trackback

I recently bought The Cult of Statistical Significance: How the Standard Error Cost Us Jobs, Justice, and Lives, by Stephen T. Ziliak and Deidre N. McCloskey.

Here’s the gist. Statistics is a great tool for demonstrating that a difference found between two sampling results is “real”. What do I mean by real? It means that if I measured the entire population, rather than just took samples, I would know that the results would be different. Because I sample, I have uncertainty, and statistics provide a way to quantify the level of uncertainty.

How different? Well, that’s the rub. We make certain assumptions about what we are measuring (normal distribution, binomial distribution), and we attempt to measure how much the data in each group differ from one another, based on the size of our sample. If the two types of results are “different enough”, based on a combination of mean, variation, and distribution, we can claim that there is a statistically significant difference. In other words, it there a real difference in this measure between the two groups?

But is the difference important? That’s the question we continually fail to ask. The book Reclaiming Conversation talks about measurements not as a result, but as the beginning of a narrative. The numbers are meaningless outside of their context.

Often a statistically significant difference becomes unimportant in a practical sense. In drug studies, for example, the study may be large enough, and the variability low enough, to confirm an improvement with an experimental drug regimen, but from a practical sense, the improvement isn’t large enough to invest to develop.

My sister Karen, a data analyst for a medical center, has pointed out to me that significance can also be in the other direction. She collects data on patient satisfaction, and points out that even minor dissatisfaction can have a large effect across both the patient population and the hospital.

That’s just one reason why the measurement is the beginning of the conversation, rather than the conclusion. The number is not the fait accompli; rather, it is the point at which we know enough about the subject to begin talking intelligently.

## Comments»

No comments yet — be the first.