I Am 95 Percent Confident June 9, 2013
Posted by Peter Varhol in Education, Technology and Culture.Tags: big data, statistics
add a comment
I spent the first six years of my higher education studying psychology, along with a smattering of biology and chemistry. While most people don’t think of psychology as a disciplined science, I found an affinity with the scientific method, and with the analysis and interpretation of research data. I was good enough at it so that I went from there to get a masters degree in applied math.
I didn’t practice statistics much after that, but I’ve always maintained an excellent understanding of just how to interpret statistical techniques and their results. And we get it wrong all the time. For example:
- Correlation does not mean causation, even when variables are intuitively related. There may be cause and effect, or it could be in reverse (the dependent variable actually causes the corresponding value of the independent variable, rather than visa versa). Or both variables may be caused by another, unknown and untested variable. Or the result may simply have occurred through random chance. Either way, a correlation doesn’t tell me anything about whether or not two (or more) variables are related in a real world sense.
- Related to that, the coefficient of determination (R-squared) does not “explain” anything in a human sense. There is no explanation in our thought patterns. Most statistics books will say that the square of the correlation coefficient explains that amount of variation in the relationship between the variables. We interpret “explains” in a causative sense. Wrong. It’s simply that the movement between two variables is a mathematical relationship with that amount of variation. When I describe this, I prefer using the term “accounts for”.
- Last, if I’m 95 percent confident there is a statistically significant difference between two results (a common cutoff for concluding that the difference is a “real” one), our minds tend to interpret that conclusion as “I’m really pretty sure about this.” Wrong again. It means that if I conducted the study 100 times, I would draw the same conclusion 95 times. And that means five times I will draw the opposite conclusion.
- Okay, one more, related to that last one. Statistically significant does not mean significant in a practical sense. I may conduct a drug study that indicates that a particular drug under development significantly improves our ability to recover from a certain type of cancer. Sounds impressive, doesn’t it? But the sample size and definition of recovery could be such that that the drug may only really save a couple of lives a year. Does it make sense to spend billions to continue development of the drug, especially if it might have undesirable side effects? Maybe not.
I could go on. Scientific experiments in the natural and social sciences are valuable, and they often incrementally advance the field in which they are conducted, even if they are set up, conducted, or interpreted incorrectly. That’s a good thing.
But even when scientists get the explanation of the results right, it is often presented to us incorrectly, or our minds draw an incorrect conclusion. A part of that is that a looser interpretation is often more newsworthy. Another part is that our minds often want to relate new information to our own circumstances. And we often don’t understand statistics well enough to draw informed conclusions.
Let us remember that Mark Twain described three types of mendacity – lies, damned lies, and statistics. Make no mistake, that last one is the most insidious. And we fall for it all the time.
Of Software, Marketing, and Diversity June 7, 2013
Posted by Peter Varhol in Technology and Culture.Tags: marketing, Silicon Valley
add a comment
Oh, Shanley. It pained me to read your latest missive on the marketing chick and the culture of misogyny. It pained me because you are sometimes right, but perhaps more often not (or, to be fair, visa versa). Yes, I’ve seen what you describe, although I would suspect not with the raw intensity you have.
Part of that raw intensity, I suspect, is driven by the Silicon Valley culture. Whatever exists in America is magnified by the hype that the Valley types like to bring to anything that exists within its confines.
Many of us are too full of ourselves to recognize the value of others in a common endeavor. Because we are not confident of our own position, we naturally if unreasonably order ourselves at the top of an uncertain food chain. That means we tend to denigrate those without our particular skill set.
But that particular culture is nowhere near universal. Many (I have no idea what percentage, but I suspect most) grow out of it. Those who don’t are sentenced to a life of bad pizza, online games, and no social life. They pay for their inability to adapt.
There is no single techie who can build, market, sell, and service a software product, and that hasn’t been possible for at least 30 years, if ever. We all know that the most elegant and advanced technical solution is not likely to win in the market. Those that build those technical solutions are at a loss to understand why they aren’t accepted, and are more likely to blame others than themselves.
So we create the marketing chick and denigrate her, even though marketing is a necessary skill for success.
It is a human failing, with the intensity increased by the win at all costs mentality in Silicon Valley. Perhaps you see so much of it because of where you are. That’s not to say it is right. But it is to say that elsewhere it may be different.
Really Big Data and the Pursuit of Privacy June 7, 2013
Posted by Peter Varhol in Technology and Culture.Tags: big data, NSA, privacy
add a comment
There’s been so much excitement these days about the commercial potential of Big Data that we’ve forgotten that the Federal government is in the best position to obtain and analyze many terabytes of data. We were reminded of that in a big way following revelations that the National Security Agency (NSA) was obtaining under secret court order information about all phone calls made by Verizon customers. I am not a Verizon customer, but I have no doubt that the same court orders exist for other carriers.
(Interesting side note: Many years ago, after I earned my MS in Math, I had a job offer to join the NSA as a civilian cryptologist. Perhaps now I wish I had taken it.)
With virtually unlimited fast computing power, the NSA can identify patterns that provide a basis for follow-up law enforcement activities.
Here’s a simple example of how it works. A computer program identifies twenty or so different phone numbers in the New York City area that have called the same number in, oh, the Kingdom of Jordan about two hundred times in the last two months. The number in Jordan is a suspected front (through other sources) for some sort of terrorist activity. This connection might provide law enforcement reason to look more closely into the activities of those making these calls. That’s not inherently a bad thing.
Of course, there are ways that terrorists and criminals can combat this, such as the use of prepaid and disposable cell phones bought with cash, calling cards, and even random pay phones. At best, analyzing call records represents one tool among many in the pursuit of wrongdoing, and not really a “Big Brother is Watching” scenario.
From a privacy standpoint, I’m mostly sanguine about the NSA collecting and analyzing calling data. I’m not engaged in terrorist or criminal activities, and my phone calls are just a few data points among the billions out there. I’m not directly threatened, or even inconvenienced.
But . . . there may be a slippery slope here. The definition of suspicious calling activity may gradually expand to include things that aren’t illegal, but perhaps just unethical or embarrassing. Once you have the data and the computing power, you can start looking for other things. Call it scope creep, an all-too-common affliction of many projects.
And in a larger sense, many of our freedoms are actually constructed on the premise that the Federal government cannot connect the dots between the myriad of records held by the many Federal agencies on each of us. Call it privacy by disorganization, but it has worked at least throughout my lifetime to protect my liberties. But thanks to the advancements made in Big Data over the last several years, we may be seeing the end of that type of protection.
Security and privacy represent direct tradeoffs. Unlike many Americans, I would prefer to be a little less secure and a little more private. But the majority does rule, and I do believe that the majority has little issue with the current state of affairs.
You Don’t Have to Retire to a University Town April 28, 2013
Posted by Peter Varhol in Education, Technology and Culture.Tags: education
add a comment
Not that I’m looking at retirement anytime soon; I love what I do for a living, and can give it a lot of energy. But there has been a push over the last decade or so for people to retire to university towns where they can experience the educational opportunities inherent in the academic environment.
I call BS on that life strategy.
I’m finishing up a MOOC through Coursera, and I have to say that the experience has rekindled an enthusiasm for higher education that I may have lost since I (voluntarily) left my tenure-track position in computer science and math, now almost seventeen years ago.
I have to give credit to Clay Shirky, whose tweet led me in the direction of the topic and course. The course is A Beginner’s Guide to Irrational Behavior, taught by Dan Ariely at Duke University. The topic fits well into my present interest in understanding and compensating for bias in software testing.
I really lacked the time to do it. But the course organization is a wonderful combination of freedom to work on your own schedule (I’ve been on business travel three times in the last three weeks), and the structure needed to see it through. You can fully participate in online hang-outs, wikis, readings, and lectures, do what is necessary to satisfactorily complete the course (this course requires an average score of 85 through all exercises and quizzes), or just pick and choose, depending on your interests and time.
Competitive person that I am, I chose to work toward course completion, while doing little of the extracurricular activities that can add spice to a learning experience. I still work for a living, after all.
The fact of the matter is that you can live just about anywhere in the world with broadband Internet access, and still experience outstanding educational opportunities, makes the idea of living in a university town less vital to intellectual stimulation. If you’re looking to a university town in retirement to keep your intellectual edge, you may be shortchanging yourself.
Testing and Tester Bias March 30, 2013
Posted by Peter Varhol in Software development.Tags: Kahneman
add a comment
Software testers are increasingly looking at how to approach the problems inherent in testing, and how the ways that we think, and the bias that we bring to our work affects the conclusions we draw. Because we can’t test every aspect of an application exhaustively, much of the testing process is based on our past practices, judgment, and decisions that we make based on incomplete and often inconclusive evidence.
Much of the foundation behind examining how we approach and solve problems comes from Daniel Kahneman’s landmark book Thinking, Fast and Slow. In the book, Kahneman, who is a psychologist and Nobel Prize-winning economist, defines two types of thinking. The first is System 1 thinking. System 1 thinking is a fast, involuntary, and largely instinctive method of thought that enables us to function on a daily basis. If we sense things in our surroundings such as movement or sounds, our System 1 thinking interprets it instantly and responds if necessary.
System 2 thinking, in contrast, is slow, deliberate, and more considered. If we’re faced with a situation we haven’t encountered before, or a difficult problem, we engage System 2 thinking and make a more focused decision. We take the time to think through a problem and come up with what we believe is the best answer or response.
Each has its respective advantages and disadvantages. System 1 thinking is good enough for most low-risk or immediate decisions, but is too simplistic for more difficult situations. If we try to make complex decisions without engaging System 2 thinking, we risk making less than optimal decisions due to our own biases or a lack of information at the time a decision is made.
While System 2 thinking is more accurate in complex situations, it takes time to engage and think through a problem. It’s a conscious process to decide to think more deeply about a situation, and to begin determining how to approach it. For most simple decisions in our lives, it’s overkill and not timely enough to be useful. System 2 thinking is also hard work, and can cause decision fatigue if done too often.
As a result, each type of thinking introduces biases into our daily work, which affects how we test and what conclusions we draw from our data. We may depend too much on System 1 to draw fast conclusions in cases where further thought is needed, or on System 2 so much that we become fatigued and begin to make mistakes because we are mentally tired.
In practice, it’s better to alternate the two types of thinking so as to not overuse either. If we have complex data to collect and evaluate, it helps if we break up that process with occasional rote activities such as automated regression testing. For example, exploratory testing is a gradual learning process that requires extensive System 2 thinking. In contrast, executing prepared test scripts is largely a rote exercise. Being able to alternate every day or two between the two approaches can keep testers sharp.
In my upcoming posts, I’ll take a closer look at biases that come about as a result of how we think as we approach a problem. Then I’ll look at how these biased decisions can affect how we test and what we find when we do.
Can Our Shopping Cards Save Our Lives? March 17, 2013
Posted by Peter Varhol in Software platforms, Technology and Culture.Tags: big data
add a comment
I’m a bit of a throwback when it comes to certain applications of technology. In addition to not using Facebook, I don’t have supermarket rewards cards, or even use a credit or debit card at the supermarket. My reasoning for the latter is simple – I would prefer not to have the supermarket chain know what I’m eating. I realize that I may be giving up coupons or other special deals by not identifying myself, but I’m willing to accept that tradeoff. It’s not a big deal either way, but it’s how I prefer to make that particular life decision.
But now there seems to be better reasons to use your supermarket reward card – according to this NBCNews.com article, it may save your life. Really.
The story goes something like this. When there is a known food contamination, health officials can see who bought that particular food, and approach those people individually, rather than send out vague alerts that not everyone sees or hears.
Count me as dubious. This is really a sort of pie-in-the-sky application of Big Data that people can dream up when they picture the potential of the data itself. It would take weeks to reach all of the buyers of a particular contaminated product, even if you could match all of the different systems and databases together somehow. By then, the scare would have run its course.
The reality is that such data is stored in hundreds or thousands of different systems, without any means of pulling them together, let alone using it to query on a specific product across millions of purchases.
And then, of course, there are people like me, who still insist on dealing in cash, and remaining somewhat anonymous. Although they could take my photo in the supermarket, and rather quickly match it up to my other identified photos on the Internet, where I am well known as a speaker and writer.
The idea is intriguing, but it falls into the same tradeoff as many other applications of technology in society today. We can do things to make ourselves safer, but at the cost of providing more information. Some don’t seem to have a problem with the latter, but I, in my doddering middle age, do.
On Silicon Valley, Productivity, and Diversity March 11, 2013
Posted by Peter Varhol in Technology and Culture.Tags: diversity, Silicon Valley
add a comment
Well, there is a huge and unmanageable topic if I ever heard one. So I’ll be brief. The thought started with a blog post by Shanley Kane, a product manager in Silicon Valley, who took issue with others who offered their take on sexism in IT.
Long story short, Shanley is mostly right. Culture is important. The older I get, the more I want to be somewhere that shares my values. We spend an awful lot of time at work (even if I work remotely), and we don’t want to feel like we are alienated during that time.
But one of my values is, well, discomfort. I want to be exposed to ideas that I haven’t been exposed to before. I want to think, and re-think, my value proposition, and what I bring to any particular table of effort. The fact that I was born a white male, in a working class and blue collar community, gives me a particular point of view. And guess what? That point of view isn’t shared by the vast majority of people in this world. And in the grand scheme of things they count; in many cases probably more so than I do at this particular time of my life.
Her point, I think, is that there is a dark underside of the culture story in high tech, and in particular in Silicon Valley. And it doesn’t boil down to race, or ethnic background, or education, or anything like that. It means different ideas. And if we aren’t different, in some fundamental way, we don’t have substantially different ideas.
I was surprised at the negative responses to Shanley’s post, and to subsequent writings on the topic. Well, maybe not particularly surprised, but certainly disappointed. If we don’t challenge our thinking, sooner or later we will probably fail, and in a spectacular way.
I think that maybe diversity, at least in high tech companies, isn’t a matter of the color of skin, or race, or anything like that. The comments that veer off into that realm miss the point in a very real way. Instead, let me ask this question. What have you done to make yourself emotionally and intellectually uncomfortable today? If the answer is nothing, you are almost certainly shortchanging yourself.
On Yahoo and Working Remotely February 25, 2013
Posted by Peter Varhol in Technology and Culture.Tags: remote, Yahoo
add a comment
By now most of us have heard that Yahoo has pretty much canned any attempt at working from home amongst its employees.
I’ve been on both sides of this equation. Circa 2002, my employer at the time, a major software vendor, summarily fired all of its remote employees (they all received their FedEx packages on the same day) and instilled a strict office policy, claiming that it wanted to instill its unique culture across the company.
The fact that its unique culture was decidedly command and control in the style of the US automotive companies didn’t seem to matter (there is a reason I use that analogy). To be honest, it was a poor culture at best. And the fact that it reversed course a few years later has more to do with laziness than belief.
I’ve worked primarily from home since 2006. Today I work mostly for a small software tools vendor in the Midwest. There is good in it, and there is challenging. I make decent money, drive a 15-year old car that accumulates perhaps 4000 miles a year, and I have a commute down two flights of stairs in the morning.
The challenging aspect is that I see the corporate culture from a distance. Many people wouldn’t pay attention to it, but in my mind it is the key part of being a remote employee. I have tried my best to fit in, and I think I do so well. Many of my colleagues bring in snacks on their birthdays; I send a basket of Boston whoopee pies (I’m told they have caused riots). Overboard? Perhaps, but most everyone there knows who I am (confession – I don’t know all of them).
I appreciate the flexibility, but I try very hard to give the appearance of the guy in the next cubicle. I think I’ve largely succeeded, and seem to be well-liked and mostly appreciated.
Of course, I do the work. That’s really the least of it, and the part I think trips up at least some remote workers. The biggest issue is fitting in, and being visible. You can’t hide under the desk. Culture is very important, and a remote worker has to do both the work and the culture to be successful.
Yahoo will likely reverse course at some point, but gradually and quietly. Unless it perceived it had a big problem, it would not have taken this step. But for those who understand and follow the culture, this too shall pass.
