jump to navigation

Bias and Truth and AI, Oh My October 4, 2017

Posted by Peter Varhol in Machine Learning, Software development, Technology and Culture.
Tags: ,
add a comment

I was just accepted to speak at the Toronto Machine Learning Summit next month, a circumstance that I never thought might happen.  I am not an academic researcher, after all, and while I have jumped back into machine learning after a hiatus of two decades, many more are fundamentally better at it than me.

The topic is Cognitive Bias in AI:  What Can Go Wrong?  It’s rather a follow-on from the presentations I’ve done on bias in software development and testing, but it doesn’t really fit into my usual conferences, so I attempted to cast my net into new waters.  For some reason, the Toronto folks said yes.

But it mostly means that I have to actually write the presentation.  And here is the rub:  We tend to believe that intelligent systems are always correct, and in those rare circumstances where they are not, it is simply the result of a software bug.

No.  A bug is a one-off error that can be corrected in code.  A bias is a systematic adjustment toward a predetermined conclusion that cannot be fixed with a code change.  At the very least the training data and machine learning architecture have to be re-thought.

And we have examples such as these:

If you’re not a white male, artificial intelligence’s use in healthcare could be dangerous.

When artificial intelligence judges a beauty contest, white people win.

But the fundamental question, as we pursue solutions across a wide range of applications, is:  Do we want human decisions, or do we want correct ones?  That’s not to say that all human decisions are incorrect, but only to point out that much of what we decide is colored by our bias.

I’m curious about what AI applications decide about this one.  Do we want to eliminate the bias, or do we want to reflect the values of the data we choose to use?  I hope the former, but the latter may win out, for a variety of reasons.


The Future is Now June 23, 2017

Posted by Peter Varhol in Algorithms, Technology and Culture.
Tags: ,
add a comment

And it is messy.  This article notes that it has been 15 years since the release of Minority Report, and today we are using predictive analytics to determine who might commit a crime, and where.

Perhaps it is the sign of the times.  Despite being safer than ever, we are also more afraid than ever.  We may not let our electronics onto commercial planes (though they are presumably okay in cargo).  We want to flag and restrict contact with people deemed high-risk.  We want to stay home.  We want the police to have more powers.

In a way it’s understandable.  This is a bias described aptly by Daniel Kahneman.  We can extrapolate from the general to the particular, but not from the particular to the general.  And there is also the primacy bias.  When we see a mass attack, was are likely to instinctively interpret that as an increase in attacks in general, rather than looking at the trends over time.

I’m reminded of the Buffalo Springfield song: “Paranoia strikes deep, into your lives it will creep.”

But there is a problem using predictive analytics in this fashion, as Tom Cruise discovered.  And this gets back to Nicholas Carr’s point – we can’t effectively automate what we can’t do ourselves.  If a human cannot draw the same or more accurate conclusions, we have no right to rely blindly on analytics.

I suspect that we are going to see increased misuses of analytics in the future, and that is regrettable.  We have to have data scientists, economists, and computer professionals step up and say that a particular application is inappropriate.

I will do so when I can.  I hope others will, too.

Being a Curmudgeon Has its Benefits August 1, 2016

Posted by Peter Varhol in Technology and Culture, Uncategorized.

I occasionally wax personal in my blog, as I did a year ago when I was facing a serious cancer diagnosis (the diagnosis was ultimately incorrect, and I am healthier than ever). Occasionally I just have to say something about a particular moment, whether or not it relates to my target blog topics.

This morning I got a regular email newsletter from Marc Cendella of The Ladders, a job search service for salaries over $100K.  The title was “When the kid interviewing you says you’re too old…”  In it, Cendella says that age discrimination in hiring is prevalent, and offers the older job seeker a checklist of items to attempt to overcome that bias.

Here is where I call a foul. Certainly there are things that a job seeker can do in order to make him- or her self appear to be a better fit for a given job.  In general, those things range from the common-sensical (be engaged and current in your profession and energetic in your life pursuits) to the absurd (facelifts and hair coloring).

But it’s a two-way street. Why not also suggest to the hiring managers that they might have a bias that is not well serving their organization, and how they might recognize and correct that deficiency?

Oh, that’s right. Businesses like The Ladders make money from those companies doing the hiring, not from job seekers.  The Ladders would rather tell the job seeker to change, rather than the hiring manager.

I would imagine that in a lengthy career spanning a dozen or more jobs and dozens of interviews, I have experienced some types of bias and discrimination. Probably everyone has; we tend to form initial impressions of someone we just met in under a second, and those first impressions can be both unconscious and difficult to overcome.

Bias in hiring is particularly difficult to demonstrate, as there could be any reason or no reason to not be selected for a job. The prospective employer certainly isn’t telling (usually), so most of this left to speculation or inference, and not even worth considering, let alone actionable.

But I found this newsletter from The Ladders to be singularly offensive. I instinctively interpreted it as “It’s not my problem that I am biased, it’s yours in that you are too old.”  I deeply resent that Cendella says that it’s a problem for job-seekers, rather than a problem for hiring managers (or for both).  If hiring managers let such biases creep into their decision process, they are doing both themselves and their organization a serious disservice.

I have always been sanguine about bias in hiring. My attitude has been that if I am discounted because of a personal characteristic outside of my control, it’s a place I probably wouldn’t want to work at anyway.

The fact of the matter is that unless we die young, or hit the jackpot, we are all destined to become older workers. Everyone, deal with it.

Cultural Fit is Bullshit January 23, 2016

Posted by Peter Varhol in Technology and Culture.
Tags: ,
1 comment so far

This post has spent months (maybe years) in the making. And yes, I like the rhyme of the title.  I revisited it yet again after reading this article on minority hiring in Silicon Valley in Bloomberg News.

I am not a mainstream techie. I earned two degrees in psychology before turning to programming and technology in general, in my late 20s.  I am not a very good techie.  I have a solid foundation to understand and explain concepts (I was a university professor at one point), but never more than an average coder.  Nevertheless, I’ve made mostly a decent living in technology, though not in Silicon Valley.  Just so you know where I’m coming from.

I used to believe that cultural fit was the preeminent job requirement. Now I understand that’s what the employer would like me to believe.  They could hire or fire on a whim, rather than what they actually need.  In fact, whether or not I could do that job has no impact on my hireability.

So minorities (and almost certainly others who don’t fit into pre-established norms) are at a disadvantage because they didn’t start coding when they were seven? This is where the bullshit starts.  Does that make them better coders?  Possibly, although certainly not provably.  Does that make them better contributors?  Now there is the rub.  I would argue no.

But we are befuddled by candidates who are savants at placing bits of data into processor registers and making it do backflips. That is a worthwhile skill, but it’s not the only skill necessary to succeed, as an individual, as a part of a team, and as a company.  Even in Silicon Valley.  If your teams are all A-list coders, you are missing out on some essential skills.  Yet you seem to be fine with that.

In my health issues over the last year, I was fortunate to encounter a couple of doctors who treated me as a person, rather than as a collection of symptoms. I challenge Silicon Valley to do the same.  Understand at a deep level what your teams need, and interview and hire based on those needs.  Understand not only the technical skills, but the social dynamics and complementary skills that are necessary for any team to succeed.  You are not doing so.

The mantra of cultural fit has enabled Silicon Valley to ignore deeper issues of team dynamics, skills needs, and what drives people to be successful. You hire people like you.  Or people that fit into a predetermined slot.  I get it, but you refuse to get out of your comfort zone to look at what might make you successful.  You are blind.  And, in the kingdom of the blind, the one-eyed man is king, if I may quote Desiderius Erasmus.

I have no right to do so, but I challenge Silicon Valley. Yes, you.

Cognitive Bias and Regression to the Mean April 29, 2014

Posted by Peter Varhol in Software development.
Tags: , ,
add a comment

We prefer to assign causality to events in our own lives, and in the world in general. If something positive happens, we tend to credit our intelligence, or dedication, or some other quality. If negative, we often blame others, or perhaps blame our own failings. Every day when the stock market closes, we read about how stocks have gone up or down for some perfectly understandable reason.

Bull. Most things in our lives and in the world don’t happen for a reason, or at least any reason we can readily identify. Our good fortune may be only peripherally related to our intelligence or abilities, and our bad fortune may simply arise from being in the wrong place at the wrong time.

Regression to the mean is simply one example of our need for causality, and how it results in bias. If we perform exceptionally well, we come to believe our own press releases, and behave as though we are high achievers. We might well be, but achievement is a slippery thing; it might disappear in a heartbeat.

Regression to the mean is a statistical concept. It simply notes that any time you get an exceptional result, it is unusual. Subsequent results are more likely to be closer to the average. It’s a concept often found in the natural sciences. For example, I am taller than either of my parents, so it is likely that my children (if I had any) would be shorter than me, since I am taller than many of my direct ancestors.

Applied to our lives, regression to the mean refers to the fact that what we do is a combination of skill and luck. We have little idea how much is skill, and how much luck. When we do exceptionally well at a task, we tend to attribute that to skill. When we do poorly, we often blame bad luck. Instead, exceptional performances are random (and rare) chance.

You can certainly argue that such a statistical concept doesn’t really apply to individual efforts, but I think the general principle holds. Sometimes we simply do better than other times, and it’s not clear that it reflects skill any more than (good or bad) luck.

Applied to software development and testing, regression to the mean gives us preconceived notions of the performance of the software based on who works on it. It makes us believe certain things about software based on the perceived superiority or inferiority of the team members based on our experiences.

The Role of Heuristics in Bias April 24, 2014

Posted by Peter Varhol in Software development.
Tags: , ,
add a comment

A heuristic is what we often refer to as a “rule of thumb”. We’ve experienced a particular situation on several occasions, and have come up with a step-by-step process for dealing with it. It’s purely System 1 thinking in action, as we assess the situation and blindly follow rules that have worked for us in the past.

And heuristics are great. They help us make decisions fast in situations that we’ve experienced in the past. But when the situation only appears similar, but is really different, applying our heuristic can have a very bad effect, if it’s not right.

Here’s a real life example. Years ago, I took flying lessons and obtained my pilot’s license. One of the lessons involved going “under the hood”. The hood is a plastic device that goes over your head (see image). When the hood is down, you can’t see anything. When the hood is raised, you can see the instrument panel, but not outside of the plane.


While the hood was down, the instructor pilot in the right seat put the plane into an unusual situation. That might be a bank, or a stall, or something that was unsustainable. When he raised the hood, I was required to use the instrument panel to analyze and diagnose the situation, and recover from it.

After several of these situations, I had developed a heuristic. I looked first at the turn and bank indicator; if we were turning or banking, I would get us back on course in straight flight. Then I would look at the airspeed indicator. If we were going too slow, I could lower the nose or advance power to get us back to a cruise speed.

This heuristic worked great, and four or five times I was able to recover the aircraft exceptionally quickly. I was quite proud of myself.

But my instructor figured out what I was doing, and the next time I applied my heuristic, it seemed to work. But I was fighting the controls! It wasn’t straight and level flight. I started scanning other instruments, and discovered that we were losing over a thousand feet a minute.

At that point, my heuristic had failed. But I wasn’t able to go back and analyze the situation. My mind froze, and if it weren’t for the instructor pilot, we may well have crashed.

The lesson is that when your heuristic doesn’t work, it may be worse than starting over at the beginning. You may simply not be able to.

Applying Cognitive Bias to Software Development and Testing April 21, 2014

Posted by Peter Varhol in Software development.
Tags: ,
add a comment

Through his psychology research, Daniel Kahneman (and his collaborator Amos Tversky) demonstrated that we are not rational investors. We make irrational decisions all the time, decisions that most definitely don’t optimize our expected utility. He proved this well enough that he was awarded a Nobel Prize in Economics.

Beyond economics, we exhibit the same behavior in other aspects of our lives, including our professional lives. Let’s take software testing as an example. We may have preconceived notions of how buggy a particular application is, and that will likely affect how we test it. We may have gotten that notion from previous experience with the development team, or from an initial use of a previous version of the software.

As a result of those preconceived notions, or biases, we are likely to plan and execute our work, and evaluate the results, differently than if they didn’t exist. If our prior experiences with the team or the software were negative, we may be overly harsh in our assessment of the software and its perceived flaws. If our experiences are positive, we may be willing to give questionable characteristics a free pass.

Lest it sounds like this is a conscious decision on our part, let me say right now that it’s almost always not so. It never occurs to us to think that we are biased. If we think of it at all, we believe that the bias is a good thing, because it puts us on alert for possible problems, or it gives us a warm fuzzy of the quality or fitness of the application.

Bias can be a good shortcut to the correct or optimal decision. More often, it is a way of analyzing a situation poorly and making an incorrect or less-than-ideal decision. Even if it might result in a good outcome, it’s incumbent of each of us to realize when we are being influenced by our own beliefs, and to question those beliefs.

We tend to think of software development and testing as highly analytical and scientific endeavors, but the fact is that they are both highly subjective and social. We work in close-knit teams, and the decisions are highly situational based on the exact circumstances of the problem. We tend to overestimate our individual and group abilities, and underestimate the complexity of the problems to be solved.

Further, we tend not to learn relevant lessons from past experiences, instead remaining overly optimistic, often in the face of a great deal of evidence to the contrary.

In subsequent postings, let’s take a look at some of the specific biases, how they affect our work, and how we can recognize and compensate for them.

Why We Are Biased in Our Development and Test Practices April 10, 2014

Posted by Peter Varhol in Software development, Strategy.
Tags: ,
add a comment

We all have biases. In general, despite the negative connotation of the word, that’s not a bad thing. Our past experiences cause us to view the world in specific ways, and incline us to certain decisions and actions over others. They are part of what make use individuals, and they assist us in our daily lives by letting us make many decisions with little or no conscious thought.

However, biases may also cause us to make many of those decisions poorly, which can hurt us in our lives, our jobs, and our relationships with others. In this series, I’m specifically looking at how biases influence the software development lifecycle – development, testing, agile processes, and other technical team-oriented processes.

Psychology researchers have attempted to categorize biases, and have come up with at least several dozen varieties. Some are overlapping, and some separate biases describe similar cognitive processes. But it is clear that how we think through a situation and make a decision is often influenced by our own experiences and biases.

Much of the foundation research comes from the work of psychologists Daniel Kahneman and Amos Tversky, and published in Kahneman’s Thinking, Fast and Slow. In this book, Kahneman describes a model of thinking comprised of two different systems. System 1 thinking is instinctive, automatic, and fast. It is also usually wrong, or at least gullible. Many biases come from our desire to apply this kind of thinking to more complex and varied situations.

System 2 thinking is slower, more deliberate, and more accurate. If we are being creative or solving a complex problem for the first time, we consciously engage our intellect to arrive at a decision. Using System 2 thinking can often reduce the possibility of bias. But this is hard work, and we generally don’t care to do it that often.

It’s interesting to note that Kahneman won a Nobel Prize in Economics for his research. He demonstrated that individuals and markets often don’t act rationally to maximize their utility. Often our biases come into play to cause us to make decisions that aren’t in our best interest.

In the case of software development and testing, our biases lead us into making flawed decisions on planning, design, implementation, and evaluation of our data. These factors likely play a significant role in our inability to successfully complete many software projects.