jump to navigation

Testing in the M2M World October 29, 2014

Posted by Peter Varhol in Software development.
Tags: ,
add a comment

We typically don’t think about all of the computers surrounding us. Our automobiles have at least a dozen processors, controlling acceleration, braking, engine operation and sensors, and telematics. Hospitals use computers in a wide variety of instruments, from EKG machines to X-ray devices to patient monitoring systems. Modern aircraft are overwhelmingly “fly by wire”, in that computers translate pilot instructions into digital decisions concerning setting engine speed and performance, and managing control surfaces. Mobile phones and tablets have some of the characteristics of traditional computers, but have different user interface interactions and often-unreliable connectivity.

We have had mobile and embedded computing devices for several decades. What makes today’s generation unique are two things – they often have user interfaces that enable users and operators to observe data and input instructions; and they are almost always interconnected. Further, the user interfaces are getting more complicated as these devices become still more powerful and more interconnected. These devices are comprised of non-standard hardware, using ASICs, FPGAs, or other custom designs.

Many traditional testers struggle with the different paradigms needed when testing applications that run on other than old-style computers. This is especially the case if the device is safety-critical; that is, if a failure can cause harm. We are trained to test to requirements, yet requirements are often incomplete, ambiguous, or make unstated assumptions concerning appropriate operation. Even if requirements are clear and complete, meeting those requires doesn’t necessarily guarantee a safe and high-quality product.

Further, virtually all of these devices are used outside of the normal and sanitized office environment. Whether outdoors, on roads, in hospital emergency rooms, or in the air at nine hundred kilometers an hour, embedded devices and software have to work in less-than-optimal conditions. We simply can’t test in all possible environments with all possible users, but with some effort we can intimately understand where our application might succeed or fail.

 

Start Small

In embedded and mobile projects, it is especially important to verify each software component as it is built, using a unit testing approach that ensures that the output corresponds to the input. However, unit testing by itself is insufficient. It typically does not look at the many ways that software running on a device can be used and abused. Testers are in a better position than developers to understand the risks inherent in the software, and device small testing tactics that exercise those risks.

That is why testers have to be intimately involved in early component testing. Developers attempt to confirm proper operation; testers are better to understand incorrect or malformed inputs, as well as user errors, and how to interpret incorrect outputs. Building unit tests that actually reflect what inputs will actually occur may in fact make a significant impact on the results. By catching poor error handling at this stage of the process, testers will actually make further testing, including GUI testing, easier and more reliable.

 

Build Up Gradually

Unit testing is a very useful first step in ensuring quality. However, when units are combined into higher level components, which comprise the application, those units may interact in unexpected ways. Testing is necessary at every level of integration in order to make sure that the application continues to work as expected.

Once a unit has passed testing, most developers are unconcerned about integration consequences. That’s where testers must take over. At each integration point, unit tests should be combined if necessary and executed again. Further, these tests should not just verify outputs for a given input, but also look at behaviors that may result in incorrect or inappropriate operation as the application continues to come together.

 

Attack the GUI

Testers have learned to test by verifying that requirements are met through GUI testing. However, testing and most especially mobile and embedded testing have to go well beyond that in order to better understand the application and its potential weaknesses.

Jon Hagar (Software Test Attacks to Break Mobile and Embedded Devices, 2013) and James Whittaker (How to Break Software, 2003) talk about attacking embedded software, and they are on the right track. You look at the risks that such software poses, and attack it based on those risks. If you are able to break it, you have exposed those risks as potential realities that need to be addressed before the software is released.

This means that test cases have to expand beyond requirements. That makes both testing and traceability more difficult than we are used to, but is a necessary part of creating an embedded application.

 

Always, Always Collect and Analyze Code Coverage Data

Most GUI testing on traditional desktop applications exercises about a third of the actual application code. In most mobile and embedded applications, it will likely be less, because more functionality is not accessible directly through the GUI. And that is not sufficient. It’s bad because it’s not clear what that code is doing, or why it is there, without testing it.

Much of that code is error-handling code, and embedded testers have to be able to generate the errors that this code is meant to handle. That means looking at malformed input, user error, unexpected events, and poor environmental conditions. These may not be part of the stated requirements, but are necessary in order to ensure the quality and proper operation of the software.

While one hundred percent code coverage is generally not achievable, project teams should decide on a code coverage goal and work to achieve that goal. Most such goals for complex systems are defined at around 70 or 80 percent of the code, depending on which type of code coverage is measured.

And code coverage makes a big difference early in the process. Whether in unit tests, or as software units are being integrated into larger components, code coverage gives testers a reality check on how much of the application is being tested. If it’s not enough, then attack test strategies must be formulated and execute to produce a better result.

GUI testing complements that by testing software features. Together, the two approaches provide a means of assessing both adherence to requirements and a deeper understanding of underlying testing effectiveness and software quality.

This approach will not result in perfect software and applications for mobile and embedded devices. However, it will provide testers with information on meeting requirements, extent of testing, and potential weaknesses.

The important thing is that testers recognize that mobile and embedded testing requires more than testing to requirements, and more than GUI testing. It means understanding the risks that the devices and their software pose to safety, mission, and overall successful operation of the application and device in general, and devising testing strategies that successfully address those risks.

What is the Deal with Self-Driving Cars? June 23, 2014

Posted by Peter Varhol in Software development, Technology and Culture.
Tags: ,
add a comment

Google, the media, and other interested parties are portraying self-driving cars as a panacea for drivers, traffic congestion, accidents, and other undesirable driving outcomes. I simply don’t get it, on multiple levels. I like the concept, but can’t connect it to any reasonable reality anytime in the future.

I’ve suspected that there would be issues with self-driving cars since they became a popular meme over the past year. At one level, there is the question of how you would test the technology. In normal system testing, you attempt to run tests that simulate actual use. But there are far too many possible scenarios for self-driving cars to reasonably test. Under other circumstances, it may be possible to test the most likely cases, but on a safety-critical system like a car, that’s simply not possible.

I’m reminded of my skepticism by this article on the utility of aircraft autopilot systems and their role in the operation and in some cases mis-operation of planes. One conclusion seems to be that autopilots actually make flying more complex, rather than simpler. That counterintuitive conclusion is based on the idea that the assumptions made by the autopilot are unexpected by the operators.

As a software guy, I’m okay with the idea that assumptions made by software can take people by surprise on occasion. It’s a difficult problem even for safety-critical systems, where people can die if the software makes an incorrect assumption. You can argue, probably successfully, that pilots shouldn’t be surprised by whatever a plane under their command does.

Drivers, not so much. As we look at aircraft autopilots, it is reasonable to draw a parallel between commercial aircraft and automobiles. Now, granted, aircraft operate in three dimensions. But automobiles have a greater range of operating options, in terms of speed, traffic, road types, road conditions, and so on. Commercial aircraft are already under positive control from the ground.

It’s not clear who will control driverless automobiles. It’s certainly unlikely that drivers are as attentive as pilots, yet will become at least as confused at times as they change where they want to go, and how they want to get there. And they won’t be observing the driving process any near as attentively as (I hope) pilots do.

Sigh. I’m not a Luddite. I’m excited about technology in general, and am an early adopter of many technologies (and, to be honest, a not-so-early adopter of others). But I simply don’t see self-driving automobiles taking off (pun intended) anytime in my lifetime.

Mindsets and Software Testing May 18, 2014

Posted by Peter Varhol in Software development, Strategy.
Tags: ,
1 comment so far

All of us approach life with certain attitudes and expectations. These attitudes and expectations can affect how we perform and what we learn, both now and in the future.

According to researcher Carol Dweck, there are two fundamental mindsets – fixed and growth. A person with a fixed mindset feels the need to justify their abilities at every possible opportunity. If they succeed, it reaffirms their status as an intelligent and capable person. If they fail, it is a reflection upon their abilities, and there is nothing to be learned from it.

A person with a growth mindset recognizes that chance, opportunity, and skill all play a role in any activity in which they engage. Success can be due to a number of factors, of which our intelligence and abilities play only a part.   More important, we can improve our abilities through failure, by not taking it personally and by delving into the lessons learned.

It’s important to understand that the idea of mindset is a continuum, so that few if any of us are entirely one or the other. And in some circumstances we may be more one than in others. I can personally attest to this through my own experiences.

This has a couple of implications to software development and testing. First, it means that we will almost certainly make mistakes. But how we respond to those mistakes is key to our futures. We can be defensive and protective of our self-perception, or we can learn and move on.

Second, and perhaps more important, is that failing at a project, creating buggy code, or failing to find bugs isn’t a reflection on our intelligence or abilities. At the very least, it’s not something that can’t be corrected. If we are willing to grow from it, we might recognize that our work is a marathon, rather than a sprint.

It also has implications to technical careers in general. I’ve failed more times that I would like to count. I’ve also succeeded many times too. With a fixed mindset, I’m not sure where that leads me. Charitably, I’m almost certainly a screw-up. With a growth mindset, I’m right where I want to be. I’ll leave the answer to that question as an exercise to the reader.

Cognitive Bias and Regression to the Mean April 29, 2014

Posted by Peter Varhol in Software development.
Tags: , ,
add a comment

We prefer to assign causality to events in our own lives, and in the world in general. If something positive happens, we tend to credit our intelligence, or dedication, or some other quality. If negative, we often blame others, or perhaps blame our own failings. Every day when the stock market closes, we read about how stocks have gone up or down for some perfectly understandable reason.

Bull. Most things in our lives and in the world don’t happen for a reason, or at least any reason we can readily identify. Our good fortune may be only peripherally related to our intelligence or abilities, and our bad fortune may simply arise from being in the wrong place at the wrong time.

Regression to the mean is simply one example of our need for causality, and how it results in bias. If we perform exceptionally well, we come to believe our own press releases, and behave as though we are high achievers. We might well be, but achievement is a slippery thing; it might disappear in a heartbeat.

Regression to the mean is a statistical concept. It simply notes that any time you get an exceptional result, it is unusual. Subsequent results are more likely to be closer to the average. It’s a concept often found in the natural sciences. For example, I am taller than either of my parents, so it is likely that my children (if I had any) would be shorter than me, since I am taller than many of my direct ancestors.

Applied to our lives, regression to the mean refers to the fact that what we do is a combination of skill and luck. We have little idea how much is skill, and how much luck. When we do exceptionally well at a task, we tend to attribute that to skill. When we do poorly, we often blame bad luck. Instead, exceptional performances are random (and rare) chance.

You can certainly argue that such a statistical concept doesn’t really apply to individual efforts, but I think the general principle holds. Sometimes we simply do better than other times, and it’s not clear that it reflects skill any more than (good or bad) luck.

Applied to software development and testing, regression to the mean gives us preconceived notions of the performance of the software based on who works on it. It makes us believe certain things about software based on the perceived superiority or inferiority of the team members based on our experiences.

The Role of Heuristics in Bias April 24, 2014

Posted by Peter Varhol in Software development.
Tags: , ,
add a comment

A heuristic is what we often refer to as a “rule of thumb”. We’ve experienced a particular situation on several occasions, and have come up with a step-by-step process for dealing with it. It’s purely System 1 thinking in action, as we assess the situation and blindly follow rules that have worked for us in the past.

And heuristics are great. They help us make decisions fast in situations that we’ve experienced in the past. But when the situation only appears similar, but is really different, applying our heuristic can have a very bad effect, if it’s not right.

Here’s a real life example. Years ago, I took flying lessons and obtained my pilot’s license. One of the lessons involved going “under the hood”. The hood is a plastic device that goes over your head (see image). When the hood is down, you can’t see anything. When the hood is raised, you can see the instrument panel, but not outside of the plane.

hood

While the hood was down, the instructor pilot in the right seat put the plane into an unusual situation. That might be a bank, or a stall, or something that was unsustainable. When he raised the hood, I was required to use the instrument panel to analyze and diagnose the situation, and recover from it.

After several of these situations, I had developed a heuristic. I looked first at the turn and bank indicator; if we were turning or banking, I would get us back on course in straight flight. Then I would look at the airspeed indicator. If we were going too slow, I could lower the nose or advance power to get us back to a cruise speed.

This heuristic worked great, and four or five times I was able to recover the aircraft exceptionally quickly. I was quite proud of myself.

But my instructor figured out what I was doing, and the next time I applied my heuristic, it seemed to work. But I was fighting the controls! It wasn’t straight and level flight. I started scanning other instruments, and discovered that we were losing over a thousand feet a minute.

At that point, my heuristic had failed. But I wasn’t able to go back and analyze the situation. My mind froze, and if it weren’t for the instructor pilot, we may well have crashed.

The lesson is that when your heuristic doesn’t work, it may be worse than starting over at the beginning. You may simply not be able to.

Applying Cognitive Bias to Software Development and Testing April 21, 2014

Posted by Peter Varhol in Software development.
Tags: ,
add a comment

Through his psychology research, Daniel Kahneman (and his collaborator Amos Tversky) demonstrated that we are not rational investors. We make irrational decisions all the time, decisions that most definitely don’t optimize our expected utility. He proved this well enough that he was awarded a Nobel Prize in Economics.

Beyond economics, we exhibit the same behavior in other aspects of our lives, including our professional lives. Let’s take software testing as an example. We may have preconceived notions of how buggy a particular application is, and that will likely affect how we test it. We may have gotten that notion from previous experience with the development team, or from an initial use of a previous version of the software.

As a result of those preconceived notions, or biases, we are likely to plan and execute our work, and evaluate the results, differently than if they didn’t exist. If our prior experiences with the team or the software were negative, we may be overly harsh in our assessment of the software and its perceived flaws. If our experiences are positive, we may be willing to give questionable characteristics a free pass.

Lest it sounds like this is a conscious decision on our part, let me say right now that it’s almost always not so. It never occurs to us to think that we are biased. If we think of it at all, we believe that the bias is a good thing, because it puts us on alert for possible problems, or it gives us a warm fuzzy of the quality or fitness of the application.

Bias can be a good shortcut to the correct or optimal decision. More often, it is a way of analyzing a situation poorly and making an incorrect or less-than-ideal decision. Even if it might result in a good outcome, it’s incumbent of each of us to realize when we are being influenced by our own beliefs, and to question those beliefs.

We tend to think of software development and testing as highly analytical and scientific endeavors, but the fact is that they are both highly subjective and social. We work in close-knit teams, and the decisions are highly situational based on the exact circumstances of the problem. We tend to overestimate our individual and group abilities, and underestimate the complexity of the problems to be solved.

Further, we tend not to learn relevant lessons from past experiences, instead remaining overly optimistic, often in the face of a great deal of evidence to the contrary.

In subsequent postings, let’s take a look at some of the specific biases, how they affect our work, and how we can recognize and compensate for them.

How Do We Fix Testing? April 17, 2014

Posted by Peter Varhol in Software development, Software tools.
Tags:
1 comment so far

Here is a presentation abstract I hope to get accepted at a conference in the near future:

Perhaps in no other professional field is the dichotomy between theory and practice more starkly different than in the realm of software testing. Researchers and thought leaders claim that testing requires a high level of cognitive and interpersonal skills, in order to make judgments about the ability of software to fulfill its operational goals. In their minds, testing is about assessing and communicating the risks involved in deploying software in a specific state.

However, in many organizations, testing remains a necessary evil, and a cost to drive down as much as possible. Testing is merely a measure of conformance to requirements, without regard to the quality of requirements or how conformance is measured. This is certainly an important measure, but tells an incomplete story about the value of software in support of our business goals.

We as testers often help to perpetuate the status quo. Although in many cases we realize we can add far more value than we do, we continue to perform testing in a manner that reduces our value in the software development process.

This presentation looks at the state of the art as well of the state of common practice, and attempts to provide a rationale and roadmap whereby the practice of testing can be made more exciting and stimulating to the testing professional, as well as more valuable to the product and the organization.

Why Do Biases Exist in the First Place? April 17, 2014

Posted by Peter Varhol in Software development, Strategy.
add a comment

If we are biased in our analysis of situations and our decisions in those situations, something must have precipitated that bias. As I mentioned in my last post, it is often because we use Kahneman’s System 1, or “fast” thinking, when we should really use the more deliberate System 2 thinking.

But, of course, System 2 thinking requires conscious engagement, which we are reluctant to do for situations that we think we’ve seen before. It simply requires too much effort, and we think we can comprehend and decide based on other experiences. It should come as no surprise that our cognitive processes favor the simple over the complex. And even when we consciously engage System 2 thinking, we may “overthink” a situation and still make a poor decision.

Bias often occurs when we let our preconceptions influence our decisions, which is the realm of System 1 thought. That’s not to say that System 2 thinking can’t also be biased, but the more we think about things, the better the chance that we make a rational decision. It’s easier to mischaracterize a situation if we don’t think deeply about it first.

As for System 2 thinking, we simply can’t make all, or even very many, of our decisions by engaging our intellect. There isn’t enough time, and it takes too much energy. And even if we could, we may overanalyze situations and make errors in that way.

There is also another, more insidious reason why we exhibit biases in analyzing situations and making decisions. That is that we have yet another bias – we believe that we make better decisions than those around us. In other words, we are biased that we believe we have fewer biases than the next person!

Are we stuck with our biases, forever consigned to not make the best decisions in our lives? Well, we won’t eliminate bias from our lives, but by understanding how and why it happens, we can reduce biased decisions, and make better decisions in general. We have to understand how we make decisions (gut choices are usually biased), and recognize situations where we have made poor decisions in the past.

It’s asking a lot of a person to acknowledge poor decisions, and to remember those circumstances in the future. But the starting point to doing so is to understand the origin of biases.

Follow

Get every new post delivered to your Inbox.

Join 410 other followers