Testing in the M2M World

We typically don’t think about all of the computers surrounding us. Our automobiles have at least a dozen processors, controlling acceleration, braking, engine operation and sensors, and telematics. Hospitals use computers in a wide variety of instruments, from EKG machines to X-ray devices to patient monitoring systems. Modern aircraft are overwhelmingly “fly by wire”, in that computers translate pilot instructions into digital decisions concerning setting engine speed and performance, and managing control surfaces. Mobile phones and tablets have some of the characteristics of traditional computers, but have different user interface interactions and often-unreliable connectivity.

We have had mobile and embedded computing devices for several decades. What makes today’s generation unique are two things – they often have user interfaces that enable users and operators to observe data and input instructions; and they are almost always interconnected. Further, the user interfaces are getting more complicated as these devices become still more powerful and more interconnected. These devices are comprised of non-standard hardware, using ASICs, FPGAs, or other custom designs.

Many traditional testers struggle with the different paradigms needed when testing applications that run on other than old-style computers. This is especially the case if the device is safety-critical; that is, if a failure can cause harm. We are trained to test to requirements, yet requirements are often incomplete, ambiguous, or make unstated assumptions concerning appropriate operation. Even if requirements are clear and complete, meeting those requires doesn’t necessarily guarantee a safe and high-quality product.

Further, virtually all of these devices are used outside of the normal and sanitized office environment. Whether outdoors, on roads, in hospital emergency rooms, or in the air at nine hundred kilometers an hour, embedded devices and software have to work in less-than-optimal conditions. We simply can’t test in all possible environments with all possible users, but with some effort we can intimately understand where our application might succeed or fail.

 

Start Small

In embedded and mobile projects, it is especially important to verify each software component as it is built, using a unit testing approach that ensures that the output corresponds to the input. However, unit testing by itself is insufficient. It typically does not look at the many ways that software running on a device can be used and abused. Testers are in a better position than developers to understand the risks inherent in the software, and device small testing tactics that exercise those risks.

That is why testers have to be intimately involved in early component testing. Developers attempt to confirm proper operation; testers are better to understand incorrect or malformed inputs, as well as user errors, and how to interpret incorrect outputs. Building unit tests that actually reflect what inputs will actually occur may in fact make a significant impact on the results. By catching poor error handling at this stage of the process, testers will actually make further testing, including GUI testing, easier and more reliable.

 

Build Up Gradually

Unit testing is a very useful first step in ensuring quality. However, when units are combined into higher level components, which comprise the application, those units may interact in unexpected ways. Testing is necessary at every level of integration in order to make sure that the application continues to work as expected.

Once a unit has passed testing, most developers are unconcerned about integration consequences. That’s where testers must take over. At each integration point, unit tests should be combined if necessary and executed again. Further, these tests should not just verify outputs for a given input, but also look at behaviors that may result in incorrect or inappropriate operation as the application continues to come together.

 

Attack the GUI

Testers have learned to test by verifying that requirements are met through GUI testing. However, testing and most especially mobile and embedded testing have to go well beyond that in order to better understand the application and its potential weaknesses.

Jon Hagar (Software Test Attacks to Break Mobile and Embedded Devices, 2013) and James Whittaker (How to Break Software, 2003) talk about attacking embedded software, and they are on the right track. You look at the risks that such software poses, and attack it based on those risks. If you are able to break it, you have exposed those risks as potential realities that need to be addressed before the software is released.

This means that test cases have to expand beyond requirements. That makes both testing and traceability more difficult than we are used to, but is a necessary part of creating an embedded application.

 

Always, Always Collect and Analyze Code Coverage Data

Most GUI testing on traditional desktop applications exercises about a third of the actual application code. In most mobile and embedded applications, it will likely be less, because more functionality is not accessible directly through the GUI. And that is not sufficient. It’s bad because it’s not clear what that code is doing, or why it is there, without testing it.

Much of that code is error-handling code, and embedded testers have to be able to generate the errors that this code is meant to handle. That means looking at malformed input, user error, unexpected events, and poor environmental conditions. These may not be part of the stated requirements, but are necessary in order to ensure the quality and proper operation of the software.

While one hundred percent code coverage is generally not achievable, project teams should decide on a code coverage goal and work to achieve that goal. Most such goals for complex systems are defined at around 70 or 80 percent of the code, depending on which type of code coverage is measured.

And code coverage makes a big difference early in the process. Whether in unit tests, or as software units are being integrated into larger components, code coverage gives testers a reality check on how much of the application is being tested. If it’s not enough, then attack test strategies must be formulated and execute to produce a better result.

GUI testing complements that by testing software features. Together, the two approaches provide a means of assessing both adherence to requirements and a deeper understanding of underlying testing effectiveness and software quality.

This approach will not result in perfect software and applications for mobile and embedded devices. However, it will provide testers with information on meeting requirements, extent of testing, and potential weaknesses.

The important thing is that testers recognize that mobile and embedded testing requires more than testing to requirements, and more than GUI testing. It means understanding the risks that the devices and their software pose to safety, mission, and overall successful operation of the application and device in general, and devising testing strategies that successfully address those risks.