jump to navigation

Testing in the M2M World October 29, 2014

Posted by Peter Varhol in Software development.
Tags: ,
add a comment

We typically don’t think about all of the computers surrounding us. Our automobiles have at least a dozen processors, controlling acceleration, braking, engine operation and sensors, and telematics. Hospitals use computers in a wide variety of instruments, from EKG machines to X-ray devices to patient monitoring systems. Modern aircraft are overwhelmingly “fly by wire”, in that computers translate pilot instructions into digital decisions concerning setting engine speed and performance, and managing control surfaces. Mobile phones and tablets have some of the characteristics of traditional computers, but have different user interface interactions and often-unreliable connectivity.

We have had mobile and embedded computing devices for several decades. What makes today’s generation unique are two things – they often have user interfaces that enable users and operators to observe data and input instructions; and they are almost always interconnected. Further, the user interfaces are getting more complicated as these devices become still more powerful and more interconnected. These devices are comprised of non-standard hardware, using ASICs, FPGAs, or other custom designs.

Many traditional testers struggle with the different paradigms needed when testing applications that run on other than old-style computers. This is especially the case if the device is safety-critical; that is, if a failure can cause harm. We are trained to test to requirements, yet requirements are often incomplete, ambiguous, or make unstated assumptions concerning appropriate operation. Even if requirements are clear and complete, meeting those requires doesn’t necessarily guarantee a safe and high-quality product.

Further, virtually all of these devices are used outside of the normal and sanitized office environment. Whether outdoors, on roads, in hospital emergency rooms, or in the air at nine hundred kilometers an hour, embedded devices and software have to work in less-than-optimal conditions. We simply can’t test in all possible environments with all possible users, but with some effort we can intimately understand where our application might succeed or fail.

 

Start Small

In embedded and mobile projects, it is especially important to verify each software component as it is built, using a unit testing approach that ensures that the output corresponds to the input. However, unit testing by itself is insufficient. It typically does not look at the many ways that software running on a device can be used and abused. Testers are in a better position than developers to understand the risks inherent in the software, and device small testing tactics that exercise those risks.

That is why testers have to be intimately involved in early component testing. Developers attempt to confirm proper operation; testers are better to understand incorrect or malformed inputs, as well as user errors, and how to interpret incorrect outputs. Building unit tests that actually reflect what inputs will actually occur may in fact make a significant impact on the results. By catching poor error handling at this stage of the process, testers will actually make further testing, including GUI testing, easier and more reliable.

 

Build Up Gradually

Unit testing is a very useful first step in ensuring quality. However, when units are combined into higher level components, which comprise the application, those units may interact in unexpected ways. Testing is necessary at every level of integration in order to make sure that the application continues to work as expected.

Once a unit has passed testing, most developers are unconcerned about integration consequences. That’s where testers must take over. At each integration point, unit tests should be combined if necessary and executed again. Further, these tests should not just verify outputs for a given input, but also look at behaviors that may result in incorrect or inappropriate operation as the application continues to come together.

 

Attack the GUI

Testers have learned to test by verifying that requirements are met through GUI testing. However, testing and most especially mobile and embedded testing have to go well beyond that in order to better understand the application and its potential weaknesses.

Jon Hagar (Software Test Attacks to Break Mobile and Embedded Devices, 2013) and James Whittaker (How to Break Software, 2003) talk about attacking embedded software, and they are on the right track. You look at the risks that such software poses, and attack it based on those risks. If you are able to break it, you have exposed those risks as potential realities that need to be addressed before the software is released.

This means that test cases have to expand beyond requirements. That makes both testing and traceability more difficult than we are used to, but is a necessary part of creating an embedded application.

 

Always, Always Collect and Analyze Code Coverage Data

Most GUI testing on traditional desktop applications exercises about a third of the actual application code. In most mobile and embedded applications, it will likely be less, because more functionality is not accessible directly through the GUI. And that is not sufficient. It’s bad because it’s not clear what that code is doing, or why it is there, without testing it.

Much of that code is error-handling code, and embedded testers have to be able to generate the errors that this code is meant to handle. That means looking at malformed input, user error, unexpected events, and poor environmental conditions. These may not be part of the stated requirements, but are necessary in order to ensure the quality and proper operation of the software.

While one hundred percent code coverage is generally not achievable, project teams should decide on a code coverage goal and work to achieve that goal. Most such goals for complex systems are defined at around 70 or 80 percent of the code, depending on which type of code coverage is measured.

And code coverage makes a big difference early in the process. Whether in unit tests, or as software units are being integrated into larger components, code coverage gives testers a reality check on how much of the application is being tested. If it’s not enough, then attack test strategies must be formulated and execute to produce a better result.

GUI testing complements that by testing software features. Together, the two approaches provide a means of assessing both adherence to requirements and a deeper understanding of underlying testing effectiveness and software quality.

This approach will not result in perfect software and applications for mobile and embedded devices. However, it will provide testers with information on meeting requirements, extent of testing, and potential weaknesses.

The important thing is that testers recognize that mobile and embedded testing requires more than testing to requirements, and more than GUI testing. It means understanding the risks that the devices and their software pose to safety, mission, and overall successful operation of the application and device in general, and devising testing strategies that successfully address those risks.

Old Versus New October 5, 2014

Posted by Peter Varhol in Technology and Culture, Uncategorized.
Tags:
add a comment

Amtrak, you suck.

I am attempting to perhaps book a train trip to New York, from Boston, as opposed to flying. Amtrak is mostly inconvenient and time-consuming unless you are in the city center, or in a relevant close suburb. In Boston, that means either South Station, or Westwood. Maybe Worcester if you’re not picky about getting the Acela. Either way, it’s 3 hours 20 minutes to get to Penn Station on the Acela from Westwood, or over four hours on the regular run. Plus at least another hour for me to drive to Westwood, or South Station, or Worcester. And that’s non-rush hour. It might work, under the best circumstances, but it mostly doesn’t.

But. When I’m trying my level best to consider the train as an alternative, the Amtrak website is down. On a Sunday. All of the day, so far.

I’m sorry, Amtrak. My airline has never done this. If you are trying to get the American public to consider you as an alternative to air travel, even in the Northeast corridor, this is the absolute wrong way to do so. Regrettably, I don’t even think you realize that there is a problem being down “for maintenance” for just about all of a day (so far, it may turn out to be even more).

You get subsidies from the US government. You don’t deserve them.  It’s as simple as that.

Amtrak, if you at all cared about making rail a viable travel option, especially in the Northeast where distances are feasible, here’s what you would do. You would provide (reliable, do I even have to say that?) transportation to the stations from outlying locations. You would make your schedules available elsewhere.

And you would not let your website fail like this.

I have no confidence that you even care about doing any of this. You get your government subsidies to cover your operational losses, and you are fine with that.

But you are dead wrong. I will never consider you as a travel option again, and I will encourage everyone else not to do so. This is entirely on you.

He’s Not Heavy, He’s My Fitbit September 24, 2014

Posted by Peter Varhol in Technology and Culture.
add a comment

I continue my exploration with personal fitness technology, and today was an interesting day. Today started at about 4:50AM, when I prepared to go out walking/running with my Fitbit (I walk perhaps 2.5 miles, run 1.5 right now). Except that I forgot my Fitbit. In my defense, I was attempting to communicate with my sister and her daughter, who I had assisted in making their first trip to Europe overnight. At pre-dawn here on the US East Coast, I was ecstatic that they had reached Munich, their gateway airport, shortly before that time (final destination – Vienna).

So I was ten minutes out when I realized I forgot my recording device. I am of the ilk that it doesn’t count unless it was recorded, but it didn’t make sense to go back, so I didn’t. I recorded 41 minutes by my watch, but it’s not in my record, sigh.

The Fitbit talks to me, you know. Its UI is limited, but I always get a message when I put on my exercise clothes early in the morning. My favorite remains “Walk me, Peter”.

So my record is less than what I would prefer. Still. When I got home this evening, my new Nike+ Smartwatch with GPS was waiting for me. I plugged it in, and it’s telling me “Are we running today?” Well.

I’m curious about the encouraging messages. They seem friendly, but I don’t need my technology to be friendly. What I need is feedback on my progress. If I achieve some goal, it seems that my self-competitive nature makes me want to better that goal. But the personalization is intriguing.

So now I have two devices to carry with me tomorrow. Should I stay with the Fitbit, use the GPS watch, or both? My inclination is both. He’s not heavy; he’s my Fitbit.

More on the NFL, Because I Was There September 16, 2014

Posted by Peter Varhol in Uncategorized.
add a comment

Mr. Goodell, I had the opportunity to hear you speak several years ago, as the keynote at a large university graduation.  You spoke passionately about your father, representing New York in Congress.  He spoke out against the Vietnam War, an unpopular position at the time, and sacrificed his political career for his stand.

Contrast that against your lack of speaking out today.  You shame  your father’s memory.  You are apparently doing so to protect your $40 million a year paycheck.  You are clearly bought and paid for, and don’t care about the ethics of what you are doing.  What does your father think of you today?

Off the Rails on Social Commentary With the NFL September 14, 2014

Posted by Peter Varhol in Uncategorized.
Tags: ,
add a comment

On this beautiful fall Sunday afternoon, I simply can’t watch or otherwise follow US professional football any more. The reason, of course, is domestic violence, and the billionaire team owners’ response to it.

I am no more than a casual fan, and am certain that I am not in the league’s target demographic. I don’t buy grossly overpriced apparel, and it’s been a long time since I’ve actually attended an overpriced game. So I’m sure that no one who matters cares at all what I think here. Still, unlike the owners, I feel compelled to take a stand.

I am especially appalled by commentators on TV and on various fan websites. As near as I can tell, these fall into one of three broad categories.

First, I want my team to win, and I don’t care if they employ the Mean Machine to do so. Grow up, it’s a game, dammit. You don’t live or die over who wins and loses.

Second, he made a mistake, and he should be entitled to due process or a second chance. Um, no. There is no entitlement here. Where did that idea even come from?  Second chances are not automatic get-out-of-jail free cards. You earn your second chances, through knowledge that you did wrong, honest and heartfelt contrition, a determination to set things right, and a track record of enlightened understanding and a better life.  I don’t see much of that going on.

Third, the perpetrators are Bad Men, and should be treated accordingly. Perhaps, but there are a lot of highly paid and presumably competent people involved in evaluating their behavior and effects. I fear they are driven mostly by the profit motive, but I hope there is some semblance of a conscience in most of them.

But my special disgust is reserved for the National Football League bureaucracy. Simply put, those highly paid denizens are either lying or incompetent.  There is certainly no conscience here.

But the bottom line is that our multimillion dollar athletes, those that own their contracts, and those that manage them have let this get completely out of control over an unconscionable need to win above all else. If sports are a microcosm of society, this is society at its very worst.

No matter what happens, I don’t think I can go back and be a fan of any sort again. Goodbye, NFL.

On This Eve of 9/11 September 11, 2014

Posted by Peter Varhol in Uncategorized.
1 comment so far

On this eve of 9/11, I once again remember Graham Berkeley and Myra Aronson, fellow Compuware employees on the planes that hit the Towers.  Please join me.  Thank you.

Reuters, Get Your Act Together Now September 6, 2014

Posted by Peter Varhol in Uncategorized.
Tags: ,
add a comment

I like the Reuters website as an alternative new source. But for the last several weeks, my visits to Reuters have been redirected to various phishing sites, to install Java or Flash. Wrong. I am up to date on all. They are simply getting hijacked. I would imagine this is happening to others, too.

I’ve sent multiple emails to the site, asking that they address this. They have not. I don’t understand why not, but I can’t continue visiting this site under these circumstances.

Goodbye, Reuters. I imagine many others will react the same.

Train Travel Is Not All it’s Cracked Up to Be September 4, 2014

Posted by Peter Varhol in Technology and Culture.
Tags:
add a comment

I cannot tell you how much I enjoy visiting Europe. I have had the opportunity to visit around 15 times in the last five years, and every single trip has been great. I’ve been to Zurich (five times) Brussels, Prague (twice), Bruges, London, Bilbao, Vienna, Sofia, Berlin (twice) and Tallinn, and have always had a really good experience. I hope to continue going back as long as I can function.

I am currently assisting my sister and her daughter in visiting Europe for their first and perhaps only time. They have chosen to go to Vienna, with day trips to Prague, Bratislava, and Budapest. The guided tour day trip routine isn’t my cup of tea, but as this may be their only visit, I applaud their initiative.

I will be in the Berlin area at the same time, speaking at Mobile App Europe, and was seeking to perhaps visit them in Vienna on their excursion. I ultimately concluded that it was both too expensive and logistically problematic, and declined to do so.

By the same token, I cannot tell you how many people have told me that I should just pop down on the train from Berlin to Vienna for the day. When I explain that the distance is in excess of 500 miles, and that excluding getting to and from train stations, the trip was still twelve hours one way, they disbelieve me, rather than alter their own uninformed beliefs.

Yes, train travel in most of the US is pretty poor. But Americans have been brainwashed (I think by our own rail advocates) to believe that train travel in Europe is easy, fast, and seamless. Well, one out of three ain’t bad. It is pretty easy. It’s not particularly seamless, and it’s not especially fast. For some reason Americans believe that distances are shorter than they are, and that bullet trains are everywhere. We think that every city in Europe is two hours by train, and four hours by plane.

Wrong on both counts. I flew into Brussels early this year, for a conference in Bruges. I shared a cab from the airport with two others (90 Euros plus tip), which took just over an hour. Those who took the train had to haul their bags onto the train (granted, the station was conveniently located in the basement of the airport), but change in Brussels Nord, and make several stops, for a journey of about two and a half hours. And it was about 45 Euros a person. Who did better here?

Many Americans suffer from the delusion that just about anything is better anywhere else in the world than where we are. Much of that is driven by constituencies with private agendas, and the rest is driven by our own belief that our systems are inadequate and underfunded. If more Americans had the opportunity to understand how things worked in other parts of the world, we would better appreciate what we had.

Follow

Get every new post delivered to your Inbox.

Join 402 other followers