The two big traps of code coverage

Code coverage is important, and improving coverage is a worthy goal. But simply chasing the percentage is not nearly so valuable as writing stable, maintainable, meaningful tests.


By Arthur Hicken, Parasoft                       Download PDF version of this article


Measurement of code coverage is one of those things that always catches my attention. On the one hand, I often find that organizations do not necessarily know how much code they are covering during testing – which is really surprising! At the other end of the coverage spectrum, there are organizations for whom the number is so important that the quality and efficacy of the tests has become mostly irrelevant. They mindlessly chase the 100% dragon and believe that if they have that number the software is good, or even the best that it can be. This is at least as dangerous as not knowing what you have tested, in fact perhaps more so since it can give you a false sense of security.

Code coverage can be a good and interesting number to assess your software quality, but it is important to remember that it is a means, rather than an end. We do not want coverage for coverage’s sake, we want coverage because it is supposed to indicate that we have done a good job testing the software. If the tests themselves are not meaningful, then having more of them certainly does not indicate better software. The important goal is to make sure every piece of code is tested, not just executed. Failing enough time and money to fully test everything, at least make sure that everything important is tested.

What this means is that while low coverage means we are probably not testing enough, high coverage by itself does not necessarily correlate to high quality – the picture is more complicated than that. Obviously, having a happy medium where you have “enough” coverage to be comfortable about releasing the software with a good, stable, maintainable test suite that has “just enough tests” would be perfect. But still, these coverage traps are common.

Parasoft DTP example dashboard

The first trap is the “we don’t know our coverage” trap. This seems unreasonable to me – coverage tools are cheap and plentiful. A friend of mine suggests that organizations know their coverage number is not good, so developers and testers are loath to expose the poor coverage to management. I would hope this is not the usual case.

One real issue that teams encounter when trying to measure coverage is that the system is too complicated. When you build an application out of pieces on top of pieces on top of pieces, just knowing where to put the coverage counters can be a daunting task. I would suggest that if it is actually difficult to measure the coverage in your application, you should think twice about the architecture.

A second way to fall into this trap happens with organizations that may have a lot of testing, but no real coverage number because they have not found a proper way to aggregate the numbers from different test runs. If you are doing manual testing, functional testing, unit testing, and end-to-end testing, you cannot simply add the numbers up. Even if they are each achieving 25% coverage it is unlikely that it is 100% when combined. In fact, it is more likely to be closer to the 25% than to the 100% when you look into it.

As it turns out, there is in fact a way to measure and add coverage together in a meaningful fashion. At Parasoft, we leverage the vast amount of granular data captured by our reports and analytics tool, Parasoft DTP, which we can use in this context to provide a comprehensive, aggregated view of code coverage. Our application monitors are able to gather coverage data from the application directly while it is being tested and then send that information to Parasoft DTP, which aggregates coverage data across all testing practices, as well as across test teams and test runs. If that sounds like a pretty significant amount of information, you are right! DTP provides an interactive dashboard to help you navigate this data and make decisions about where to focus testing efforts.

If multiple tests have covered the same code, it will not be overcounted, while untested parts of the code are quick and easy to see. This shows you which parts of the application have been well tested and which ones have not.  So, no more excuses for not measuring coverage. This leads us to the next trap – the “coverage is everything” perspective. Once teams are able to measure coverage, it is not uncommon for managers to say “let’s increase that number.” Eventually the number itself becomes more important than the testing. Therein lies the problem – mindless coverage is the same as mindless music. The coverage needs to reflect real, meaningful use of the code, otherwise it is just noise.

And speaking of noise… the cost of coverage goes up as coverage increases. Remember that you not only need to create tests, but you have to maintain them going forward. If you are not planning on reusing and maintaining a test, you should probably not waste time creating it in the first place. As the test suite gets larger, the amount of noise increases in unexpected ways. Twice as many tests may mean two or even three times as much noise. The meaningless tests end up creating more noise than good tests because they have no real context, but have to be dealt with each time the tests are executed. Talk about technical debt! Useless tests are a real danger.

Now, in certain industries, safety-critical industries for example, the 100% coverage metric is a requirement. But even in that case, it is all too easy to treat any execution of a line of code as a meaningful test, which is simply not true. I have two basic questions I ask to determine if a test is a good test.  What does it mean when the test fails? What does it mean when the test passes?

Ideally, when a test fails, we know something about what went wrong, and if the test is really good, it will point us in the right direction to fix it. All too often when a test fails, no one knows why, no one can reproduce it, and the test is ignored. Conversely, when a test passes we should be able to know what was tested – it should mean that a particular feature or piece of functionality is working properly.

If you cannot answer one of those questions, you probably have a problem with your test. If you cannot answer either of them, the test is probably more trouble than it is worth.

The way out of this trap is first to understand that the coverage percentage itself is not the goal. The real goal is to create useful meaningful tests. This of course takes time. In simple code, writing unit tests is simple, but in complex real-world applications it means writing stubs and mocks and using frameworks. This can take quite a bit of time and if you are not doing it all the time, it is easy to forget the nuances of the APIs involved. Even if you are serious about testing, the time it takes to create a really good test can be more than you expect.

At Parasoft we have an answer for this: the Unit Test Assistant inside the Java development testing tool (Parasoft Jtest) takes on the tedious tasks of getting mocks and stubs right. It can also help expand existing tests in a useful way to increase coverage – helping you create good unit tests as well as make recommendations to improve test coverage and test quality as well.


Related


Give Your Product a Voice with Alexa

Join us for a deep dive into the system architecture for voice-enabled products with Alexa Built-In. Device makers can use the Alexa Voice Service (AVS) to add conversational AI to a variety of produc...

Securing the smart and connected home

With the Internet of Things and Smart Home technologies, more and more devices are becoming connected and therefore can potentially become entry points for attackers to break into the system to steal,...

Accurate and fast power integrity measurements

Increasing demands on power distribution networks have resulted in smaller DC rails, as well as a proliferation of rails that ensure clean power reaches the pins of integrated circuits. Measuring r...

 

Perfect Motion Control For the Networked World

We live in a physical world where everything is connected. Trinamic transforms digital information into physical motion with accessible, flexible, and easy to use toolkits putting the world’s be...


New High-Performance Serial NAND: A Better High-Density Storage Option for Automotive Display

The automotive requirements: speed, reliability and compatibility. Winbond's high-performance serial NAND Flash technology offers both cost and performance advantages over the SPI NOR Flash typica...


President Tung-Yi talks about Winbond

Winbond is a leading specialty memory solution provider with a wide rage of product portfolio. Owned technology and innovation are our assets for our industry and our customers. Winbond we are high qu...


New Memory and Security Technologies for Designers of IoT Devices

Internet of Things (IoT) edge nodes are battery-powered, often portable, and are connected to an internet gateway or access point wirelessly. This means that the most important constraints on new I...


Winbond TrustMe Secure Flash - A Robust and Certifiable Secure Storage Solution

Winbond has introduced the TrustMe secure flash products to address the challenge of combining security with advanced process nodes and remove the barriers for adding secure non-volatile storage to pr...


Ultra-Low-Power DRAM: A “Green” Memory in IoT Devices

Winbond is offering a new way to extend the power savings available from Partial Array Self-Refresh (PASR), which was already specified in the JEDEC standard by implementing a new Deep Self-Refresh (D...


Polytronics Thermal Conductive Board (TCB) at Electronica 2018

This video introduce the basic product structure, advantage, and application of Polytronics thermal conductive board (TCB). Polytronics exhibit wide range of circuit protection products and thermal ma...


Arrow and Analog Devices strategic partnership and collaborative approach to provide solutions for our customers.

Mike Britchfield (VP for EMEA Sales) talks about why Analog Devices have a collaborative approach with Arrow Arrow’s design resources are key, from regional FAEs in the field to online des...


WE MAKE IT YOURS! Garz & Fricke to present the latest HMIs and SBCs at Electronica 2018

Sascha Ulrich, Head of Sales at Garz & Fricke, gives you a quick overview about the latest SBC, HMI and Panel-PC Highlights at electronica 2018. Learn more about the SANTOKA 15.6 Outdoor HMI, the ...


Macronix Innovations at electronica 2018

Macronix exhibited at electronica 2018 to showcase its latest innovations: 3D NAND, ArmorFlash secure memory, Ultra Low Vcc memory, and the NVM solutions with supreme quality mainly focusing on Automo...


ams CEO talks about their sensor solutions that define the mega trends of the future

In this video Alexander Everke, ams’ CEO, talks to Alix Paultre of EETimes about their optical, imaging and audio sensor solutions in fast-growing markets – from smartphones, mobile device...


Intel accelerated IoT Solutions by Arrow

Arrow is showing Intel’s Market Ready Solutions in a Retailer shop with complete eco environment. From sensors via gateways into the cloud, combined with data analytics, the full range of Intel ...


CSTAR - Manufacturers of cable assembly from Taiwan

CSTAR was founded in 2010 in Taipei, Taiwan. Through years of experience, we are experts in automotive products, LCD displays, LCD TVs, POS, computers, projectors, laptops, digital cameras, medical ca...


NXP Announces LPC5500 MCU Series

Check this video to discover the new NXP microcontroller LPC5500, the target application and focus area. Links to more information: LPC5500 Series: World’s First Arm® Cortex® -M...


Molex Meets Solutions at Electronica

These are exciting times in the electronics world as Molex migrates from a pure connectors company to an innovate solutions provider. Solutions often start at the component level, such as the connecto...


Alix Paultre investigates Bulgin's new optical fiber rugged connector range at Electronica 2018

Alix Paultre interviews Bulgin's Engineering Team Leader Christian Taylor to find out more about the company's new range of optical fiber connectors for harsh environments. As the smallest rug...


Cypress MCU and Connectivity are the best choice for real-world IoT solutions.

Cypress’ VP of Applications, Alan Hawse, explains why people should use Cypress for their IoT connectivity and MCU needs. Cypress wireless connectivity and MCU solutions work robustly and sea...


Chant Sincere unveils their latest High Speed/High Frequency connection solutions at Electronica 2018

Chant Sincere has been creating various of product families to provide comprehensive connection solutions to customers. USB Series Fakra Series QSFP Series Metric Connector Series Fibro ...


Addressing the energy challenge of IoT to unleash billions of devices

ON Semiconductor introduces various IoT use cases targeted towards smart homes/buildings, smart cities, industrial automation and medical applications on node-to-cloud platforms featuring ultra-low po...


ITECH, world leading manufacturer of power test instruments, shinned on electronica 2018

ITECH, as the leading power electronic instruments manufacturer, attended this show and brought abundant test solutions, such as automotive electronics, battery test, solar array simulator, and electr...


ITECH new series give users a fantastic user experience

ITECH latest series products have a first look at the electronics 2018, such as IT6000B regenerative power system, IT6000C bi-directional programmable DC power supply, IT6000D high power programmable ...


SOTB™ Process Technology - Energy Harvesting in Embedded Systems is Now a Reality

Exclusive SOTB technology from Renesas breaks the previous trade-off between achieving either low active current or low standby current consumption – previously you could only choose one. With S...


Power Integrations unveils their new motor control solution

In this video friend of the show Andy Smith of Power Integrations talks to Alix Paultre from Aspencore Media about their new BridgeSwitch ICs, which feature high- and low-side advanced FREDFETs (Fast ...


Panasonic talks about their automotive technology demonstrator

In this video Marco from Panasonic walks Alix Paultre of Aspencore Media through their automotive technology demonstrator at electronica 2018. The demonstrator highlights various vehicle subsystems an...