Code Coverage Everything you need to know
Code coverage metrics show you how much of your code has been covered under tests and how much has been left uncovered. Coverage metrics assist you in your software testing.
This post will walk through what it does, why it’s useful, and some strategies for getting the most out of it.
What is the purpose of code coverage analysis?
Code coverage measurement is an important tool in the arsenal of any software engineer. It’s used to measure how much of your codebase is covered by tests or automated tests, and thus provides good insight into whether you’re testing enough (or too much) to improve code quality.
How do YOU measure code coverage?
Then, how do we generate code coverage metrics reports? To begin with, you will need to integrate the code coverage tool(code coverage analyzer) into the application build system.
Build and run tests i.e. unit tests or integration tests, or functional tests on the application. Once the testing is completed, you will need to generate a code coverage report. The report can be generated in different formats like HTML ,XML.
Code Coverage tool works on the principle of code instrumentation or object code instrumentation to track the execution of code at runtime and to generate coverage report.
For example, you can generate an HTML report that displays the uncovered and executed code in three different colors.
- A green highlighted code indicates that it has been covered.
- A red-colored code shows that it has been uncovered.
- A yellow highlighted code indicates that it has been partially covered.
In this way, we can determine the extent of our testing and how effective our test cases are.
Code Coverage vs Test Coverage
Code coverage is simply an aggregation of the overall coverage of an application that is based on the different phases of the application life cycle, such as unit testing, integration testing, and functional testing.
Test coverage is the process of running a unit test case or functional test case and checking the resultant coverage of the function for which we are running a unit test case or checking the coverage of functionality as a result of running functional tests.
In addition, functions call another function (integration testing), and the type of coverage generated is called path coverage.
Types of code coverage
Function coverage gives us a bird’s-eye view of which functions were executed at least once when we test our applications. However, it does not tell us whether the function definition was covered. It only tells us whether the function was executed or not.
int fun(int a int b)
We can use function coverage in order to have an idea of how many functions have been executed when testing our application.
Line coverage shows the number of executable lines covered by tests (excluding comments) and untested code. It is computed by dividing the number of lines executed (or touched) by the number of lines in the program.
Line Coverage= Number of lines of code executed during testing/Total number of executable lines of code in the program.
Line coverage is essential because it provides you with a bird’s-eye view of the total lines of code covered by color.
- Green color = Covered
- Yellow color = partially covered
- Red color = Not covered
A statement is a block of code that performs a certain task. An assignment statement, for example, assigns a value to a variable. A “for” statement is used to create a loop. Curly brackets can be used to group statements together as one statement in C, C++, and C#.
A statement is a simple statement such as a semicolon, which is executable code, or anything ending with a semicolon. Statements with no machine code, just ‘”;” and a semicolon, are not considered statements.
Statement Coverage= Number of statements executed during testing/Total number of executable statements in an application
Statements are not equal to lines
a++; b+=2; if (a) b=0; //Here we have four statements in a single line.
Statement 1: a++;
Statement 2: b+=2;
Statement 3: if (a) b=0;
Statement 4: b=0;
Branch Coverage or Decision Coverage
The branch coverage or decision coverage determines whether you took the true or false branch for each condition (if, while, for). The number of branches will be twice as many as the number of conditionals.
If (a>b || b>c )
If the if and else statements have been executed once to true and once to false consecutively, we will have 100% branch coverage. Here sub-conditions (a>b and b>c) are ignored. However, the Boolean expressions are ignored.
Condition coverage is also termed as expression coverage metric. Every sub-expression must be executed at least once to true and once to false.
If (a>b || b>c )
Condition coverage is similar to branch coverage but also checks for individual sub-expressions (a>b and b>c) outcomes for turn and false. It is used to check individual outcomes for each logical condition. However, the complete coverage in condition coverage does not imply a total branch/decision coverage rate.
This is also known as a combination of condition coverage. A multiple condition coverage looks at every true/false value with the combination. A multiple condition coverage looks at every single condition in the decision. The coverage is sought by dividing the total number of successful condition combos and line blocks by the total number of conditions and blocks in the project.
Multi-condition coverage is the most efficient and reliable coverage metric. You can be sure that the other covers are covered as well.
There are four multiple condition coverage points covered in a given decision block if we pass value==3 to a given “if” block. Only one condition is covered.
Now let’s pass values 3,5,7 and 89. Now we can see that all the multiple conditions points are covered.
Modified Condition Decision Coverage
In a given decision block, each sub-condition should be executed at least once to true and once to false. As a result, the decision block should be changed to true or false.
“if (value < 4 || value == 5 || value == 7)” //complete if condition is decision block.
- Sub-condition 1: value < 4
- Sub-condition 2: value ==5
- Sub-condition 3: value ==7
Let’s say we have the following decision “if (value < 4 || value == 5 || value == 7)” block with three sub-conditions i.e “value < 4”, “value == 5” , “value == 7”. If we pass value==3 then sub condition “value<4” will be executed for true, a value 3 is less then 4.However, MC/DC states sub condition should be executed at least once to true and once to false, and as a result of it decision block should be changed to either true or false. Since “value<4” has been executed only to true, so MC/DC condition has not been met.
Let’s say we pass two values, i.e., value==3 and then value==4, so that the sub-condition “value<4” will be executed once to true and one to false as a result, a decision block will be executed to the true or false condition.
You can see that we have covered MC/DC coverage for sub condition “value<4,” and you need to take the same approach for the remaining two MC/DC conditions.
What is code coverage in unit testing?
Testing the smallest part(unit) of your application, i.e. function or method by means of test script or test suite. Scripts refer to test cases that examine specific responses by passing a set of arguments based on function input parameters and use assertions to compare the results with the expected ones. These tests are called unit tests. Unit tests provide a means of verification for refactoring code.
Code coverage metric is the amount of code executed by unit tests, be it lines coverage, branch coverage, or condition coverage. If your application has only two conditional branches (branch a and branch b), a unit test that checks branch code coverage will report 50% coverage for conditional branch.
Merge results from different runs?
Almost all code coverage tools will show results from the latest test runs. Merging coverage result or merging code coverage results comes into play if you combine data from multiple test run or if you’re targeting safety-critical embedded devices or secure servers.
In order to achieve maximum coverage, it is best to combine system testing, unit testing, and functional testing, and mark relevant code as covered by manual analysis. You might need to merge coverage information or data from different test runs when you’re checking your code coverage at various stages in the testing process.
If you change the application or refactor source code, most code coverage tools won’t let you merge results as there is will mismatch in counters recorded. You can merge coverage results from unit testing, system testing, and functional testing as long as you don’t modify the application code.
However, the RKTracer code coverage tool allows you to merge results from different tests even if the code has changed or been refactored.
Exclude elements/source files from code coverage results ?
When we are working on a real-world software application you will end up with situations where code coverage tools by default generate code coverage data for third party source code over which we don’t have control and if the third party source has minimal code coverage, overall code coverage for the application will reduce overall code coverage.
In embedded systems, data and program memory are usually on the same chip as the processor. As opposed to desktop PCs, laptops, tablets or even smartphones use microprocessors, where the memory is separate from the processor chip.
But most embedded microcontrollers have anywhere from a few hundred bytes to perhaps a few hundred KBs of RAM. Dynamic allocation is generally not used in embedded systems.
As a result of memory limitations in embedded systems, one can instrument whole applications and run them on embedded targets.
In these situations, we can enable the code coverage tool on selected modules or folders or functions or even on selected functions and generate code coverage.
Instrument /folder-name/ filename.cpp
What percentage of coverage should I aim for?
There are no gold bullet for coverage criteria and 80% coverage is one good target for test code. Doing an attempted expansion might prove expensive while providing not necessary coverage at the same time ensuring sufficient benefits.
The most crucial part for the testing team is providing time for users to consider testing from the end-user perspective and not using static tests. For example in the above example at 100% coverage by testing, 100 and 34 was a multiple of 10.
How will we solve this question without using numbers in the name of our service? Can you give real or false predictions? Do you really want the exception? In particular.
Make code coverage part of your continuous integration flow
If you do not achieve a sufficiently high percentage of coverage after establishing a workflow of continuous integration, you can try failing tests. Of course, as we said it would not be reasonable if this threshold had become too high.
If your objective is 80% coverage, consider establishing a failure limit of 70% as a measure of security. Please do not send the wrong message as pressing the team to achieve able coverage could lead to bad testing practices. For example, 90% of all buildings would suffer if a cover wasn’t maintained or if the cover got worn down a lot and failed.