Code coverage shows the parts of your code that are exercised during test execution. A code coverage tool is a program that captures code execution data and generates a report of source code that was covered and uncovered during the testing process. 

Code Coverage can be used for measuring the effectiveness of your tests or test suite’s and checking how well your tests cover the application source code or software code.

In this post will walk through what it does, why it’s useful, and some strategies for getting the most out of it in this post.  

What are the advantages of Code Coverage analysis?

• Shows the proof of how many lines and conditions have been covered by the tests.
• Testing status can be tracked easily by developers and compared with the source code. This will allow them to pay more attention to the untouched parts of the code and improve testing.
• It makes writing tests or unit tests and integration testing easier. By proper unit testing, integration testing or functional testing, you can enhance the productivity and predictability of your software releases. 
• Scalability of a code is determined by its coverage data. Developers generate coverage reports based on the test runs analysis. As a result, if new code gets added, it will be covered by existing tests or more tests will have to be added to cover it.
• It tells about the strength and limitations of each testing technique. More precisely measures the efficiency of test implementations. 
• It helps in identifying the areas of test cases which are having no use for the current project.
• It is a benchmark proof that the tests written by testers have covered the majority of code in the application and have been tested properly according to the requirements and demands of the application.
• When code goes through multiple release cycles, there may be a chance for unit tests to atrophy. So, to avoid test entropy we need code coverage.
• Continual analysis aids the developer in completing the project faster by examining good and bad aspects and finding easy and faster solutions accordingly.
• Software is tested properly before it is released. So, the developers are well confident to have minimal problems.
• It helps in increasing ROI by satisfying customers.
• It allows companies to launch numbers of software within a short period of time.
• Through various testing methods, it prevents bugs in software indirectly.

How does code coverage analysis work?

An approach known as Instrumentation monitors build systems and inserts trace statements into preprocessed source code or byte code to collect code execution data at runtime, which is then mapped to source code to generate coverage reports.

The instrumentation approach is mainly composed of three types

1. Source Code Instrumentation

In source code instrumentation first instrumentation statements are added in source code then the source code is compiled by using the normal tool chain(compiler or cross compiler). After successful compilation of source code, instrumented assembly is generated. 

Runtime environment execution data is gathered and mapped with code source file to generate coverage reports.

2. Runtime Instrumentation

In this case, instrumentation occurs in optimized code, so most of the code will be optimized (GCov, LCov, etc.) then preprocessed optimized code will be instrumented. It is not reliable, as you will be able to see the code coverage for striped code, regardless of whether you pass optimization flags to reduce the executable size or strip the code. However, the compiler will strip the code or optimize it. Such instrumentation is not reliable.

Runtime environment execution data is gathered and mapped with only the stripped code source file, and coverage reports will be generated.

3. Intermediate Code Instrumentation

In intermediate code instrumentation, a new instrumented class is generated by adding byte codes to compiled class files.

Code coverage tools which are available in the market now majorly use one of the above techniques to measure code coverage metrics. As a result based on your requirement you choose the instrumentation approach and a right code coverage tool for your project.

How do you measure code coverage of a program?

Then, how do we generate code coverage metrics reports? To begin with, you will need to integrate the code coverage tool into the application build system.

Build and execute tests, e.g. unit tests, integration tests, or functional tests for an application. After the testing is complete, you will need to generate a coverage report. Reports can be generated in a variety of formats, such as HTML and XML.   

Code Coverage tool works on the principle of code instrumentation or byte/optimized code instrumentation to track the execution of code at runtime and to generate coverage reports.

For example, you can generate an HTML report that displays the uncovered and executed code in three different colors.

• A green highlighted code indicates that it has been covered.
• A red-colored code shows that it has been uncovered.
• A yellow highlighted code indicates that it has been partially covered.

In this way, we can determine the extent of our testing and how effective our test cases are.

Code Coverage VS Test Coverage

Test coverage and code coverage are two of the most effective methods for determining the effectiveness of a code. Both topics have become popular topics in the QA/DevOps community.   

Most of the time it has been seen that both development team and testing team are confused in both the terminology while others think both are much similar having no difference. But code coverage and test coverage are different, having their own roles and responsibilities.

CODE COVERAGE	TEST COVERAGE
It is the total percentage of code that is executed or tested as a whole.	It refers to measuring the features being tested as part of functional requirements.
If your source code has an if-else block, then the if and else blocks have been executed. This means you have 100% code coverage.	Suppose you want to check the compatibility of your software application with an embedded system, a network, or a browser. In test coverage, you will learn that it is valid and compatible on a wide range of platforms, networks, and browsers.
It can be performed by various testing methods like unit testing,integration testing ,functional testing then combining the coverage data from different tests.	You run tests either unit tests or integration test or functional tests and generate test coverage report and effectiveness of  tests.
Developers can monitor the various automated tests that are run by using coverage html reports.	Developers can monitor and measure of the amount of code that has been executed by the tests.
It comes under White Box testing mentality.To check if all the parts of code is executed or not with various tests.But does not check quality of tests.	While Test coverage comes under Gray Box testing mentality. It checks effectiveness of tests and coverage data.
From a functional requirement point of view code coverage is objective as it does not tell about how well a software is tested.	From a functional requirement point of view test coverage is subjective as it tells about how well a software is tested.
Some tools for code coverage are RKTracer,Gcov, JaCoCo, JCov etc.	Some tools for test coverage are RKTracer, Gcov,Junit, PyUnit etc.

Types of Code Coverage

How code coverage is calculated and measure the lines of code that are actually tested and verified. Code coverage metrics can be calculated in several ways. Below are some examples.

Function coverage

The function coverage enables us to see which functions were executed at least once when testing our applications. However, it does not indicate whether the function definition was covered. All it tells us is whether the function was executed or not.

For ex:

 int fun(int a int b) 

 {

      int c;

      c=a+b;

      return c;

 }

When testing an application, we can use function coverage to determine how many functions have been executed.

Line Coverage

Line coverage shows how many executable lines are covered by tests (excluding comments and untested code). This number is calculated by dividing the number of lines that are executed (or touched) by the number of lines in the program.

Line Coverage= Number of lines of code executed during testing/Total number of executable lines of code in the program.

You need line coverage because it gives you a bird’s eye view of how many lines of code are colored.

• Green color = Covered
• Yellow color = partially covered
• Red color = Not covered 

Statement coverage

Statements are blocks of code that perform a specific task. For example, an assignment statement assigns a value to a variable. A “for” statement creates a loop. Statements can be grouped together as one statement using curly brackets in C, C++, and C#.

An example of a statement is a semicolon, which is executable code, or anything that ends with a semicolon. It is not considered a statement if it has only a semicolon and no machine code.

Statement Coverage= Number of statements executed during testing/Total number of executable statements in an application 

Statements are not equal to lines

For example:

a++; b+=2; if (a) b=0; //Here we have four statements in a single line.

Statement 1: a++;

Statement 2: b+=2;

Statement 3: if (a) b=0;

Statement 4: b=0;

Branch Coverage or Decision Coverage

Branch coverage tells you if you took the true or false branch for each condition (if, while, for). We’ll have twice as many branches as conditionals.

  If (a>b || b>c )

  {

  else

  }

By executing the if and else statements once to true and once to false consecutively, we will have 100% branch coverage. In this case, sub-conditions (a>b and b>c) are ignored. In addition, Boolean expressions are ignored.

Condition Coverage

Conditional statement coverage is also referred to as expression coverage. A sub-expression must be evaluated at least once to true and once to false. 

  If (a>b || b>c )

  {

  else

  }

Condition coverage is similar to branch coverage, except that it also checks for individual subexpression outcomes for true or false (a>b and b>c). Each logical condition is checked individually. The complete coverage in condition coverage does not imply a complete branch or decision coverage rate.

Multi-condition coverage

A combination of conditions is also referred to as multi condition coverage. When multi-condition coverage is used, it looks at every true/false value in a combination. Multiple condition coverage examines every condition in the decision. The coverage is calculated by dividing the total number of successful condition combos and line blocks by the total number of conditions and blocks in the project.

Multi-condition coverage is the most accurate and reliable coverage metric. It ensures that other coverages are also included.

If we pass value=3 to a given “if” block, we will cover four multiple condition coverage points. Only one condition is covered.

Now let’s pass values 3,5,7 and 89. Now we can see that all the multiple conditions points are covered. 

Modified Condition Decision Coverage

In a given decision block, each sub-condition should be evaluated at least once as true and once as false. Therefore, the decision block should be changed to true or false.

Decision block:

 “if (value < 4 || value == 5 || value == 7)”    //complete if condition is decision block.

• Sub-condition 1: value < 4
• Sub-condition 2: value ==5
• Sub-condition 3: value ==7

Let’s say we have the following decision “if (value < 4 || value == 5 || value == 7)” block with three sub-conditions i.e. “value < 4”, “value == 5” , “value == 7”. If we pass value==3 then sub condition “value<4” will be executed for true, a value 3 is less then 4.However, MC/DC states sub condition should be executed at least once to true and once to false, and as a result of it decision block should be changed to either true or false. Since “value<4” has been executed only to true, so MC/DC condition has not been met.

Let’s say we pass two values, i.e., value==3 and then value==4, so that the sub-condition “value<4” will be executed once to true and one to false as a result, a decision block will be executed to the true or false condition.

You can see that we have covered MC/DC coverage for sub conditions “value<4,” and you need to take the same approach for the remaining two MC/DC conditions.

Path Coverage

The concept of path coverage refers to determining whether all paths through a given function or method have been executed. A path is a sequence of branches from the function entry to the function exit. This is also referred to as Predicate coverage.

Path Coverage = (Number of path executed / Total number of possible paths in a function) * 100

Loop Coverage

Loop coverage refers to determining if the loop body (such as a while loop or for loop) is executed exactly zero times, once or more than once.

What is code coverage in unit testing?

Testing the smallest part(unit) of your application, i.e. function or method by means of test script or test suite. Scripts refer to test cases that examine specific responses by passing a set of arguments based on function input parameters and use assertions to compare the results with the expected outcome. These tests are called unit tests. Unit tests provide a means of verification for refactoring code. 

Test coverage refers to the amount of code executed by a unit test, whether it is line coverage, branch coverage, or condition coverage. A unit test that checks “branch a” will report code coverage as 50% branch coverage, if your application has only two branches (branch a and branch b). In order to achieve 100% branch coverage, you will need additional test cases to cover uncovered branch paths.

Merge results from different test runs?

A majority of code coverage tools will display results from the most recent test runs. Merging coverage results is important if you combine coverage data from multiple test runs, or if you combine unit testing coverage and functional testing coverage when targeting safety-critical embedded devices or secure servers.  

In order to achieve maximum coverage, it is advisable to combine unit testing, functional testing, and system testing and mark relevant code as covered by manual analysis.

Most code coverage tools will not allow you to merge results if you change the application code or refactor the source code. This is because there will be mismatches in the numbers recorded. As long as you do not modify the application code, you can combine coverage results from unit testing, system testing, and functional testing.   

However, the RKTracer code coverage tool allows you to merge results from different tests even if the code has changed or been refactored.

Exclude elements/source files from code coverage results ?

You may need to exclude source code files coverage reports in the following situation.

If you are working on a real-world software application, you will come across situations where code coverage tools, by default, generate coverage reports for third party source code, over which we have no control, and if the third party source has minimal coverage, the overall coverage of the application will be reduced.  

Embedded systems typically contain both data and program memory on the same chip. Unlike desktop PCs, smartphones are powered by microprocessors, in which the memory is a separate component from the processor chip. 

However, most embedded microcontrollers have anywhere from a few hundred bytes to perhaps a few hundred KB of RAM. Embedded systems do not generally make use of dynamic memory allocation. 

As a result of memory limitations in embedded systems, one cannot instrument whole applications and run them on embedded targets.

For these scenarios, we can enable the code coverage tool on selected modules, folders or functions and generate code coverage for those selected modules, folders or functions.  

For example:

Ignore *

Instrument /folder-name/ filename.cpp 

Make code coverage part of your continuous integration flow

If you do not achieve a sufficiently high percentage of coverage after establishing a workflow of continuous integration, .Run tests and check if it has met the threshold of 85%. If not, check whether test failures are a result of timeouts or for some other unknown reason and add additional tests for uncovered parts of your code by analyzing coverage reports.  

Drawbacks of code coverage analysis

• It does not evaluate if the program is bug-free or not. It is not useful for debugging the code.
• It does not determine if the code is correct or not. 
• It does not determine the correctness of the logic.
• It does not provide a 100% coverage guarantee for an application.
• It does not provide total accuracy so sometimes according to the analysis results may be unpredictable.
• Different code coverage levels/metrics can not measure and ensure the quality of tests.

What are the challenges of implementing code coverage analysis ?

• Mapping is unclear between source functionality and the code captured.
• Code coverage faces a challenge to capture implicit logic.
• Finding and selecting a good code coverage tool for your project which suits your language and environment and tracks all types of test coverage like unit, API, System and even manual testing is challenging.

Note: While many code coverage tools available in the market are still failing to provide enhanced service to the user that’s why still there are a number of drawbacks for available code coverage tools. Even developers face many challenges due to these code coverage tools which are not up to mark. But now you have a code coverage tool like RKTracer tool which is a multi-platform, multi-language code coverage tool capable enough to meet all your requirements with less cost and reduced time.

Conclusion:

As the world becomes more and more tech-driven, customers are more and more looking for low-cost, high-quality software products. At the same time, developers and testers are also responsible for providing the end user with the best quality service. In this case, code coverage analysis becomes a valuable tool to the development team, allowing it to shine a light on your source code’s quality and effectiveness, resulting in an efficient product.

Finally, don’t forget to begin your code coverage analysis with a code coverage tool such as RKTracer along with your development, testing, and Quality Assurance efforts. A code coverage tool such as this provides a powerful way of improving code quality and positively impacting software quality.