Code coverage measures the extent to which your source code has been tested. It’s crucial to evaluate the quality of your tests and their effectiveness on your software source code, ultimately determining the quality of your project.

Code Coverage analysis is performed with the help of software which is called a code coverage tool

Various tools for code coverage analysis have been created over time which serve different code coverage needs during development and testing of software projects.

Code coverage tools are available both as open source tools and as commercial tools. Each tool has its own benefits and limitations. 

They can be integrated with various build tools like Makefile, Cmake, Gradle, Ant, Maven, etc, Code Quality tools like SonarQube, Continuous Integration tools like Jenkins and a number of other tools which are used during a software development process.

There are different code coverage tools available on the market and they are used by organizations according to their needs. Different tools use different metrics, languages, platforms, and so on.

How does code coverage tools work ?

It adds measurement points/counters at build time and saves the data from counters during execution of tests in a runtime environment. The process of adding measuring points or counters is called instrumentation.

In general, code coverage analysis tools work on two principles.

• Source code instrumentation: It is a very reliable method of instrumentation. This is because each and every line of code and condition is added with a measurement point after the compiler preprocesses the source code. When tests are run against instrumented code, code execution data is collected and saved. The data is then mapped to the source code and code coverage reports are generated. Therefore, you will be able to see the code coverage of every part of your software source code. 

• Byte code or optimized code Instrumentation: In this case, instrumentation is done on bytecode or compiler optimized code. As soon as execution data from the runtime environment (i.e. when we run tests) is collected, the collected code execution data will be mapped only to optimized parts of the source code and code coverage reports are generated.

The following are the list of code coverage tools.

Open-Source Code Coverage Tools

For C & C++

GCov/LCov Tool

GCov is a free open-source code coverage tool which comes as a standard utility with GNU Compiler collection for C/C++ language. It is a statement-by-statement profiling tool.

Due to the fact that GCov instrumented compiler optimized code, you will only be able to see coverage data for compiler optimized code that was instrumented.

Lcov is a free open-source code coverage tool which acts as a graphical front end for Gcov.

Key Features

• It generates a test coverage analysis for specially optimized code.
• It tells how many times a line was executed in a program.
• It gives branch coverage.

Pros

• Gcov is a free open-source code coverage tool.
• It works well with a few Gcov commands.
• It displays the coverage rate of pure C/C++ code more accurately.

Cons

• It is limited to C and C++ code coverage only.
• It is limited to work only on code compiled with the GCC tool chain.
• The coverage will only be displayed for code that has been optimized by the compiler.
• Gcov simply indicates that a line has been covered if any part of code is executed within a line. It will not show the part of code that has partial coverage in a line, if the line has multiple statements and does not execute all paths in branches.
• There is no support for statement coverage. There may be multiple statements on a single source code line or a single statement may be on multiple lines. 
• Embedded safety critical software needs to be checked for multiple conditions, so it will not show the multiple condition coverage or MD/DC coverage.
• There is no back-up support if you run into bugs or need to get coverage from embedded target devices.
• It is not possible to merge code coverage reports from different test runs if source code has changed.
• Delta code coverage and mutation testing code coverage will not be shown.

Cost : Free

For Java/Kotlin

JaCoCo Tool

JaCoCo is a free open-source code coverage tool which is distributed under the terms of the Eclipse Public License for Java language. It is developed by the EclEmma team under the EclEmma plugin for Eclipse and acts as a replacement for EMMA code coverage tool.

JaCoCo tool works on the principle of bytecode instrumentation. You will need to manually configure or add the JaCoCo plugin to pom.xml in the maven build system.

Key Features

• It can instrument Java bytecode on-the-fly means while running the code as well as offline means prior to execution.
• Supports all Java versions Java 7, Java 8, Java 9, Java 10, Java 11, Java 12, Java 13, Java 14, Java 15, Java 16 and Java 17.
• It can be integrated with various tools like Maven, Gradle, Ant, Jenkins, TeamCity, Eclipse IDE, Visual Studio Code IDE, NetBeans and many others.
• Generates reports in HTML, XML and CSV format.
• It has coverage metrics for line, branch, and source code metrics as cyclometric complexity.

Pros

• JaCoCo is an open-source code coverage tool.
• Provides better performance for large scale Java applications at reduced time.
• It can be configured with many other tools and reports generated by it are colorful and easy to understand.
• Works well with minimal dependencies on external libraries or any other resources. 

Cons

• Support Java and Kotlin languages
• It is required to specify JavaAgent as a JVM parameter.
• Manual configuration is required.
• Does not provide condition or multiple condition coverage.
• It can generate code coverage only for unit testing or test suites.
• It is not possible to merge code coverage reports from different test runs if source code has changed.
• Delta code coverage and mutation testing code coverage will not be shown.

Cost : Free

Cobertura Tool

Cobertura is a free open-source code coverage tool for the Java language. It is very simple to use and one of the most accepted tools for its wide features and it is developed based on Jcoverage. Through Cobertura your task can be executed through Maven, Ant etc. 

Key Features

• It has off-line instrumentation which means instrument class files before execution.
• It has coverage metrics for line, branch, global coverage and pre-test coverage through Sonar.
• Generates reports in HTML. XML and Test format via cobertura:check and reports have features of filtering, ascending and descending.

Pros

• Cobertura is an open-source code coverage tool.
• Reports are easy to understand with various filter options.
• Well designed for developers and testers with a wide variety of options.

Cons

• Does not generate reports in CSV, JSON format.
• It is limited to Java code coverage only.
• It supports less test frameworks.
• No support for multiple condition code coverage.
• No support for delta code coverage or mutation testing code coverage.

Cost : Free

JCov Tool

JCov is a free open-source code coverage tool which is developed and implemented with SUN Open JDK for Java language. It can be plugged into any Java environment and you can get the power of almost all coverage metrics.

Key Features

• It can instrument Java bytecode on-the-fly means while running the code as well as offline means prior to execution.
• It covers metrics like Block coverage, Line coverage, Branch coverage, Method coverage and almost all other coverages like Field coverage, API coverage, Pre-test coverage information.
• Collected coverage data local file system and on a server.

Pros

• JCov is an open-source code coverage tool.
• Shows execution paths of your Java application only for unit testing or test suite.

Cons

• It is limited to Java code coverage only.
• It can generate code coverage for the test suite or unit testing only.

Cost : Free

CodeCover Tool

CodeCover is a free open-source glass-box testing tool which is licensed under the Eclipse Public License (EPL) for Java and COBOL. This tool can be integrated with various development and testing tools like Jenkins, Ant, Junit, Eclipse etc.

Key Features:

• Java and COBOL testing are performed through this glass box tool.
• Coverage metrics for statements, methods, branches, loops, MCs, as well as coverage measurements for pre-tests, term coverage, and synchronized coverage are included in this document.
• Generates coverage reports in template engine velocity format.

Pros:

• CodeCover is a free open-source code coverage tool.
•  It is an excellent choice for improving testing quality and creating new test cases.

Cons:

• It is limited to Java and COBOL code coverage only.
• It is limited to instruments in a single source directory only.
• It is limited to only unit testing

Cost : Free

Emma Tool

Emma is a free open-source code coverage tool which was developed by Vlad Rubtsov and it is distributed under the Common Public License (CPL) for the Java language. It is one of the oldest and popular coverage tools, but currently it is not under active development.

Key Features

• It can instrument classes on-the-fly means while running the code as well as offline means prior to execution.
• It supports large scale Java applications.
• It covers coverage types for Class, Method, Line, Basic block.
• Generates reports in HTML, XML and plain text format where HTML reports support source code linking.
• It allows merge of coverage data obtained from various instruments.
• It has no external library dependencies.

Pros

• Emma is a free open-source code coverage tool.
• It is an easy to use tool.

Cons:

• Works only on Java source code.
• Because development has ceased, it is not well maintained and does not support the latest Java versions.

Cost : Free

For Python

PyTestCov Tool

PyTestCov is a free open-source code coverage tool for the Python language.

Key Features

• PyTestCov plugin supports centralized testing and distributed testing in both load and each mode.
• It supports coverage of sub processes.
• All the features offered by the coverage package can be availed by pytest-cov or by coverage’s config file.
• It allows you to skip any test method from a group of test methods during execution.

Pros

• It is faster and easier to learn.
• Multiple tests can be executed in parallel which reduces execution time.
• It detects test files and test functions automatically.

Cons

• It is limited to Python code coverage only.
• The pytest-cov package needs to be installed for distributed testing and for sub process measurement.
• Does not provide coverage in details ,like interline coloring for partially executed conditions.
• Does not provide multiple condition coverage.
• In the event of source code changes, code coverage reports cannot be merged.
• Code coverage for delta and mutation testing will not be shown.

Cost : Free

Coverage.py Tool 

Coverage.py is a free open-source code coverage tool which was originally developed by Gareth Rees and maintained by Ned Batcgelder and team. It is a coverage tool for the Python language. It monitors every bit of your program and tells which parts of the code were executed and which part was not executed by identifying the source code.

Key Features

• Allows you to specify the source files that you want to analyze through the configuration files.
• Coverage.py by default can measure line coverage.
• It generates reports in various formats like Text, HTML, XML, LCOV and JSON.
• It provides an API where you can avail result data in an SQLite database.

Pros

• It is a free open-source code coverage tool.
• It can be used by using command line or by using API.

Cons

• It is limited to Python code coverage only.
• When source code has changed, it is not possible to merge code coverage reports from different test runs.
• Delta code coverage and mutation testing code coverage will not be displayed.

Cost : Free

For .Net & C#

OpenCover Tool

OpenCover is a free open-source code coverage tool for .Net software which supports .Net 2 and above software. Does not have support for MONO.

Key Features

• You can install it by NuGet package, MSI or ZIP file.
• It provides support for both 64-bit and 32. Bit for .Net 4 and .Net 2
• OpenCover is a command line tool.
• Reports can be generated in  XML format .

Pros

• OpenCover is a free open-source code coverage tool.
• It provides user friendly documentation for OpenCover installation.

Cons

• It works on Windows only.
• It is limited to .Net software only.
• Works only for unit testing and does not work for functional testing of applications.
• Does not show partial code coverage for partially executed lines.
• When source code has changed, it is not possible to merge code coverage reports from different test runs.
• Delta code coverage and mutation testing code coverage will not be shown.

Cost : Free

AltCover Tool

AltCover is a free open-source code coverage tool for .Net software and Mono.

Key Features

•  It weaves IL for its coverage.
•  It is a command line tool for code coverage.
•  It generates XML, LCov, Cobertura Reports.
•  It can work on any platform with read/write access.

Pros

• It can be run in a number of ways which includes global tools, dotnet tests, powershell integration and MSBuild tasks.

Cons

• It is limited to .Net software and Mono only.
• It is limited to unit testing coverage and no support for functional testing coverage.
• It does not support multiple condition coverage.
• If any part of code is executed in line, it simply shows as covered. No partial code coverage information.

Cost : Free

For JavaScript

Istanbul Tool

Istanbul is one of the most popular free open source code coverage tools which is licensed under BSD License for JavaScript.

Key Features

• You can generate reports for unit tests, server-side functional tests or browser tests .
• It covers metrics like statement, branch and function coverage and it instruments code with line counters so easily you can track the exercise of unit tests on your source code.
• It can use a middleware while serving JavaScript files which need to be tested on the browser.
• It supports coverage of Node JS unit tests.
• It generates reports in multiple formats like HTML, LCOV, Cobertura etc.
• Both a command line and a library can be used with it.

Pros

• It provides easy installation.
• Results can be displayed directly on your terminal.

Cons

• It is limited to JavaScript only.
• Does not show code coverage for partially covered code.

Cost : Free

Karma Tool

Code coverage for JavaScript is provided by Karma, an open-source tool. With Karma, developers can test their code effectively.

Key Features

• It does require manual configurations to use Karma.
• It allows you to control the complete workflow from the command line or from the IDE.
• It can be simply integrated with Jenkins, Travis and Semaphore.
• It generates reports in multiple formats like HTML, LCOV, Text, Cobertura etc.

Pros

• Real devices and browsers can be used to test your code.
• Karma supports a plethora of testing frameworks.
• It is easy to use.

 Cons

• It is limited to JavaScript only.
• Does not show the coverage report for partially covered code.

Cost : Free

Commercial Code Coverage Tools

Bullseye Tool

Bullseye is a commercial code coverage tool developed by Bullseye Testing Technology for the C and C++ language. It is mostly used as an advanced tool for improving software quality in the medical, aerospace, and defense sector with limited support for compilers and cross compilers.

Key Features

• It works well for any application written in C and C++ including system and Kernel modes provided in supported environments.
• Reports can be generated in HTML, XML and can be imported to bullseye GUI.
• It provides additional features like code excluding, merging and visualizing etc.

Pros

• It provides high speed execution.
• It takes less renewal charges i.e. $200 annually.
• This tool is user friendly as it can be used by following simple steps.

Cons

• It is limited to C and C++ languages.
• Not all compilers and cross compilers are support. for example Freescale, Keil uVision, Renesas, LLVM, Intel C++, Microchip etc. are not support by tool.
• It allows merging of results only at functional level but not at statement level or condition level.
• In the event of source code changes, code coverage reports cannot be merged.
• We will not show delta code coverage and mutation testing code coverage.

Price: $800 per license for the first year ,$200 renewal fee.

Squish COCO

Squish Coco is a code coverage tool developed by FrogLogic for C, C++, C#, System C, Tcl code and QML.

Key Features

• It acts as a complement to FrogLogic’s UI testing tool Squish.
• The language supports C, C++, System C, C#, Tcl, and QML code.
• Reports can be generated in various formats which includes HTML, Text, XML, Junit and Cobertura.
• It can be easily integrated with multiple builds, test tools and CI frameworks.
• It covers metrics like statements, branches , conditions MC/DC coverage.

Pros

•  It provides a free trial version and demo before use.

Cons

• You can not use it for Java and Python coverage testing.
• Limited compilers support
• When source code has changed, it is not possible to merge code coverage reports from different test runs.
• Delta code coverage and mutation testing code coverage will not be shown.

Cost: Price not available

RKTracer Tool

The RKTracer code coverage tool is developed by RKValidate for C, C++, C#, Java, Kotlin, JavaScript, Python, Golang, and Swift programming languages. If you have a software application with one of the languages supported, you can generate code coverage no matter what compiler, cross compiler, embedded targets, or servers you use or how you test your application. One solution that works for all host and target environments.

Key Features

• Very easy to use tool
• Supports all compiler, Cross-Compilers, Embedded targets and IDEs
• Auto generation of unit tests in GTest and Unity frameworks
• Supports unit ,integration, functional testing and embedded target testing.
• Generates coverage reports in function, Line, Statement , Branch or Decision, Condition, MC/DC Coverage and Multiple condition coverage.
• Generates a user friendly coverage report in HTML, XML format. Also, it is easier to publish reports in Jenkins, SonarQube and in Azure DevOps with built-in plugins.
• It highlights the parts of code executed in clear cut reports green, yellow and red to determine which statement was executed fully or partially or not at all.
• It comes with a Plug-in for multiple IDEs IAR embedded workbench IDE, DS-5 IDE, Eclipse IDE, Visual Studio IDE, Android Studio IDE, Tizen Studio IDE, etc.

Pros

• It is compatible with all compilers and cross compilers.
• Works with all build and embedded target environments.
• It provides 24*5 support and user guidance.
• Generated reports are clear and concise with coverage for partially covered code.
• You can merge code coverage reports from different test runs if source code has changed.
• You can generate delta code coverage and mutation testing code coverage.

Cons

•  It is not free software.

Cost: Price not available

VectorCast tool

VectorCast is a code coverage tool developed by Vector Software Inc. for C and C++ programming languages which supports a wide range of compilers, simulators and process architecture.

Key Features

• You can easily perform quality trends and change impact analysis.
• It allows parallel testing.
• It supports statement, branch and multiple condition/decision coverage criteria.
• It supports unit, integration, functional and embedded target testing.
• It automated regression testing.
• It provides a web-based quality dashboard.
• Creates unit and integration tests for C and C++.
• It supports Agile and Test-Driven development methods.

Pros

• You can get a free trial before use.
• It allows automatic test creation based on decision paths.

Cons

• You can not use it for widely used Java and Python coverage testing.
• Limited compiler and cross-compilers supported.
• It does not support resource limited embedded systems.
• It is not possible to merge code coverage reports from different test runs if source code has changed.
• Delta code coverage and mutation testing code coverage will not be shown.

Cost: Price not available

Parasoft Tool

Parasoft C/C++Test/Jtest/C# developed by Parasoft provides code coverage for C, C++, Java and .NET programs.

Key Features

• The solution provides coverage for lines, statements, blocks, decision/branch, paths, MC/DC, and conditions.
• Can generate code coverage for unit, integration testing by mapping to source code in eclipse IDE.As a result you can directly edit source code.
• Tool is built on the Eclipse IDE interface or in the command line mode.

Pros

• You can use it with all build tools, CI tools, and IDEs.
• It’s pictorial representation of coverage details is more user friendly.

Cons

•  Developers will find it more user-friendly and it is more appropriate for traditional corporations.
• You can not use its service for free of cost and it is very expensive also.
• It is not possible to merge code coverage reports from different test runs if source code has changed.
• Delta code coverage and mutation testing code coverage will not be shown.

Cost: Price not available

CANTATA Tool

Cantata tool developed by QA Systems company is a very good automated testing suite for C and C++ languages.

Key Features

• It supports almost all coverage levels.
• It provides unparalleled filtering of code coverage.
• It generates reports in HTML, XML reports.
• You can integrate coverage analysis with Cantata tests or with any other tests.
• Generate unit tests and integration tests

Pros

• It cuts the cost of standard compliance and takes lower testing costs.
• Generates auto unit tests and integration tests
• It reports failures more clearly and precisely for complex applications.

Cons

• It is limited to C and C++ only.
• Does not support all compilers and cross compilers
• It is not possible to merge code coverage reports from different test runs if source code has changed.
• Delta code coverage and mutation testing code coverage will not be shown.

Cost: Price not available

NCover Tool

NCover is a code coverage tool developed by Peter Waldschmidt for .Net applications.

Key Features

• Reports for  statement coverage and branch coverage.
• Centralized code coverage reports.
• It provides extensive documentation and user support.
• It works confidently in an agile environment.
• It presents the reports in HTML format.

Pros

• It is available on commercial license as well as open source.
• Code coverage tests can be performed manually or automatically with this tool.

Cons

• It is limited to the .Net platform only.

Cost: $480 to $658 per year.

DotCover Tool

DotCover is a commercial code coverage tool which is offered by JetBrains for .Net applications.

Key Features

• You can easily integrate it with Visual Studio IDE.
• You can perform statement level code coverage for .NET, Silverlight or .NET Core
• It provides a console utility to use with a continuous integration server.
• You can easily customize your coverage analysis with a coverage filter.

Pros

• It provides detailed reporting.

Cons

• It is limited to the .Net platform only.

 Cost: 11.90 to $39.90 per month

How to choose a code coverage tool ?

Consider the following factors when selecting a tool for code coverage analysis. Regardless of how long one has been in business and how many clients one has, you must always perform proof of concept(POC).

• Is the tool compatible with all the programming languages used in my organization.
• If one is looking for a code coverage tool for C or C++, does the tool support all compilers, cross compilers, and embedded targets? Does this tool have limitations that may impact future updates of compilers or cross compilers used during software development.
• Is a code coverage tool easy to use? It should be easy to integrate with existing development and testing environments. This allows a developer or tester to generate code coverage with just a few additional steps. 
• When you are developing software for embedded systems, a code coverage tool should be capable enough to auto detect and add run-time libraries to build systems based on embedded target architectures.
• No matter what the company claims about its support processes. You should determine for yourself how fast they can resolve issues in POC or how well the support was provided in POC.

Conclusion

Many code coverage tools are available in the market now. The important thing is to pick the right code coverage tool for your project which meets all your requirements.

The preferred approach is to choose a code coverage tool that can meet all your requirements without any limitations and has good technical support, rather than selecting a tool that works in selected environments.

So don’t forget to check out RKTacer tool to ensure quality code coverage and ultimately quality software projects.