Test Coverage

Introduction

TICS has the abitily to collect and show test coverage output of data generated by third-party test coverage tools. The definition of test coverage (also known as code coverage) according to Wikipedia:

Code coverage is a measure used in software testing. It describes the degree to which the source code of a program has been tested. It is a form of testing that inspects the code directly and is therefore a form of white box testing.

TICS can collect three metrics of test coverage, insofar as those metrics are supported by the third-party tool:

Branch Coverage (or Decision/Condition Coverage)

Has each evaluation point (such as a true/false decision) been executed?

Statement Coverage (or Line Coverage)

Has each line of the source code been executed?

Function Coverage

Has each function in the program been executed?

When test coverage has been enabled (by configuring one or more test coverage tools, see below), the TICSQServer run will collect the test coverage results generated by the test coverage tools and store them in the TICS quality database, which can be inspected with the TICS viewer, and aggregated results are sent to the TICS Enterprise Dashboard (TED), if this has been configured.

Configuring Test Coverage

To configure test coverage for TICS, specify which test coverage tools are used to generate the test coverage results. TICS supports a number of test coverage tools. Most test coverage tools support only a limited number of programming languages (mostly only a single one). To reflect the language dependency, test coverage tools are configured in the LANGUAGES section of the SERVER.yaml configuration file.

For example, configuring VectorCAST/Cover for C++ would look as follows:

LANGUAGES:
  CPP:
    TESTCOVERAGE:
      TOOLS: [ 'vectorcast' ]

If you wish to configure multiple tools for one language this can also be done. For example, configuring Cobertura and Jacocofor Java would look as follows:

LANGUAGES:
  JAVA:
    TESTCOVERAGE:
      TOOLS: ['Cobertura', 'Jacoco']

Test coverage tools generate their results in a directory that can be within a TICS branch directory or outside such a directory. In the former case, no configuration is needed. TICS looks into the branch directories by default. If the test coverage results are outside the project branch directories, this directory must to be configured.

The test coverage result directory must be configured per branch. Project branch properties are configured in the PROJECTS.yaml file.

For example, configuring the location of the test coverage results (C:\temp\myresults) for branch trunk would look as follows:

myproject:
  VIEWS:
    trunk:
      UNITTESTCOVERAGE:
        RESULTDIR: 'C:\temp\myresults'

For more information on the PROJECTS.yaml file, see the PROJECTS.yaml Reference Guide.

VectorCAST/Cover

VectorCAST/Cover is a tool of Vector Software Inc. VectorCAST/Cover supports both statement and branch coverage. For more information about this VectorCAST/Cover, see their manuals.

It is important to know that TICS uses the <projectname>_metrics_report.html files to collect the test coverage results from. These files must be generated separately from TICS, either by adding a so-called prepare step, running VectorCAST/Cover from the command line, or from the VectorCAST/Cover GUI.

VectorCAST/Cover Limitations

VectorCAST/Cover only stores the base name of a file and not the complete path to the source file in the result reports. As a consequence, the current implementation of TICS cannot uniquely identify to which source file the results belong to in case there are more than one source files with the same name.

In such cases, TICS will identify this and not collect the test coverage results for those files, but instead mark them as failed, with an error message indicating the multiple file name problem.