Test Coverage

Introduction

TICS has the abitily to collect and show test coverage output of data generated by third-party test coverage tools. The definition of test coverage (also known as code coverage) according to Wikipedia:

Code coverage is a measure used in software testing. It describes the degree to which the source code of a program has been tested. It is a form of testing that inspects the code directly and is therefore a form of white box testing.

TICS can collect three metrics of test coverage, insofar as those metrics are supported by the third-party tool:

Branch Coverage (or Decision/Condition Coverage)
Has each evaluation point (such as a true/false decision) been executed?
Statement Coverage (or Line Coverage)
Has each line of the source code been executed?
Function Coverage
Has each function in the program been executed?

When test coverage has been enabled (by configuring a test coverage tool, see below), the TICSQServer run will collect the test coverage results generated by the test coverage tools and store them in the TICS quality database, which can be inspected with the TICS viewer, and aggregated results are sent to the TIOBE Enterprise Dashboard (TED), if this has been configured.

Configuring Test Coverage

To configure test coverage for TICS, specify which test coverage tools are used to generate the test coverage results. TICS supports a number of test coverage tools. Most test coverage tools support only a limited number of programming languages (mostly only a single one). To reflect the language dependency, test coverage tools are configured in the 'LANGUAGES' section of the SERVER.txt configuration file.

For example, configuring VectorCAST/Cover for C++ would look as follows:

'LANGUAGES' => {
  'CPP' => {
    'TESTCOVERAGE' => {
      'TOOL' => 'vectorcast'
    }
  }
}

Not all required properties are shown in the example. For a complete list of options and supported test coverage tools, see the SERVER.txt Reference Guide.

Test coverage tools generate their results in a directory that can be within a TICS branch directory or outside such a directory. In the former case, no configuration is needed. TICS looks into the branch directories by default. If the test coverage results are outside the project branch directories, this directory must to be configured.

The test coverage result directory must be configured per branch. Project branch properties are configured in the PROJECTS.txt file.

For example, configuring the location of the test coverage results (C:\temp\myresults) for branch trunk would look as follows:

'myproject' => {
  'VIEWS' => {
    'trunk' => {
      'UNITTESTCOVERAGE' => {
        'RESULTDIR' => 'C:\temp\myresults',
      }
    }
  }
}

For more information on the PROJECTS.txt file, see the PROJECTS.txt Reference Guide.

VectorCAST/Cover

VectorCAST/Cover is a tool of Vector Software Inc. VectorCAST/Cover supports both statement and branch coverage. For more information about this VectorCAST/Cover, see their manuals.

It is important to know that TICS uses the <projectname>_metrics_report.html files to collect the test coverage results from. These files must be generated separately from TICS, either by adding a so-called prepare step, running VectorCAST/Cover from the command line, or from the VectorCAST/Cover GUI.

VectorCAST/Cover Limitations

VectorCAST/Cover only stores the base name of a file and not the complete path to the source file in the result reports. As a consequence, the current implementation of TICS cannot uniquely identify to which source file the results belong to in case there are more than one source files with the same name.

In such cases, TICS will identify this and not collect the test coverage results for those files, but instead mark them as failed, with an error message indicating the multiple file name problem.