cmreport – Code coverage report generation

cmreport is a utility which generates Text, HTML, XML or CSV reports from an instrumentation database (a .csmes file). It generates exactly the same reports as those generated by CoverageBrowser (see Generation of reports).

The simplest use case is the generation of an HTML report from a single instrumentation database. If the name of the file is project.csmes, we may write:

$ cmreport --csmes=project.csmes --html=report

This command creates a file report.html and a directory report_html. The file report.html is the main page of the report and can be opened with the browser, while report_html contains all the other files. Most of the other command line options just modify the content of such a report.

Syntax

cmreport -m <csmes_file> ...

Options

• -m <filename> | --csmes=<filename>: Set the name of the main .csmes file (mandatory).
• -p <filename> | --patch=<filename>: Enable patch/diff file analysis. <filename> is the name of a file in unified diff format (see Generation of diff files). The file must contain the differences between the files in the directory tree covered by the .csmes file and a modified version of the same directory tree. Patch reports can be generated in the CSV and in the HTML format.

  With the option --section (see Options for HTML or CSV reports) you can select which sections of the report are displayed.

• --source-type=<argument>: Source type used for the analysis. Possible values:
  • preprocessed: The preprocessed code is analyzed.
  • original: The original source code is analyzed. C macros are ignored in this mode.
• -l <number> | --level=<number>: Set the code coverage level. <number> must be an integer greater than 0.
• --max-threads=<number>: Maximal number of threads used for the report computation. By default, there are as many threads as there are CPUs on the system.
• --stat: Print global coverage statistics to the standard output
• --max-lines-per-page=<number>: Maximal number of lines per page for HTML tables.
• @<path>: Read command line options from the file at <path> and insert them at the position of this option. The option file is a text file with one option per line. Leading and trailing blanks and empty lines are ignored.

File selection options

The following options specify the source files shown in the report.

• --include-file-abs-regex=<regex>: Include the source files that match the regular expression <regex> in the report.
• --exclude-file-abs-regex=<regex>: Exclude the source files that match the regular expression <regex> from the report.
• --include-file-abs-wildcard=<wildcard>: Include the source files that match the wildcard pattern <wildcard> in the report.
• --exclude-file-abs-wildcard=<wildcard>: Exclude the source files that match the wildcard pattern <wildcard> from the report.
• --sources-without-instrumentation: Include source files that are instrumented but that do not have any code to cover.

If two or more of these options match a file, the last one decides whether it is included or excluded. For the syntax of regular expressions and wildcard patterns, see Filtering with wildcards or regular expressions.

Execution selection options

The following options specify the executions that occur in the report.

• -s <regex> | --select=<regex>: Select executions using regular expression. If not used, all executions are implicitly selected.
• -d <regex> | --deselect=<regex>: Deselect executions using regular expression.
• --deselect-to-check: Deselect all executions with the status CHECK_MANUALLY.
• --select-passed: Select all executions with the status PASSED.
• --deselect-passed: Deselect all executions with the status PASSED.
• --select-failed: Select all executions with the status FAILED.
• --deselect-failed: Deselect all executions with the status FAILED.
• --select-unknown: Select all executions with unknown status.
• --deselect-unknown: Deselect all executions with unknown status.
• --executions-reference-from-csmes=<argument>: Select all executions present in the specified .csmes file.

Release comparison

With the following options, one can compare the coverage of two releases.

• --language=<language>: Set cmreport report language.
• -mr <filename> | --csmes-reference=<filename>: Set the name of the .csmes reference file. With this option, cmreport compares two software releases.
• --release-comparison=<argument>: Comparison mode when comparing two software releases. Possible values:
  • none: Display only the differences between two software releases.
  • modified_functions: Display code coverage only for modified functions.
  • unmodified_functions: Display code coverage only for identical functions.
• -sr <regex> | --select-reference=<regex>: Select reference executions using regular expression. This option activates the comparison of executions.
• -dd <regex> | --deselect-reference=<regex>: Deselect reference executions using regular expression. This option activates the comparison of executions.
• --deselect-to-check-reference: Deselect all reference executions with the status CHECK_MANUALLY.
• --select-passed-reference: Select all reference executions with the status PASSED.
• --deselect-passed-reference: Deselect all reference executions with the status PASSED.
• --select-failed-reference: Select all reference executions with the status FAILED.
• --deselect-failed-reference: Deselect all reference executions with the status FAILED.
• --select-unknown-reference: Select all reference executions with unknown status
• --deselect-unknown-reference: Deselect all reference executions with unknown status

Options for HTML or CSV reports

Command line arguments:

• -h <filename> | --html=<filename>: Generate a multi-page HTML report.

  The report consists of a file <file_name>.html, which directs to the index file of the report, and a directory <file_name>_html, which contains all other files in the report.

  If <filename> ends with .html, the ending is ignored, so that no file name or directory is generated that ends with a double html.

• --html-single=<filename>: Generate a single HTML report as file <filename>.

  If <filename> does not end with .html, it is added automatically.

• --csv-excel=<filename>: Generate a CSV report.

  The report is in a format that is used by Excel and OpenOffice. If <filename> does not end with .csv, it is added automatically.

• --title=<string>: Set the title of the report.
• --execution-level=<number>: Set the required level of code coverage to <number>.

  A piece of code is counted as covered in the generated report only if it is at least <number> times executed. The default level is 1.

• --section=<section_name>: Select the sections of the generated report. This option can be repeated to select more than one section type. If nothing is selected, cmreport generates all relevant sections automatically.

  The following sections are possible:

  • global: Global statistics
  • function: Function statistics
  • function-tree: Function tree with statistics
  • class: Class/namespace statistics
  • execution: Execution statistics
  • source: Source file statistics
  • source-tree: Source tree with statistics
  • directory: Directory file statistics
  • manually-validated: Manually validated code fragments
  • unexecuted: Unexecuted code fragments
  • dead-code: Dead code fragments
  • executed: Executed code fragments

  If the option --patch is used, the following sections are also possible.

  • patch-execution-statistic: Global execution statistics of a patch
  • patch-source-statistic: Global source code statistics of a patch
  • patch-execution: Executions impacted by a patch
  • patch-source: Annotated patch file
  • patch-source-instrumented: Annotated patch file for instrumented files only

  For the content of these sections, see Patch Analysis. By default, if no --section option is set, all of them occur in the report.

• --coverage-mcc: Report multiple condition code coverage measurements.
• --coverage-mcdc: Report MC/DC coverage measurements.
• --coverage-condition: Report condition coverage measurements.
• --coverage-decision: Report decision coverage measurements.
• -b | --coverage-statement-block: Report statement (block) coverage measurements.
• --line-coverage: Report line coverage measurements.
• --function-coverage: Report function coverage measurements.
• --coverage-line: Report line coverage measurements (alias to --line-coverage).
• --coverage-function: Report function coverage measurements (alias to --function-coverage).
• -T | --test-coverage: Test count mode
• -D | --debug: Debug flag
• --no-line-metrics: Do not report the eLOC (Effective Lines of Code) metric.
• --no-mccabe-metrics: Do not report the McCabe (or Cyclomatic Complexity) metric.
• --mccabe-group-case-metrics: Report the McCabe metric (or Cyclomatic Complexity) with consecutive cases of a switch grouped.
• --mccabe-condensed-switches-metrics: Report the McCabe metric (or Cyclomatic Complexity) with condensed switches.

Options only for HTML

• --css=<filename>: CSS style sheet
• --icon=<filename>: Icon
• --global-threshold-low-medium=<num>: Global threshold setting
• --global-threshold-medium-high=<num>: Global threshold setting
• --source-threshold-low-medium=<num>: Source/directory threshold setting
• --source-threshold-medium-high=<num>: Source/directory threshold setting
• --function-threshold-low-medium=<num>: Function, class, or namespace threshold setting
• --function-threshold-medium-high=<num>: Function, class, or namespace threshold setting

Options only for CSV

• --csv-field-separator=<char>: Field separator for a CSV file
• --csv-comma=<char>: Separator (, or .) used for floats in a CSV file

Options for text reports

Command line arguments:

• -t <filename> | --text=<filename>: Text report output file name
• --format-executed=<argument>: Line format for the executed for code fragments
• --format-unexecuted=<argument>: Line format for the unexecuted for code fragments
• --format-manually=<argument>: Line format for the manually validated for code fragments
• --format-dead-code=<argument>: Line format for dead code fragments

The following escape sequences are recognized:

• %f: Absolute source code file name
• %r: Relative to the current build directory source code file name
• %l: Line number
• %c: Column number
• %m: Explanation
• %t: Source code fragment
• %e: List of tests executing the instrumentation

It is allowed to to set the output file to an empty string (i.e.: --text=), in this case the standard output of the console is used.

Options for EMMA-XML reports

The following options are used if a report in the EMMA XML format is generated:

• --emma=<filename>: Generate an EMMA-XML report. <filename> is the name of the XML file that is generated.
• --emma-source-pattern=<argument>: Source file pattern for EMMA-XML reports.
• --emma-package-pattern=<argument>: Package name pattern for EMMA-XML reports.

The patterns have the following format:

• %f: file name (without path)
• %b: file name without path and extension
• %e: file extension
• %p: file path (without file name)
• %P: absolute file path (without file name)

Options for JUnit reports

Command line arguments:

• -j <filename> | --junit=<filename>: Generate a JUnit report

JUnit reports do not contain any code coverage information. The report lists only the tests executed, their status (passed or failed) and their comments.

Options for Coberatura reports

To generate a Coberatura report, you need to specify either the first two or the last one of the following options:

• --cobertura=<file>: Generate a Coberatura-XML report. <file> is the path to the generated output file.
• --cobertura-azure-devops: Generate a Coberatura-XML report for Azure DevOps.
• --cobertura-manual-validation-hits=<int>: For manual validations, set the line coverage hits with the parameter in the Coberatura-XML report.
• --cobertura-source=<directory>: Root directory of the source code when generating a Coberatura-XML report.

  This option is required when --cobertura is present; otherwise it is meaningless.

• --sonarqube-project=<path>: Generate a Coberatura-XML report for a SonarQube project.

  <path> is the path to the sonar-project.properties file. The project's root directory is determined from the location of this file. The path to the output XML file is then extracted from its content. It is the value of the sonar.cxx.coverage.reportPath, the sonar.cxx.coverage.itReportPath or the sonar.cxx.coverage.overallReportPath property. Setting any of these properties has the same effect.

  Note: To read Cobertura-XML reports, SonarQube requires a third-party plugin which may not be available in each installation. Instead, you can use the options for SonarQube reports below which generate a report based on SonarQube's generic test coverage format.

Options for SonarQube reports

The following options are used to generate a SonarQube-XML report:

• --sonarqube=<file>: Generate a SonarQube-XML report. <file> is the path to the generated output file.
• --sonarqube-report=<file>: Alias to --sonarqube. It is provided for backwards compatibility.
• --sonarqube-source=<directory>: Root directory of the source code when generating a SonarQube-XML report. It is used to determine the relative paths of source files in the report.

  This option is required when --sonarqube or --sonarqube-report are present; otherwise it is meaningless.

• --sonarqube-manual-validation: Include manual validations in the line coverage in the SonarQube-XML report.