Code coverage for a QML application

In this tutorial, we will illustrate how to use Coco to measure the code coverage of a Qt AUT written in C++ and QML. We describe how to measure the code coverage for each language independently and how they can be combined into a single report.

We will be using a simple GUI application called parser_qml, which is an extension of the parser example described in Instrumentation of a simple project. It uses C++ backend similar to the previous examples, and has a graphical interface written in QtQuick, and some code written in JavaScript.

Setting up the tutorial

Prerequisites

For simplicity, we assume you are running a Linux machine with gcc compilers installed. However, the corresponding steps can be performed on Windows or macOS, Visual Studio or Clang, and other supported toolchains.

This tutorial further assumes that you have:

• a Coco installation in your environment PATH,
• a Qt 6 installation in your environment PATH, and
• a copy of the CocoQML add-on which can be downloaded separately from your Qt account.

Getting the tutorial source code

The source code for this example can be found in SquishCoco/samples/parser/parser_qml on Linux and macOS, and <Windows Coco>\parser\parser_qml on Windows. We first have to copy this directory to our workspace:

$ cd /home/user/cocoqml-tutorial
$ cp -r /path/to/SquishCoco/samples/parser/parser_qml .

We can also take this step to browse the source code. As an option, we can also compile and execute the application without code coverage:

$ mkdir parser_qml-build && cd parser_qml-build
$ qmake ../parser_qml
$ make
$ ./parser_qml

Setting up CocoQML

Extracting the CocoQML package

Next, we extract the contents of the CocoQML package in our workspace:

$ tar xvzf cocoqml-7.2.0-linux.tgz
$ ls -F cocoqml-7.2.0-linux
bin/  trackerplugin/

The version number 7.2.0 should be replaced by whichever version of CocoQML you have. This results in a directory cocoqml-7.2.0-linux containing:

• bin/cocoqmlscanner - the application that instruments the QML source code with coverage counters, and
• trackerplugin directory - the QML plugin which records the values of the coverage counters during execution.

For a more detailed explanation on how CocoQML works, you can read our dedicated QML Coverage reference page.

Compiling the trackerplugin

The next step is to compile the trackerplugin. It is a qmake project and must be compiled with the same Qt installation as the AUT. We compile it in a separate build directory called trackerplugin-build:

$ cd cocoqml-7.2.0-linux
$ mkdir trackerplugin-build && cd trackerplugin-build
$ qmake ../trackerplugin
$ make

This creates a directory QmlJsCoverage, which contains a plugin called cocoqmltracker. This module, which must be loaded by the instrumented QML application, takes care of recording the coverage counters into a .csexe file. Take note of the location of the trackerplugin-build directory since it will be needed in a later step.

Instrumenting the source code

This step analyzes the source code and inserts counters to determine which parts of the code have been executed. This has to be done separately for the C++ and QML parts of the source code.

QML Instrumentation

First, we use cocoqmlscanner to instrument the QML source code.

$ cd /home/user/cocoqml-tutorial/parser_qml
$ ../cocoqml-7.2.0-linux/bin/cocoqmlscanner .

cocoqmlscanner modifies the .qml and .js source files by inserting coverage counters in the code while keeping copies of the original code in backup files. It also creates an instrumentations database for the QML and JS source files in the form of a .csmes file called cocoqmlscanner_result.csmes. Its filename can be specified using the -c or --csmes-name options.

More details about cocoqmlscanner's command line options can be found in its dedicated reference page.

C++ Compilation and Instrumentation

Next, the instrumentation of the C++ code is done by CoverageScanner, which is part of the main Coco package. To do this, one of the CoverageScanner wrappers is called instead of the compiler. For example, csg++ is called instead of g++.

In parser_qml.pro, this can be enabled using the CodeCoverage configuration:

CodeCoverage {
    COVERAGE_OPTIONS = --cs-mcdc --cs-exclude-file-abs-wildcard=* --cs-include-path=$$PWD

    QMAKE_CFLAGS   += $$COVERAGE_OPTIONS
    QMAKE_CXXFLAGS += $$COVERAGE_OPTIONS
    QMAKE_LFLAGS   += $$COVERAGE_OPTIONS

    defineReplace(toCoco) {
        cmd = $$1
        path = $$take_first(cmd)
        prog = $$basename(path)

        return(cs$$prog $$cmd)
    }

    QMAKE_AR = $$toCoco($$QMAKE_AR)
    QMAKE_CC = $$toCoco($$QMAKE_CC)
    QMAKE_CXX = $$toCoco($$QMAKE_CXX)
    QMAKE_LIB = $$toCoco($$QMAKE_LIB)
    QMAKE_LINK = $$toCoco($$QMAKE_LINK)
    QMAKE_LINK_SHLIB_CMD = $$toCoco($$QMAKE_LINK_SHLIB_CMD)
}

This tells qmake to use CoverageScanner for compilation. The given COVERAGE_OPTIONS tell CoverageScanner to measure MC/DC coverage and exclude all other source files outside of this directory.

To compile with code coverage enabled, we simply need to add CodeCoverage to the configuration:

$ cd /home/user/cocoqml-tutorial
$ mkdir parser_qml-coverage && cd parser_qml-coverage
$ qmake ../parser_qml CONFIG+=CodeCoverage
$ make

This creates the instrumented executable parser_qml and the instrumentations database parser_qml.csmes for the C++ source files.

Further details about qmake integration can be found here.

Executing the instrumented application

In order for the instrumented QML code to work, we need to point the QML runtime to the trackerplugin that we built in the previous step:

$ export QML_IMPORT_PATH=/home/user/cocoqml-tutorial/cocoqml-7.2.0-linux/trackerplugin-build

Failing to do this often results in the error message: module "QmlJsCoverage" is not installed.

Now, we can execute the application:

$ ./parser_qml

Using the graphical interface, you can input various arithmetic operations and click the buttons in order to cover different parts of the source code.

Once you exit the application, two .csexe files are written in the current working directory:

• parser_qml.csexe containing C++ coverage counters and
• ParserQML_qml.csexe containing QML coverage counters

Importing and merging coverage data

Next, we import the coverage counters into the corresponding instrumentations database using the cmcsexeimport utility. We do this for C++ and QML separately:

$ cmcsexeimport -m parser_qml.csmes -e parser_qml.csexe -t "C++"
$ cmcsexeimport -m ../parser_qml/cocoqmlscanner_result.csmes -e ParserQML_qml.csexe -t "QML"

If we want to see the coverage data for all languages in one place, we can merge the two .csmes files together using the cmmerge utility:

$ cmmerge -o parser_qml.csmes -a ../parser_qml/cocoqmlscanner_result.csmes

Finally, we can browse the contents of the coverage data using CoverageBrowser:

$ coveragebrowser -m parser_qml.csmes

If everything was done right, you should be able to see coverage data for C++, QML and JS files in the Sources pane, as well as the entries "C++" and "QML" in the Executions pane.