Qt Test Security Considerations

The Qt Test framework manages the execution of tests and generates reports on their results. When using it, many of the considerations related to Using Qt's Developer Tools Securely should be borne in mind. As explained in the Qt Shared Security Model for Qt generally, those using Qt Test need to take care about how they use Qt Test and the data it will produce.

A good suite of tests will, in places, stress the code under test and thus be in danger of failure, including by crashing or exercising undefined behavior. As the code under test evolves, from time to time it shall thereby catch mistakes before they reach end-users. Such failure may have surprising side-effects, including corrupting other resources accessible to the test process, such as source code or build artifacts.

Test failure may be accompanied by extra information describing the nature of the failure. Even when tests pass, they or the code they test may generate output, either due to triggering warning or debug messages or in response to command-line options. The form of this data may render it difficult to report in some cases, or may interfere with formatting of any report in which it is included, either directly by Qt Test or by some other process consuming output produced by Qt Test.

Users of Qt Test are advised to bear these points in mind:

• Test output may contain sensitive data, thereby making it accessible to anyone in a position to read the output.
• Particularly when the test code or code under test is not robustly trusted, tests should be run by an unprivileged process wherever possible and in a sandbox (such as a virtual machine) from which only the report of test results shall be retained after testing.
• Artifacts present on the test system should not be trusted to remain as they were before tests were run.
• Although Qt Test takes such care as it can to escape or otherwise adapt data it embeds in its test output, so that the output adheres to the format asked for, ensure tools that consume that output are robust against the kinds of output your tests could produce.

Build-time artifacts

Even before the test code is compiled, the build-time configuration of a test may adapt the build environment, for example by adding to the build configuration a requirement for software implicated in tests.

It is therefore important to understand what impacts such adaptations may have on the build itself. This may include changes to where build-time tools find source code to incorporate or libraries to link against. Where possible, adaptations needed for a test should not be applied to the building of end-user-facing deliverables, but only to the building of each test that needs them.

This is particularly important when changes to the build-time configuration come from untrusted sources. Such changes have been implicated in security compromises of software packages.

General considerations

Automated testing is not a silver bullet. Existing tests may fail to catch accidental errors or malicious changes and the very fact of running tests may give such a change an opportunity to cause harm. New tests added with a change to the code under test may fail to catch significant issues in the change, or issues in existing code that come to light as a result of the change.

Following Qt Test Best Practices may help to make test behavior more predictable and make it easier to avoid risks, including those to security. As noted there, external dependencies may lead to problems, as described in Avoid external dependencies.

As for Using Qt's Developer Tools Securely, you are responsible for the integrity of the test environment, test code and code under test. Qt Test is not designed to cope with malicious test code or code under test, only to provide a convenient framework for writing tests that can give informative reports when those tests fail.

Separate test and production builds

It is generally prudent to build deliverable software artifacts separately from test artifacts and to save the deliverable artifacts where neither the building of test artifacts nor the effects of running tests can interfere with them.

Keeping these separate avoids potential problems that may arise due to some malfunction in the building of the test, the test itself or the code under test. All potentially have the ability to modify files on disk. This can be a particular concern if any of these steps uses a network connection to a remote server that might be compromised.

Test artifacts should not normally be needed in production systems. Where there is a need to test on production systems, it is prudent to package the tests separately from the code they test, to facilitate uninstalling them once the tests have been run on the system to verify that all is well. This also makes it possible to install only the primary deliverables on copies of the production system, once enough copies have been tested to establish confidence that all systems shall work as intended.

Check output file names

As with any software which writes to files, Qt Test programs have options to control where the output will be saved and in what format.

Where the output file names passed to such options are generated automatically, for example by the build system or a quality management system running the tests, care must be taken to ensure that the generated names are as intended. The system automatically generating such names should check that the names it does generate meet expectations and do not place the output in places where it may be misinterpreted by other systems, might overwrite or replace data from other sources or is itself apt to be overwritten or replaced by such data.