The androidtestrunner Tool

Introduction

The androidtestrunner tool runs Qt tests on Android devices and emulators. It automates the steps required to execute tests, including managing APK installation, test execution, and retrieving results.

Before using the androidtestrunner, ensure you have configured your Qt test project with CMake or qmake.

How it works

1. Firstly, it begins by executing the APK build command to generate the required APK for the test.
2. Following this, it installs the test app on the target device and starts the test app, setting the testing process in motion.
3. The test results are written to the app’s data directory on the device, ensuring that comprehensive test outputs are captured.
4. Once the tests conclude, the runner writes an exit code file beside the result files.
5. At this stage, the androidtestrunner retrieves these result files from the device, examining the exit code for failures.
6. If issues are detected, it immediately prints the app’s logcat logs, including any potential crash stack traces, which are beautified to reveal file names and line numbers for each frame. Additionally, it captures and reports Application Not Responding (ANR) logs if such events occur during the test execution.

To enhance the integration experience, the test runner propagates QT or QTEST environment variables from the host environment of the test runner process to the app, ensuring consistency and a seamless testing workflow.

Running tests with the test wrapper

Here is an example that runs the tst_android test while running it on a specific emulator instance, passing an environment variable and executing only testAssets test case:

ANDROID_SERIAL=emulator-5554 QT_DEBUG_PLUGINS=1 ./tst_android testAssets

Retrieving results

By default, and if the stdout test output is not explicitly disabled, the test results are printed to the host as the test is executing. After running the tests, any explicitly requested test files are pulled into the specified output path with their respective formats.

For comprehensive details about Qt Test Framework, see Qt Test Overview.

How to use it

The basic syntax to run the androidtestrunner is as follows:

androidtestrunner [ARGUMENTS] -- [TESTARGS]

To run the test on a specific device/emulator, use the adb environment varialbe ANDROID_SERIAL or ANDROID_DEVICE_SERIAL.

Mandatory Arguments

The test runner always expects the following arguments to be passed:

• --path <build-path>: The path where the Android Gradle package is built, commonly under android-build-testname.
• --make <build-command>: The command used to build the test APK, for example, cmake --build <build-dir> --target <target>_make_apk.

  Note: Pass this argument quoted so it's not treated as multiple arguments of the test runner but as the value of --make argument.

• --apk <apk-path>: The Path to the test APK that is generated by the build command and that is installed on the device.

Optional arguments

You can also pass the following optional arguments:

• --adb <adb-path>: Specifies a custom ADB command path. Defaults to the adb path found in the system's $PATH.
• --activity <activity-name>: Specifies a custom activity to run. Defaults to the first activity defined in the AndroidManifest.xml.
• --timeout <seconds>: Sets the timeout for running the test. Defaults to 600 seconds (10 minutes).
• --skip-install-root: Avoids appending INSTALL_ROOT to the make command and is mainly useful when using qmake.
• --show-logcat: Prints the logcat output to stdout regardless of test failure. Logs are printed in case of failure and can include crash stacktraces or ANR events.
• --ndk-stack <command-path>: Specifies the path to the ndk-stack tool for symbolizing crash stack traces. Defaults to the tool path found under $ANDROID_NDK_ROOT.
• -- <arguments>: Passes anything after the dashes as test arguments.
• --verbose: Prints verbose output.
• --help: Displays the help information.

Example usage

Here is an example that runs the tst_android test, executing only testAssets test case:

androidtestrunner \
    --path ~/tst_android/build/android-build-tst_openssl \
    --make "cmake --build ~/tst_android/build --target apk" \
    --apk ~/tst_android/build/android-build-tst_openssl/tst_openssl.apk \
    --skip-install-root \
    testAssets