The androidtestrunner Tool

Introduction

The androidtestrunner tool runs Qt tests on Android devices and emulators. It automates the steps required to execute tests, including managing APK installation, test execution, and retrieving results.

Before using the androidtestrunner, ensure you have configured your Qt test project with CMake or qmake.

How it works

1. Firstly, it begins by executing the APK build command to generate the required APK for the test.
2. Following this, it installs the test app on the target device and starts the test app, setting the testing process in motion.
3. The test results are written to the app’s data directory on the device, ensuring that comprehensive test outputs are captured.
4. Once the tests conclude, the runner writes an exit code file beside the result files.
5. At this stage, the androidtestrunner retrieves these result files from the device, examining the exit code for failures.
6. If issues are detected, it immediately prints the app’s logcat logs, including any potential crash stack traces, which are beautified to reveal file names and line numbers for each frame. Additionally, it captures and reports Application Not Responding (ANR) logs if such events occur during the test execution.

To enhance the integration experience, the test runner propagates QT or QTEST environment variables from the host environment of the test runner process to the app, ensuring consistency and a seamless testing workflow.

Running tests with the test wrapper

Qt generates a per-target wrapper script alongside each Android test (named after the test target) that invokes androidtestrunner with the right paths and forwards any extra arguments to the test binary. The example below runs tst_android on a specific emulator, forwards a Qt environment variable, and executes only the testAssets test case:

ANDROID_SERIAL=emulator-5554 QT_DEBUG_PLUGINS=1 ./tst_android testAssets

Retrieving results

By default, and if the stdout test output is not explicitly disabled, the test results are printed to the host as the test is executing. After running the tests, any explicitly requested test files are pulled into the specified output path with their respective formats.

For comprehensive details about Qt Test Framework, see Qt Test Overview.

How to use it

The basic syntax to run the androidtestrunner is as follows:

androidtestrunner [ARGUMENTS] -- [TESTARGS]

To run the test on a specific device/emulator, pass --serial <serial> or set the adb environment variable ANDROID_SERIAL or ANDROID_DEVICE_SERIAL. The explicit option takes precedence.

Mandatory Arguments

The test runner always expects the following arguments to be passed:

• --path <build-path>: The path where the Android Gradle package is built, typically <build-dir>/android-build-<target>.
• --make <build-command>: The command used to build the test APK, for example, cmake --build <build-dir> --target <target>_make_apk.

  Note: Pass this argument quoted so it's not treated as multiple arguments of the test runner but as the value of --make argument.

• One of:
  • --apk <apk-path>: Path to the test APK generated by the build command and installed on the device.
  • --aab <aab-path>: Path to the test AAB; requires --bundletool. Both options cannot be set at the same time, and neither may be specified more than once.

Optional arguments

You can also pass the following optional arguments:

• --bundletool <path>: Path to the Android bundletool jar. Required when --aab is used.
• --manifest <path>: Custom AndroidManifest.xml path. Defaults to a file discovered under the build path or its app/ subdirectory.
• --adb <adb-path>: Specifies a custom ADB command path. Defaults to the adb path found in the system's $PATH.
• --serial <serial>: Android device serial to target. Overrides ANDROID_SERIAL and ANDROID_DEVICE_SERIAL.
• --activity <activity-name>: Specifies a custom activity to run. Defaults to the first activity defined in the AndroidManifest.xml.
• --timeout <seconds>: Sets the timeout for running the test. Defaults to 600 seconds (10 minutes).
• --pre-test-adb-command <command>: Call the adb <command> after installation and before the test run. Can be passed multiple times.
• --skip-install-root: Suppresses the automatic INSTALL_ROOT=<path> install suffix that is appended to --make when the build command is a make-family tool (make, gmake, nmake, mingw32-make, jom; matched case-insensitively). The suffix is never appended for cmake, ninja or other build drivers, so this option only has an effect with the listed tools.
• --ndk-stack <command-path>: Specifies the path to the ndk-stack tool for symbolizing crash stack traces. Defaults to the tool path found under $ANDROID_NDK_ROOT.
• --show-logcat: Prints the logcat output to stdout regardless of test failure. Reads the main, system and crash buffers, and includes system_server lines when an ANR is detected.
• --verbose: Prints verbose output.
• -- <arguments>: Passes anything after the dashes as test arguments.
• --help: Displays the help information.

Example usage

Here is an example that runs the tst_android test, executing only testAssets test case:

androidtestrunner \
    --path ~/tst_android/build/android-build-tst_openssl \
    --make "cmake --build ~/tst_android/build --target apk" \
    --apk ~/tst_android/build/android-build-tst_openssl/tst_openssl.apk \
    testAssets