How to Test Multiple AUTs from a Single Test Script, Using ApplicationContext

Usually, a single application under test is specified for each test suite. This AUT is then executed and accessed by each test case. All the tutorials show this one test suite/one AUT approach, but you can also start multiple applications and access and test all of them from within a single test suite. This makes it possible to test the interaction between different applications or between multiple instances of the same application. For example, being able to test multiple applications is essential for testing client/server systems.

Whenever an AUT is started, a corresponding Application Context object is created, and it is this object that is used by Squish to provide access to the AUT. Squish allows us to access the ApplicationContext object directly in our code, and this means that we can query the AUT for information such as the command line it was launched with, its current state, and so on. This information can also be accessed by making use of the context object returned by the ApplicationContext currentApplicationContext() function.

How to Start and Access Multiple Applications Under Test

When testing multiple applications from a single test script, the first step is to ensure that no application is set to be automatically started. Using the Squish IDE, click the Test Suite Settings toolbar button in the Test Suites view to make the test suite's Test Suite Settings view visible. Now, in the editor's Application Under Test (AUT) section, make sure that the Automatically start the AUT checkbox is unchecked.

The function used to start an application is ApplicationContext startApplication(autName). This function starts the given application (assuming it is located in an application path—see AUTs and Settings) using the given command line arguments and returns a corresponding ApplicationContext object. The application context object is a handle that refers to the application.

Optionally, as the second and third parameters, a host and port can be passed to the ApplicationContext startApplication(autName) function. This way, the ApplicationContext startApplication(autName) function will connect to the squishserver on the specified host and listen to the specified port, instead of using the default host and port (as specified in the Squish IDE's settings or on the squishrunner's command line). This allows us to control multiple applications on multiple computers from a single test script.

Special care must be taken if the application is using a different GUI toolkit than the test suite's default toolkit. The global testSettings object object allows us to set the configuration of the toolkit wrapper on a per-AUT basis. For more information, see testSettings.setWrappersForApplication(application, wrapperList).

If we run two or more AUTs within a test script, which one should test code apply to? We can make one of the AUTs the active application by using the setApplicationContext(contextHandle) function, passing an ApplicationContext as the sole parameter. Once the call is made, all script code applies to the active application—unless another setApplicationContext(contextHandle) call is made to change the active application. Whenever we call the ApplicationContext startApplication(autName) function, not only is the application's ApplicationContext object returned, but the application is automatically set to be the active application.

We can obtain a list of all the currently running AUTs' ApplicationContext objects by calling the SequenceOfApplicationContexts applicationContextList() function. And we can retrieve the context object of the active application by calling the ApplicationContext currentApplicationContext() function.

We will now look at some examples that show how to start multiple AUTs and how to use ApplicationContext objects to query them.

We will take as an example a client/server chat system. The system has a chat server called chatserver, written in Qt, which must be running for communication to take place, and two chat clients, one written in Qt called chatclientqt, and the other written in Windows called chatclientwin. We will use a Squish for Qt package, which includes the wrappers for Qt and native Windows applications, and a Qt test suite.

In the test we will first start the chat server. Then we start two clients; these automatically connect to the chat server at startup. We will then type something into the message editor of the first client and check that the second client received the message.

startApplication("chatserver")
client1 = startApplication("chatclientqt")
testSettings.setWrappersForApplication("chatclientwin", ["Windows"])
client2 = startApplication("chatclientwin")

setApplicationContext(client1)
editor = waitForObject("ChatWindow.messageEditor")
type(editor, "Message for client #2")

setApplicationContext(client2)
msgView = waitForObject("ChatWindow.messageView")
test.compare(msgView.text, "Message for client #2")

startApplication("chatserver");
var client1 = startApplication("chatclientqt");
testSettings.setWrappersForApplication("chatclientwin", ["Windows"]);
var client2 = startApplication("chatclientwin");

setApplicationContext(client1);
var editor = waitForObject("ChatWindow.messageEditor");
type(editor, "Message for client #2");

setApplicationContext(client2);
var msgView = waitForObject("ChatWindow.messageView");
test.compare(msgView.text, "Message for client #2");

startApplication("chatserver");
my $client1 = startApplication("chatclientqt");
testSettings->setWrappersForApplication("chatclientwin", ("Windows"));
my $client2 = startApplication("chatclientwin");

setApplicationContext($client1);
my $editor = waitForObject("ChatWindow.messageEditor");
type($editor, "Message for client #2");

setApplicationContext($client2);
my $msgView = waitForObject("ChatWindow.messageView");
test::compare($msgView->text, "Message for client #2");

startApplication("chatserver")
client1 = startApplication("chatclientqt")
testSettings.setWrappersForApplication("chatclientwin", ("Windows"))
client2 = startApplication("chatclientwin")

setApplicationContext(client1)
editor = waitForObject("ChatWindow.messageEditor")
type(editor, "Message for client #2")

setApplicationContext(client2)
msgView = waitForObject("ChatWindow.messageView")
Test.compare(msgView.text, "Message for client #2")

startApplication "chatserver"
set client1 [startApplication "chatclientqt"]
testSettings setWrappersForApplication chatclientwin { Windows }
set client2 [startApplication "chatclientwin"]

setApplicationContext $client1
set editor [waitForObject "ChatWindow.messageEditor"]
invoke type $editor "Message for client #2"

setApplicationContext $client2
set msgView [waitForObject "ChatWindow.messageView"]
test compare [property get $msgView text] "Message for client #2"

We begin by starting each of the applications in turn, although we only keep references to the client AUTs' ApplicationContext objects since we don't directly access the server in the test. For the Windows client we set the toolkit wrapper to use the Windows wrapper instead of the default Qt wrapper. Once the applications are running we make the first client the active AUT since the active AUT is currently client2 since that was the AUT started by the most recent ApplicationContext startApplication(autName) call. Then we get a reference to the client's chat editor and type some text into it. And at the end, we make the second client the active AUT, get a reference to its chat editor (a different widget this time since the toolkit is different—Java rather than Qt), and we compare the second client's editor's text with the text we sent from the first client.

Notes on using startApplication with Android

For Android the autName in ApplicationContext startApplication(autName) function is the name of the package. It may be followed by a slash plus activity. Without an activity, the package main activity is started, otherwise the one given.

When the activity is a dash, then the application is started but no activity is started. Use this when your AUT uses activities from other packages.

Finally, the package name may be prefixed with a device string, a set of launcher argument settins plus colon. So when specifying everything, the autName looks like device{option1,...}:package/activity and without options device:package/activity. Likewise device maybe omitted when options are wanted but the device is the default one (provided by the Squish IDE or squishrunner).

If for example only one of multiple started apps should have their settings cleared when started by Squish, the ApplicationContext startApplication(autName) command can use the --clear-app-settings launcher argument in the application string. E.g. startApplication("{clear-app-settings}:com.froglogic.addressbook").

We will take as example a package com.example.android.tools from which a package com.example.android.carpenter launches an activity when tapping a button. A tool selection will automatically close the activity and we are back at our main activity.

ctx0 = startApplication("com.example.android.tools/-")
ctx1 = startApplication("com.example.android.carpenter")
tapObject(waitForObject(":Choose Tool_Button")
setApplicationContext(ctx0)
tapObject(waitForObjectItem("_List", "Hammer"))
setApplicationContext(ctx1)

var ctx0 = startApplication("com.example.android.tools/-");
var ctx1 = startApplication("com.example.android.carpenter");
tapObject(waitForObject(":Choose Tool_Button");
setApplicationContext(ctx0);
tapObject(waitForObjectItem("_List", "Hammer"));
setApplicationContext(ctx1);

my $ctx0 = startApplication("com.example.android.tools/-");
my $ctx1 = startApplication("com.example.android.carpenter");
tapObject(waitForObject(":Choose Tool_Button");
setApplicationContext(ctx0);
tapObject(waitForObjectItem("_List", "Hammer"));
setApplicationContext(ctx1);

ctx0 = startApplication("com.example.android.tools/-")
ctx1 = startApplication("com.example.android.carpenter")
tapObject(waitForObject(":Choose Tool_Button")
setApplicationContext(ctx0)
tapObject(waitForObjectItem("_List", "Hammer"))
setApplicationContext(ctx1)

set ctx0 [startApplication "com.example.android.tools/-"]
set ctx1 [startApplication "com.example.android.carpenter"]
invoke tapObject [waitForObject ":Choose Tool_Button"]
setApplicationContext $ctx0
invoke tapObject [waitForObjectItem "_List", "Hammer"]
setApplicationContext $ctx1

How to Use ApplicationContext Objects

Use an ApplicationContext object to retrieve information about the AUT it refers to. The application context of the AUT defined in the test suite settings can be retrieved using the ApplicationContext defaultApplicationContext() function, and of the currently running AUT by the ApplicationContext currentApplicationContext() function. When multiple AUTs are started, there should not be any AUT defined in the test suite settings. Each AUT's context object can be retrieved as the return value of the call to the ApplicationContext startApplication(autName) function, which is used to start the AUT, or from the SequenceOfApplicationContexts applicationContextList() function, which returns all the AUTs' context objects.

The Application Context section details the properties and functions that are accessible from ApplicationContext objects. Here are some examples:

ctx = currentApplicationContext()
test.log(ctx.commandLine)
test.log(ctx.cwd)

var ctx = currentApplicationContext();
test.log(ctx.commandLine);
test.log(ctx.cwd);

my $ctx = currentApplicationContext();
test::log($ctx->commandLine);
test::log($ctx->cwd);

ctx = currentApplicationContext()
Test.log(ctx.commandLine)
Test.log(ctx.cwd)

set ctx [currentApplicationContext]
test log [applicationContext $ctx commandLine]
test log [applicationContext $ctx cwd]

Here we print the command line the AUT was invoked with and its current working directory—both are properties:

ctx = currentApplicationContext()
peakMemory = 0
while ctx.isRunning:
    peakMemory = max(ctx.usedMemory, peakMemory)
    if not ctx.isFrozen(20):
    break
test.log("Peak Memory: %d" % peakMemory)

var ctx = currentApplicationContext();
var peakMemory = 0;
while (ctx.isRunning) {
    peakMemory = Math.max(ctx.usedMemory, peakMemory);
    if (!ctx.isFrozen(20))
    break;
}
test.log("Peak Memory: " + peakMemory);

my $ctx = currentApplicationContext();
my $peakMemory = 0;
while ($ctx->isRunning) {
    if ($ctx->usedMemory > $peakMemory) {
    $peakMemory = $ctx->usedMemory;
    }
    if (!$ctx->isFrozen(20)) {
    last;
    }
}
test::log("Peak Memory: $peakMemory")

ctx = currentApplicationContext()
peakMemory = 0
while ctx.isRunning
    peakMemory = ctx.usedMemory > peakMemory ? ctx.usedMemory : peakMemory
    if !ctx.isFrozen(20)
    break
    end
end
Test.log("Peak Memory: #{peakMemory}")

set ctx [currentApplicationContext]
set peakMemory 0
while {[applicationContext $ctx isRunning] == 1} {
    if {[applicationContext $ctx usedMemory] > $peakMemory} {
    set peakMemory [applicationContext $ctx usedMemory]
    }
    if {![applicationContext $ctx isFrozen 20]} {
    break
    }
}
test log "Peak Memory: $peakMemory"

Here we access the currently running AUT and keep track of the maximum amount of memory it is using. We break out of the loop if the application stops running (in which case isRunning will be false), or if the application becomes unresponsive (frozen), after waiting 20 seconds:

ctx = currentApplicationContext()
test.log("STDOUT", ctx.readStdout())
test.warning("STDERR", ctx.readStderr())

var ctx = currentApplicationContext();
test.log("STDOUT", ctx.readStdout());
test.warning("STDERR", ctx.readStderr());

my $ctx = currentApplicationContext();
test::log("STDOUT", $ctx->readStdout());
test::warning("STDERR", $ctx->readStderr());

ctx = currentApplicationContext()
Test.log("STDOUT", ctx.readStdout())
Test.warning("STDERR", ctx.readStderr())

set ctx [currentApplicationContext]
test log "STDOUT" [applicationContext $ctx readStdout]
test warning "STDERR" [applicationContext $ctx readStderr]

We added everything that the AUT has written to stdout and stderr to the test log, classifying all stderr messages as warnings.