Data Types, Generators and File Formats

Overview

Coco Test Engine uses the JSON format to store test data and configuration options. In the following sections, we discuss the correspondence between JSON and C++ types, the generators used to create input data, and the format and structure of the data files and configuration files.

The Data Directory

The configuration and data files for the Coco Test Engine are stored in a common directory. Depending on the way Coco Test engine is run, this directory is specified in a different way.

• With the cocotestengine program, it is specified via the --data-dir option.
• With the third-party framework integration, there are three possibilities that are tried in the following order:
  • The directory that is defined in the source code with the COCOTEST_SET_DATADIR macro,
  • The content of the COCOTEST_DATADIR environment variable, or
  • The current working directory.

Data Types

Test data is serialized in JSON file format. The values are read from their string representation in JSON and parsed into their corresponding C++ types through the COCOTEST_FETCH macro. On the other hand, the values in the C++ types are serialized into the corresponding JSON type through the COCOTEST_CHECK macro. The following is a list of all supported data types and their JSON and C++ equivalents.

Test Data File

The data file contains the inputs used to execute a particular test and expected output values with which to compare the result. Its file name takes the form <testname>.data.json. The base name <testname> must be the same as the argument of the COCOTEST and COCOTEST_BEGIN macros in the source code of the test.

The data file has a top-level JSON object with an optional "types" member followed by a required "data" member.

{
    "types": {
        ...
    },
    "data": [
        ...
    ]
}

The "types" member is an object which contains required members "inputs" and "outputs". These, in turn, are also objects whose members are the names of the input and output parameters, respectively. Each parameter is also an object with a required property "type". The value of "type" must be one of entries in the Name column of the table in the data types section.

"types": {
    "inputs": {
        "<input1>": {
            "type": "<input1 type>"
        },
        "<input2>": {
            "type": "<input2 type>"
        },
        ...
    },
    "outputs": {
        "<output1>": {
            "type": "<output1 type>"
        },
        "<output2>": {
            "type": "<output2 type>"
        },
        ...
    }
}

The "data" member is an array of objects. Each object represents a row of data, and contains the required members "inputs" and "outputs", and optional members "validated" and "crash". The members "inputs" and "outputs" must contain name/values pairs where the names are same parameter names defined in the "types" field and the values are the data that will be used for executing the tests. In particular, the input parameters will be used to fill in the values of variables declared with the COCOTEST_FETCH macro, while the output parameters will be used to verify the argument variables to the COCOTEST_CHECK macro.

"data": [
    {
        "inputs": {
            "<input1>": <input1 value>,
            "<input2>": <input2 value>,
            ...
        },
        "outputs": {
            "<output1>": <output1 value>,
            "<output2>": <output2 value>,
            ...
        },
        "validated": true,
        "crash": true
    },
    {
        "inputs": {
            "<input1>": <input1 value>,
            "<input2>": <input2 value>,
            ...
        },
        "outputs": {
            "<output1>": <output1 value>,
            "<output2>": <output2 value>,
            ...
        }
    }
]

Generators

[*] Exponential distribution means it is equally likely to get values from 0 to mean as it is from mean to infinity.

Configuration File

The configuration file for the cocotestengine is written in the JSON format. Its name has the form <testname>.cfg.json, where <testname> is the name of the test to which it refers.

It has two sections. The "inputs" section contains entries about the parameters of the COCOTEST_FETCH macros in the test code and the "outputs" section contains the names of the variables that are used by the COCOTEST_CHECK macros in the test.

Each entry in the "inputs" section has the name of the variable that is declared by the COCOTEST_FETCH macro; in the example below, it is <input1> or <input2>. Each entry is a dictionary that specifies the generator that creates the values for the variable. The dictionary has an entry, "generator", that specifies the name of the generator; note that the type of the values that the generator creates must agree with the type of the variable that is declared by the COCOTEST_FETCH statement in the test code. There are further entries in the dictionary, one for each parameter of the generator, and its value. All of these entries are required, but which ones are needed depends on the generator.

The "outputs" section contains the names of all variables in the test that are parameters of COCOTEST_CHECK macros; their order does not matter.

{
    "inputs": {
        "<input1>": {
            "generator": "<generator1 name>",
            "<param1_1>": <param1_1 value>,
            "<param1_2>": <param1_2 value>,
            ...
        },
        "<input2>": {
            "generator": "<generator2 name>",
            "<param2_1>": <param2_1 value>,
            "<param2_2>": <param2_2 value>,
            ...
        },
        ...
    },
    "outputs": [
        "<output1>",
        "<output2>",
        ...
    ]
}

Run Configuration File

The run configuration file is used for integration with third-party frameworks. It is written in the JSON format. Its name has the form <testname>.run.json, where <testname> is the name of the test to which it refers.

It has the following format, where only "exec_mode" and "max_tries" are required:

{
    "exec_mode": <Mode>,
    "max_coverage": <Float>,
    "max_time": <Integer>,
    "max_tries": <Integer>,
    "save_interval": <Integer>,
    "verbose": <Boolean>
}

The parameters have the following effect:

• "exec_mode": Determine the execution mode. The value may be one of:
  • "Test": Run the existing rows and verify whether the results agree with those in the data file.
  • "Learn": Run the existing rows; their outputs are then stored in the data file.
  • "Discovery": Discover new tests.

  This entry is required. See run modes for details.

• "max_coverage": Set the coverage after which discovery stops.

  The value is a percentage, given as a floating-point number. If during discovery, the coverage reaches or exceeds this value, discovery stops. If the parameter has a negative value or is missing, coverage is ignored when checking for the end of the discovery process.

• "max_time": Set the maximal duration of the discovery process.

  The value is an integer and stands for the number of seconds the discovery process may take. If this time is reached or exceeded, discovery stops. If the parameter has a value of 0 or is missing, the duration of the discovery process is ignored.

• "max_tries": Set the maximal number of newly generated rows that are used in the discovery process.

  The value is an integer. If that number is exceeded, discovery stops. This entry is required.

• "save_interval": Time between automatic saves of the test data that where found so far.

  The parameter is an integer which specifies a time in seconds. If it is greater than 0, it specifies the interval between the times when the currently found rows are saved in the data file. If the value is zero, no data is saved before the end of the discovery process. The default value is 60 seconds.

• "verbose": Print additional output, if there is any.

Execution States

When performing regression testing, the executions recorded in the .csexe file will also contain the state of the row. This is a combination of whether all the comparisons done by COCOTEST_CHECK() succeeded for that row and whether the row is validated or not.

• PASSED: the test was successful and it was validated
• CHECK_MANUALLY: the test was successful and it was not validated
• FAILED: the test was not successful and it was validated
• INCIDENT: the test was not successful and it was not validated