Support for specific test frameworks

CppUnit

CppUnit

Note: Project page of CppUnit: http://cppunit.sourceforge.net

is a unit test framework for C++. This environment can easily be adapted to get the code coverage from each unit test.

The following code is an example how this can be done:

    #include <cppunit/TestListener.h>
    #include <cppunit/BriefTestProgressListener.h>
    #include <cppunit/CompilerOutputter.h>
    #include <cppunit/extensions/TestFactoryRegistry.h>
    #include <cppunit/TestResult.h>
    #include <cppunit/TestResultCollector.h>
    #include <cppunit/TestRunner.h>

    class CoverageScannerListener : public CppUnit::TestListener
    {
        public:
            CoverageScannerListener() {}

            void startTest( CppUnit::Test *test )
            {
                m_testFailed = false;
    #ifdef __COVERAGESCANNER__
                int pos;
                // Adjusting the name of the test to display the tests
                // in a tree view in CoverageBrowser
                std::string testname = "CppUnit/" + test->getName();
                while ( ( pos = testname.find( "::", 0 ) ) != std::string::npos )
                    testname.replace( pos, 2, "/" );

                // Reset the code coverage data to get only the code coverage
                // of the actual unit test.
                __coveragescanner_clear();
                __coveragescanner_testname( testname.c_str() ) ;
    #endif
            }

            void addFailure( const CppUnit::TestFailure &failure )
            {
                m_testFailed = true;
            }

            void endTest( CppUnit::Test *test )
            {
    #ifdef __COVERAGESCANNER__
                // Recording the execution state in the coverage report
                if ( m_testFailed )
                    __coveragescanner_teststate( "FAILED" );
                else
                    __coveragescanner_teststate( "PASSED" );

                // Saving the code coverage report of the unit test
                __coveragescanner_save();
                __coveragescanner_testname( "" );
    #endif
            }

        private:
            bool m_testFailed;
            // Prevents the use of the copy constructor and operator.
            CoverageScannerListener( const CoverageScannerListener &copy );
            void operator =( const CoverageScannerListener &copy );
    };

    int main( int argc, char* argv[] )
    {
    #ifdef __COVERAGESCANNER__
        __coveragescanner_install( argv[0] );
    #endif
        // Create the event manager and test controller
        CPPUNIT_NS::TestResult controller;

        // Add a listener that colllects test result
        CPPUNIT_NS::TestResultCollector result;
        controller.addListener( &result );

        // Add a listener that print dots as test run.
        CPPUNIT_NS::BriefTestProgressListener progress;
        controller.addListener( &progress );

        // Add a listener that saves the code coverage information
        CoverageScannerListener coveragescannerlistener;
        controller.addListener( &coveragescannerlistener );

        // Add the top suite to the test runner
        CPPUNIT_NS::TestRunner runner;
        runner.addTest( CPPUNIT_NS::TestFactoryRegistry::getRegistry().makeTest() );
        runner.run( controller );

        return result.wasSuccessful() ? 0 : 1;
    }

In the example, we have done the following steps:

  1. We write a CppUnit listener class which records the code coverage of unit each test after it is completed.

    We want to be able to run the program with and without Squish Coco. Therefore we use in the code the macro __COVERAGESCANNER__ for conditional compilation. The macro is defined in every file that is instrumented by Squish Coco, without the need to #include anything.

    In the listener class, CppUnitListener, we use the following member functions:

    • startTest(): This function is called before each test begins.

      In it, we compute a test name with the information provided by CppUnit and pass it to the Squish Coco library with __coveragescanner_testname().

      We also call the function __coveragescanner_clear(): It empties the internal database and so makes sure that the coverage of the code that was executed before this test is ignored.

    • addFailure(): This function is called after a test fails. It just sets a flag that is used by the other functions.
    • endTest(): This function is called after a test has ended.

      It uses __coveragescanner_teststate() to record the execution status ("PASSED" or "FAILED") and then saves the code coverage report itself with __coveragescanner_save().

  2. We call __coveragescanner_install() in the main() function.
  3. We add this listener in the test manager of CppUnit, the class CPPUNIT_NS::TestResult. In the example above, this is done by the following lines:
                CoverageScannerListener coveragescannerlistener;
                controller.addListener( &coveragescannerlistener );

QTestLib

QTestLib is a unit test framework for Qt. It can easily be adapted to get the code coverage for each unit test.

Proceed as follows:

  1. Call __coveragescanner_install() in the main() function.
  2. Write a subclass of QObject, named TestCoverageObject. It must record the code coverage at the end of every unit test.
  3. Instead of inheriting from QObject, let all your test cases inherit from TestCoverageObject.
  4. The TestCoverageObject class provides its own init() and cleanup() slots, which use the CoverageScanner API to save the code coverage report. If these slots are also declared in the test case classes, it is necessary to rename them to initTest() and cleanupTest().
  5. Compile your project with code code coverage enabled.

TestCoverageObject header:

    #ifndef _TEST_COVERAGE_OBJECT_H
    #define _TEST_COVERAGE_OBJECT_H
    #include <QObject>
    class TestCoverageObject : public QObject
    {
      Q_OBJECT
      public:
        virtual void initTest() {}
        virtual void cleanupTest() {}
      protected slots:
        void init() ;
        void cleanup();
    };
    #endif

TestCoverageObject source:

    #include "testcoverageobject.h"
    #include <QTest>
    #include <QMetaObject>
    #include <QString>

    void TestCoverageObject::init()
    {
    #ifdef __COVERAGESCANNER__
      __coveragescanner_clear();
    #endif
      initTest();
    }

    void TestCoverageObject::cleanup()
    {
      cleanupTest();
    #ifdef __COVERAGESCANNER__
      QString test_name="unittest/";
      test_name+=metaObject()->className();
      test_name+="/";
      test_name+=QTest::currentTestFunction();
      __coveragescanner_testname(test_name.toLatin1());
      if (QTest::currentTestFailed())
        __coveragescanner_teststate("FAILED");
      else
        __coveragescanner_teststate("PASSED") ;
      __coveragescanner_save();
      __coveragescanner_testname("");
    #endif
    }

GoogleTest

GoogleTest

Note: Project page of GoogleTest: http://code.google.com/p/googletest/

is a unit test framework for C++. This environment can easily be adapted to get the code coverage from each unit test.

Simply proceed as follows:

  1. Call __coveragescanner_install() in the main() function.
  2. Write a TestEventListener class which records the code coverage report upon every unit test completion. The listener should set the name (using __coveragescanner_testname()) and clear the instrumentation (using __coveragescanner_clear()) before executing a test item (class member startTest()) to ensure to get only the coverage data of the concerned test. When an test item is executed, the instrumentation and the execution status should be saved (using __coveragescanner_teststate() and __coveragescanner_save()) in the class member endTest(). The class CodeCoverageListener give an implementation example.
  3. Add this listener in the Append function of the GoogleTest listener (function ::testing::UnitTest::GetInstance()->listeners().Append()).
  4. Compile the unit test using CoverageScanner.
    #include <gtest/gtest.h>
    #include <direct.h>
    #include <stdlib.h>

    class CodeCoverageListener : public    ::testing::TestEventListener
    {
      public:
        // Fired before any test activity starts.
        virtual void OnTestProgramStart(const ::testing::UnitTest& unit_test)
        {
        }

        // Fired before each iteration of tests starts.  There may be more than
        // one iteration if GTEST_FLAG(repeat) is set. iteration is the iteration
        // index, starting from 0.
        virtual void OnTestIterationStart(const ::testing::UnitTest& unit_test, int iteration)
        {
        }

        // Fired before environment set-up for each iteration of tests starts.
        virtual void OnEnvironmentsSetUpStart(const ::testing::UnitTest& unit_test)
        {
        }

        // Fired after environment set-up for each iteration of tests ends.
        virtual void OnEnvironmentsSetUpEnd(const ::testing::UnitTest& unit_test)
        {
        }

        // Fired before the test case starts.
        virtual void OnTestCaseStart(const ::testing::TestCase& test_case)
        {
        }

        // Fired before the test starts.
        virtual void OnTestStart(const ::testing::TestInfo& test_info)
        {
    #ifdef __COVERAGESCANNER__
          __coveragescanner_clear();
          std::string test_name=
            std::string(test_info.test_case_name())
            + '/'
            + std::string(test_info.name());
          __coveragescanner_testname(test_name.c_str());
    #endif
        }

        // Fired after a failed assertion or a SUCCESS().
        virtual void OnTestPartResult(const ::testing::TestPartResult& test_part_result)
        {
        }

        // Fired after the test ends.
        virtual void OnTestEnd(const ::testing::TestInfo& test_info)
        {
    #ifdef __COVERAGESCANNER__
          __coveragescanner_teststate("UNKNOWN");
          if (test_info.result())
          {
            if (test_info.result()->Passed())
              __coveragescanner_teststate("PASSED");
            if (test_info.result()->Failed())
              __coveragescanner_teststate("FAILED");
          }
          __coveragescanner_save();
    #endif
        }

        // Fired after the test case ends.
        virtual void OnTestCaseEnd(const ::testing::TestCase& test_case)
        {
        }

        // Fired before environment tear-down for each iteration of tests starts.
        virtual void OnEnvironmentsTearDownStart(const ::testing::UnitTest& unit_test)
        {
        }

        // Fired after environment tear-down for each iteration of tests ends.
        virtual void OnEnvironmentsTearDownEnd(const ::testing::UnitTest& unit_test)
        {
        }

        // Fired after each iteration of tests finishes.
        virtual void OnTestIterationEnd(const ::testing::UnitTest& unit_test, int iteration)
        {
        }

        // Fired after all test activities have ended.
        virtual void OnTestProgramEnd(const ::testing::UnitTest& unit_test)
        {
        }
    } ;

    int main(int argc, char **argv){

      //initialize CoverageScanner library
    #ifdef __COVERAGESCANNER__
      __coveragescanner_install(argv[0]);
    #endif

      ::testing::FLAGS_gtest_output = "xml";
      ::testing::UnitTest::GetInstance()->listeners().Append(new CodeCoverageListener);

      ::testing::InitGoogleTest(&argc, argv);   //init google test framework

      return RUN_ALL_TESTS();   //run all tests which are listed in this project
    }

CxxTest

CxxTest

Note: Project page of CxxTest: http://cxxtest.com is a unit test framework for C++. This environment can easily be adapted to get the code coverage from each unit test.

Proceed as follows:

  1. Call __coveragescanner_install() in the main() function.
  2. Create a CxxTest TestListener class CoverageScannerListener by subclassing an existing listener.

    In the example below this is ErrorPrinter. It will record the code coverage report upon every unit test completion. To ensure to get only the coverage data of the concerned test, the listener should set the name with __coveragescanner_testname() and clear the instrumentation with __coveragescanner_clear() before executing a test item (class member enterTest()).

    When an test item is executed, the instrumentation and the execution status should be saved in the member function leaveTest() with __coveragescanner_teststate() and __coveragescanner_save().

    Finally, all test failure members of this class must be reimplemented to record the test failures.

  3. In the main() function call the run() function of CoverageScannerListener instead of CxxTest::ErrorPrinter().run().
  4. Compile the unit test with CoverageScanner activated.

Example:

    #include <cxxtest/TestRunner.h>
    #include <cxxtest/TestListener.h>
    #include <cxxtest/TestTracker.h>
    #include <cxxtest/ValueTraits.h>
    #include <cxxtest/ValueTraits.h>
    #include <cxxtest/ErrorPrinter.h>

    class CoverageScannerListener : public CxxTest::ErrorPrinter
    {
      public:
        CoverageScannerListener(std::ostream &o=std::cout,
                            const char *preLine = ":",
                            const char *postLine = "")
        : CxxTest::ErrorPrinter( o, preLine , postLine ) {}

        int run()
        {
          return CxxTest::ErrorPrinter::run();
        }

        void enterTest( const CxxTest::TestDescription & desc)
        {
          test_passed=true;

    #ifdef __COVERAGESCANNER__
          // Adjust the name of the test to display the tests
          // in a tree view in CoverageBrowser
          std::string testname="CxxTest/";
          testname += desc.suiteName();
          testname += "/";
          testname += desc.testName();

          // Reset the code coverage data to get only the code coverage
          // of the actual unit test.
          __coveragescanner_clear();
          __coveragescanner_testname(testname.c_str());
    #endif
          return CxxTest::ErrorPrinter::enterTest( desc );
        }

        void leaveTest( const CxxTest::TestDescription & desc)
        {
    #ifdef __COVERAGESCANNER__
          // Record the execution state in the coverage report
          if (test_passed)
            __coveragescanner_teststate("PASSED");
          else
            __coveragescanner_teststate("FAILED");

          // Save the code coverage report of the unit test
          __coveragescanner_save();
    #endif
          return CxxTest::ErrorPrinter::leaveTest( desc );
        }

        void failedTest(const char *file, int line, const char *expression)
        { // Only record that the test fails
          test_passed=false;
          return CxxTest::ErrorPrinter::failedTest( file, line, expression );
        }

        void failedAssert(const char *file, int line, const char *expression)
        { // Only record that the test fails
          test_passed = false;
          return CxxTest::ErrorPrinter::failedAssert( file, line, expression );
        }

        void failedAssertEquals(const char *file, int line,
                            const char *xStr, const char *yStr,
                            const char *x, const char *y)
        { // Only record that the test fails
          test_passed=false;
          return CxxTest::ErrorPrinter::failedAssertEquals( file, line,
                                                        xStr, yStr,
                                                        x, y );
        }

        void failedAssertSameData(const char *file, int line,
                              const char *xStr, const char *yStr, const char *sizeStr,
                              const void *x, const void *y, unsigned size)
        { // Only record that the test fails
          test_passed=false;
          return CxxTest::ErrorPrinter::failedAssertSameData( file, line,
                                                          xStr, yStr, sizeStr,
                                                          x, y, size );
        }

        void failedAssertDelta(const char *file, int line,
                           const char *xStr, const char *yStr, const char *dStr,
                           const char *x, const char *y, const char *d)
        { // Only record that the test fails
          test_passed=false;
          return CxxTest::ErrorPrinter::failedAssertDelta( file, line,
                                                       xStr, yStr, dStr,
                                                       x, y, d );
        }

        void failedAssertDiffers(const char *file, int line,
                             const char *xStr, const char *yStr, const char *value)
        { // Only record that the test fails
          test_passed=false;
          return CxxTest::ErrorPrinter::failedAssertDiffers(file, line,
                                                        xStr, yStr, value );
        }

        void failedAssertLessThan(const char *file, int line,
                              const char *xStr, const char *yStr,
                              const char *x, const char *y)
        { // Only record that the test fails
          test_passed=false;
          return CxxTest::ErrorPrinter::failedAssertLessThan(file, line,
                                                         xStr, yStr,
                                                         x, y );
        }

        void failedAssertLessThanEquals(const char *file, int line,
                                    const char *xStr, const char *yStr,
                                    const char *x, const char *y)
        { // Only record that the test fails
          test_passed=false;
          return CxxTest::ErrorPrinter::failedAssertLessThanEquals( file, line,
                                                                xStr, yStr,
                                                                x, y );
        }

        void failedAssertRelation(const char *file, int line, const char *relation,
                              const char *xStr, const char *yStr,
                              const char *x, const char *y)
        { // Only record that the test fails
          test_passed=false;
          return CxxTest::ErrorPrinter::failedAssertRelation( file, line, relation,
                                                          xStr, yStr,
                                                          x, y);
        }

        void failedAssertPredicate(const char *file, int line, const char *predicate,
                               const char *xStr, const char *x )
        { // Only record that the test fails
          test_passed=false;
          return CxxTest::ErrorPrinter::failedAssertPredicate( file, line, predicate,
                                                           xStr, x);
        }

        void failedAssertThrows(const char *file, int line, const char *expression,
                            const char *type, bool otherThrown)
        { // Only record that the test fails
          test_passed=false;
          return CxxTest::ErrorPrinter::failedAssertThrows( file, line, expression,
                                                        type, otherThrown );
        }

        void failedAssertThrowsNot(const char *file, int line, const char *expression)
        { // Only record that the test fails
          test_passed=false;
          return CxxTest::ErrorPrinter::failedAssertThrowsNot( file, line, expression );
        }

      private:
        bool test_passed;
    };

    int main()
    {
    #ifdef __COVERAGESCANNER__
      __coveragescanner_install(argv[0]);
    #endif

      // Use "CoverageScannerListener().run()" instead of "CxxTest::ErrorPrinter().run()"
      return CoverageScannerListener().run();
    }

boost::test

boost::test

Note: Project page of Boost::Test: http://www.boost.org/doc/libs/1_64_0/libs/test

is a unit test framework for C++ which is part of the boost libraries.

Proceed as follows:

  1. Implement a TestObserver, which derives from 'boost::unit_test_framework::test_observer' and store as "BoostTestObserver.hpp".
  2. Also implement (within the same file) a 'boost::test::fixture' and define it to be executed in the beginning and in the end of each executed boost::test by: BOOST_GLOBAL_FIXTURE(FixtureName). The file then should look something similar to this:
                #ifndef COVERAGE_TEST_OBSERVER_INCLUDED
                #define COVERAGE_TEST_OBSERVER_INCLUDED
    
                #include <boost/test/framework.hpp>
                #include <boost/test/tree/observer.hpp>
                #include <boost/test/included/unit_test.hpp>
    
                #ifdef __COVERAGESCANNER__
    
                class BoostTestObserver
                    : public boost::unit_test_framework::test_observer
                {
                public:
                    BoostTestObserver(const char* moduleName)
                        : test_observer()
                        , m_testSuiteName(moduleName)
                        , m_isAnyTestcaseOpen(false)
                    {}
    
                    ~BoostTestObserver()
                    {
                        if (m_isAnyTestcaseOpen)
                            close_current_testcase();
                    }
    
                ///////////////////////////////////////////////////////////////////////////////
                // test suite related events:
    
                    virtual int priority()
                    {
                        return 1;
                    }
    
                    virtual void test_finish()
                    {
                        if (m_isAnyTestcaseOpen) {
                            close_current_testcase();
                        }
                    }
    
                    virtual void test_aborted()
                    {
                        if (m_isAnyTestcaseOpen) {
                            close_current_testcase();
                        }
                    }
    
                    const char* get_testsuite_name()
                    {
                        return m_testSuiteName.data();
                    }
    
                    // called in the very beginning of the test suite:
                    virtual void test_start(
                        boost::unit_test_framework::counter_t  NumberOfTestCases)
                    {
                        m_currentTestNumber = -1;
                        m_numberOfTestCases = NumberOfTestCases;
                        __coveragescanner_install(m_testSuiteName.c_str());
                    }
    
                ///////////////////////////////////////////////////////////////////////////////
                // test case related events:
    
                    // called before each testcase's start
                    virtual void test_unit_start(
                        boost::unit_test_framework::test_unit const &unit)
                    {
                        // skip any calls which may being triggered by the 'TestWrap' fixture
                        if(unit.full_name().find('/') == -1) {
                            return;
                        }
    
            if (++m_currentTestNumber)
                            close_current_testcase();
    
            m_currentCaseHasFailed = false;
                        __coveragescanner_clear();
                        m_currentTestName.assign( unit.full_name().c_str());
                        __coveragescanner_testname(m_currentTestName.c_str());
                        m_isAnyTestcaseOpen = true;
                    }
    
                    // should be called after each testcase has finished...
                    // but which definitveley, for unknown reason is NOT
                    virtual void test_unit_finish(
                        boost::unit_test_framework::test_unit const &unit,
                        unsigned number)
                    {
                        close_current_testcase();
                    }
    
        // workaround for missing 'test_unit_finish' event being not triggered:
                    void close_current_testcase()
                    {
                        if (m_isAnyTestcaseOpen) {
                            __coveragescanner_teststate(m_currentCaseHasFailed
                                                       ? "FAILED"
                                                       : "PASSED");
                            __coveragescanner_save();
                            m_currentTestName.append("_invalid");
                            m_isAnyTestcaseOpen = false;
                        }
                    }
    
                    // called once on each FAILED or PASSED result
                    virtual void assertion_result(boost::unit_test::assertion_result result)
                    {
                        if (result == boost::unit_test::AR_FAILED)
                            m_currentCaseHasFailed = true;
                    }
    
                protected: // Depricated boost event:
                    virtual void assertion_result(bool result)
                    {
                        if (!result) m_currentCaseHasFailed = true;
                    }
    
                ///////////////////////////////////////////////////////////////////////////////
                //  member variables:
    
                    std::string m_testSuiteName;
                    std::string m_currentTestName;
                    int         m_currentTestNumber;
                    int         m_numberOfTestCases;
                    bool        m_verboseLogging;
                    bool        m_currentCaseHasFailed;
                    bool        m_isAnyTestcaseOpen;
                };
    
                //////////////////
                // boost fixture:
                // will be called in the very beginning and in the end of executing test
                ///////////////////////////////////////////////////////////////////////////////
    
                struct TestWrap
                {
                    BoostTestObserver observer;
    
        TestWrap()
                    : BoostTestObserver(
                        boost::unit_test::framework::master_test_suite().argv[0])
                    {
                        boost::unit_test::framework::register_observer(observer);
                    }
    
                   ~TestWrap()
                    {
                        observer.close_current_testcase();
                        boost::unit_test::framework::deregister_observer(observer);
                    }
                };
    
                ///////////////////////////////////////////////////////////////////////////////
    
                BOOST_GLOBAL_FIXTURE(TestWrap);
    
                  #endif
                #endif
  3. If it is included by any of your already existing 'boost::test' unit test .cpp files (must be done right after the in 'boost::test' mandatory BOOST_TEST_MODULE definition), It is being instantiated and initialized by the boost::test automatically.
  4. The 'boost::test' then will trigger a __coveragescanner_install() call just in the beginning, and __coveragescanner_clear() and __coveragescanner_testname() calls for each test case section which is being executed. Also it triggers __coveragescanner_teststate() and __coveragescanner_save() on each test case's end.
  5. Additionally here is a small example which shows how to include this into your already existing 'boost::test' modules:
                // define the name for yourBoost::Test module.
                // (*this must be done BEFORE including BoostTestObserver.hpp)
                #define BOOST_TEST_MODULE MyBoostTest
                #include <boost/test/framework.hpp>
                #include <boost/test/tree/observer.hpp>
                #include <boost/test/included/unit_test.hpp>
    
                // any of your already existing Boost::Test modules can be made
                // compatible to Coco by including "BoostTestObserver.hpp"
                // just after the module's BOOST_TEST_MODULE definition.
                #include "BoostTestObserver.hpp"
    
                // declare a boost test suite:
                // (when "BoostTestObserver.hpp" is included, this will
                // automatically instanciate a BoostTestObserver then.)
                BOOST_AUTO_TEST_SUITE(BOOST_TEST_MODULE)
    
                    // declare first test case:
                    BOOST_AUTO_TEST_CASE(testA)
                    {
                        // code for testcase A
                    }
    
                     //...  declare any other testcases
    
                    // last testcase:
                    BOOST_AUTO_TEST_CASE(testX)
                    {//...
                    }
    
                // in the end of the test, close the suite's scope
                // (which at least also will unload our BoostTestObserver)
                BOOST_AUTO_TEST_SUITE_END()
  6. Compile the boost test with CoverageScanner activated.

xUnit

When Squish Coco is used with xUnit,

Note: https://xunit.github.io it is possible to save the coverage for each single test. To do this, we define collections and hooks that are executed before and after each test.

The tests themself need not to be instrumented, but the code that is tested must be built with CoverageScanner. A small adaptation of the xUnit test suite must be made. These changes work even if the library that is tested is not instrumented, but in this case no code coverage information will be generated.

Proceed as follows:

  1. Add to your xUnit test library the following code:

    It provides a set of functions to handle the coverage information and to save the execution report (the .csexe file).

                using System;
                using System.Collections.Generic;
                using System.Text;
                using Xunit;
                using System.Reflection;
                using Xunit.Sdk;
                using System.Diagnostics;
                using Xunit.Abstractions;
    
                class Coverage
                {
                    public enum State { PASSED, FAILED, CHECK_MANUALLY, UNKNOWN };
    
                    public static void CoverageCleanup()
                    {
                        AppDomain MyDomain = AppDomain.CurrentDomain;
                        Assembly[] AssembliesLoaded = MyDomain.GetAssemblies();
    
                        foreach (Assembly a in AssembliesLoaded)
                        {
                            Type coco = a.GetType("CoverageScanner");
                            if (coco != null)
                            {
                                // clear all coverage information to only get the code coverage of the current executed unit test
                                coco.InvokeMember("__coveragescanner_clear", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, null);
                                coco.InvokeMember("__coveragescanner_clear_html_comment", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, null);
                            }
                        }
                    }
    
                    public static void CoverageTextLog(String text, bool bold, bool italic)
                    {
                        String style_begin = "";
                        String style_end = "";
                        if (italic)
                        {
                            style_begin += "<I>";
                            style_end += "</I>";
                        }
                        if (bold)
                        {
                            style_begin += "<B>";
                            style_end += "</B>";
                        }
    
                        String comment = "<HTML><BODY><TT>"
                                        + style_begin
                                        + text.Replace("&", "&amp;")
                                                .Replace("<", "&lt;")
                                                .Replace(">", "&gt;")
                                                .Replace("\"", "&quot;")
                                                .Replace("'", "&apos;")
                                                .Replace("\n", "<BR>")
                                                .Replace("\r", "")
                                        + style_end
                                        + "</TT></BODY></HTML>"
                                        ;
                        CoverageHtmlLog(comment);
                    }
    
                    public static void CoverageHtmlLog(String comment)
                    {
                        AppDomain MyDomain = AppDomain.CurrentDomain;
                        Assembly[] AssembliesLoaded = MyDomain.GetAssemblies();
    
                        foreach (Assembly a in AssembliesLoaded)
                        {
                            Type coco = a.GetType("CoverageScanner");
                            if (coco != null)
                            {
                                coco.InvokeMember("__coveragescanner_add_html_comment", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, new object[] { comment });
                            }
                        }
                    }
    
                    public static void CoverageRecord(String csexe_filename, String testName, State result)
                    {
                        if (csexe_filename == "")
                            return;
    
                        AppDomain MyDomain = AppDomain.CurrentDomain;
                        Assembly[] AssembliesLoaded = MyDomain.GetAssemblies();
    
                        foreach (Assembly a in AssembliesLoaded)
                        {
                            Type coco = a.GetType("CoverageScanner");
                            if (coco != null)
                            {
                                String name = testName;
                                String state = "";
    
                                switch (result)
                                {
                                    case State.FAILED:
                                        state = ("FAILED");
                                        break;
                                    case State.CHECK_MANUALLY:
                                        state = ("CHECK_MANUALLY");
                                        break;
                                    case State.PASSED:
                                        state = ("PASSED");
                                        break;
                                }
    
                                name = name.Replace('.', '/');
    
                                // set the execution state: PASSES, FAILED or CHECK_MANUALLY
                                coco.InvokeMember("__coveragescanner_teststate", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, new object[] { state });
    
                                if (name.Length > 0)
                                    // Test name: Namespace/Class/Testfunction
                                    coco.InvokeMember("__coveragescanner_testname", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, new object[] { name });
    
                                // File name is <classname>.csexe
                                coco.InvokeMember("__coveragescanner_filename", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, new object[] { csexe_filename });
                                // saves the code coverage data
                                coco.InvokeMember("__coveragescanner_save", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, null);
                                // clear all coverage information to only get the code coverage of the current executed unit test
                                coco.InvokeMember("__coveragescanner_clear", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, null);
                                coco.InvokeMember("__coveragescanner_clear_html_comment", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, null);
                            }
                        }
                    }
                }
  2. Add to your xUnit test library the following code:

    It contains a definition of a xUnit collection which saves the coverage after each unit test has been executed. If a collection is already present in the code, it is necessary to adapt the existing one with this file as guideline.

                using System;
                using System.Collections.Generic;
                using System.Text;
                using Xunit;
                using System.Reflection;
                using Xunit.Sdk;
                using System.Diagnostics;
                using Xunit.Abstractions;
    
                [AttributeUsage(AttributeTargets.Class | AttributeTargets.Method, AllowMultiple = false, Inherited = true)]
                public class SquishCocoHooks : BeforeAfterTestAttribute
                {
                    public override void Before(MethodInfo methodUnderTest)
                    {
                        Coverage.CoverageCleanup();
                    }
    
                    public override void After(MethodInfo methodUnderTest)
                    {
                        string base_name = "xUnit";
                        string method_name = methodUnderTest.Name;
                        string class_name = methodUnderTest.ReflectedType.FullName;
    
                        Coverage.CoverageRecord("coverage.csexe", base_name + "/" + class_name + "/" + method_name, Coverage.State.UNKNOWN);
                    }
                }
    
                public class CustomFixture : IDisposable
                {
                    public CustomFixture()
                    {
                        // Some initialization
                    }
    
                    public void Dispose()
                    {
                        // Some cleanup
                    }
                }
    
                [CollectionDefinition("SquishCoco")]
                [SquishCocoHooks]
                public class SquishCocoDefinition :  ICollectionFixture<CustomFixture>
                {
                }
  3. Activate the collection for each test class. To do it, just add "[Collection("SquishCoco")]" to the definition of each test class. For example:
                [Collection("SquishCoco")]
                public class Class2Tests
                {
                    [Fact()]
                    public void dummyTest()
                    {
                        Assert.True(false, "Dummy test");
                    }
                }

After running the tests, a file coverage.csexe will be generated; it can then be imported into the corresponding coverage database (.csmes file). With CoverageBrowser, it is then possible to analyze the coverage of each test separately.

NUnit

Squish Coco provides an addin for NUnit

Note: http://nunit.org/ version 2.4.4 and above as sample. To install it proceed as followings:

  1. Build NUnitSquishCoco.dll using the Microsoft® Visual Studio® project NUnitSquishCoco.vsproj provided in the sample directory.
  2. Copy NUnitSquishCoco.dll to the addins folder located in the bin folder where NUnit.exe can be found.
  3. Start NUnit.exe and verify that the addin "NUnit Squish Coco" is loaded.

Once installed, as soon as NUnit's test driver is executing a C# or C++ managed unit test test.dll, it generates automatically a code coverage execution report test.dll.csexe automatically if test.dll is instrumented with Squish Coco. The code coverage information is organized into a tree containing the coverage and the execution status for each single unit test. The execution report can be then imported into the application's instrumentation database with CoverageBrowser or cmcsexeimport.

Catch2

Catch2

Note: Project page of Catch2: https://github.com/catchorg/Catch2

is a unit test framework for C++ which can be easily adapted to get the code coverage from each unit test section.

Complete code example:

    #define CATCH_CONFIG_RUNNER
    #include "catch.hpp"
    #include <string>
    #include <deque>

    class CoverageScannerListener : public Catch::TestEventListenerBase
    {
    public:
        using TestEventListenerBase::TestEventListenerBase;

        virtual void sectionStarting( Catch::SectionInfo const& sectionInfo ) override
        {
            m_testNameHierarchy.push_back( sectionInfo.name );
    #ifdef __COVERAGESCANNER__
            // Adjusting the name of the test for the CoverageBrowser
            std::string testname = implodeTestNameHierarchy();
            // Reset the code coverage data to get only the code coverage
            // of the actual unit test.
            __coveragescanner_clear();
            __coveragescanner_testname( testname.c_str() );
    #endif
        }

        virtual void sectionEnded( Catch::SectionStats const& sectionStats ) override
        {
            m_testNameHierarchy.pop_back();
    #ifdef __COVERAGESCANNER__
            // Recording the execution state in the coverage report
            if ( sectionStats.assertions.allPassed() )
                __coveragescanner_teststate( "PASSED" );
            else
                __coveragescanner_teststate( "FAILED" );
            // Saving the code coverage report of the unit test
            __coveragescanner_save();
            __coveragescanner_testname( "" );
    #endif
        }

    private:
        std::deque<std::string> m_testNameHierarchy;

        /* Helper method which generates a hierarchical name string usable by the coveragebrowser. */
        std::string implodeTestNameHierarchy() {
            std::string fullTestName ( "Catch2" );
            std::deque<std::string>::const_iterator it = m_testNameHierarchy.cbegin();
            while ( it != m_testNameHierarchy.cend() ) {
                fullTestName += "/" + *it;
                it++;
            }
            return fullTestName;
        }
    };

    // Register the coveragescanner listener with Catch2
    CATCH_REGISTER_LISTENER( CoverageScannerListener )

    int main( int argc, char* argv[] )
    {
    #ifdef __COVERAGESCANNER__
        __coveragescanner_install( argv[0] );
    #endif
        return Catch::Session().run( argc, argv );
    }

Follow these steps:

  1. Call __coveragescanner_install() and Catch::Session.run() in the main() function. Catch2 provides a predefined main(), which needs to be disregarded by defining only CATCH_CONFIG_RUNNER before including the Catch2 header:
                #define CATCH_CONFIG_RUNNER
                    #include "catch.hpp"
  2. Create a Catch2 listener class - inheriting Catch::TestEventListenerBase - which records the code coverage of each section after it is completed. (Note: Testcases also count as sections in Catch2. There is no need to listen to testcase events specifically.)

    In the created listener class (CoverageScannerListener) we use the following member functions:

    • sectionStarting(): This function is called before each testcase and section begins.

      In it, we compute a test name with the information provided by Catch2 and pass it to the Squish Coco library with __coveragescanner_testname().

      We also call the function __coveragescanner_clear(): It empties the internal database and so makes sure that the coverage of the code that was executed before this test is ignored.

    • sectionEnded(): This function is called after a testcase or section has ended.

      It uses __coveragescanner_teststate() to record the execution status ("PASSED" or "FAILED") and then saves the code coverage report itself with __coveragescanner_save().

  3. We add this listener by adding the following Catch2 macro:
                CATCH_REGISTER_LISTENER( CoverageScannerListener )

Note: Remember to exclude any test framework sources from instrumentation (see Beyond the minimal instrumentation)!

Squish

It is easily possible to run the GUI testing tool Squish together with Squish Coco in order to get the C/C++coverage of a Squish test suite. A more in-depth analysis is then possible, correlating each test case (and its results) with the respective coverage information. This is especially true since Squish Coco features the comparison of individual executions, including the calculation of an optimal order.

General Approach

The approach depicted below is based on the possibility to apply information about the name, the result and a free-form comment to each execution (stored in .csexe files). See Chapter (see Execution comment, name and status) for the details.

As an example we'll use Squish for Qt's addressbook on a Unix-based system and a JavaScript test script.

  1. First we'll initialize the execution data with the name of the Squish test case that is being run.
                function main()
                {
                    var currentAUT = currentApplicationContext();
                    var execution = currentAUT.cwd + "\\" + currentAUT.name + ".exe.csexe"
                    var testCase = squishinfo.testCase;
                    var testExecutionName = testCase.substr(testCase.lastIndexOf('/') + 1);
                    var file = File.open(execution, "a");
                    file.write("*" + testExecutionName + "\n");
                    file.close();
                    var ctx = startApplication("addressbook");
                ...
  2. Insert the main test script at this point
  3. After the main test script we'll log the result of the test for the coverage tool:
                ...
                    // wait until AUT shutdown
                    while (ctx.isRunning) {
                        snooze(1); // increase time if not enough to dump coverage data
                    }
    
                    // test result summary and status
                    var positive = test.resultCount("passes");
                    var negative = test.resultCount("fails") + test.resultCount("errors") + test.resultCount("fatals");
                    var msg = "TEST RESULTS - Passed: " + positive +  " | " + "Failed/Errored/Fatal: " + negative;
                    var status = negative == 0 ? "PASSED" : "FAILED";
                    var file = File.open(execution, "a");
                    file.write("<html><body>" + msg + "</body></html>\n");
                    file.write("!" + status + "\n")
                    file.close();
                }

When you execute the scripts containing these steps, the Squish Coco Execution Report loads with the test case name, status and execution summary in the execution details and comments.

Simplified for Reuse

  1. Create a file called squishCocoLogging.js in Test Suite Resources with the following functions:
                function getExecutionPath() {
                    var currentAUT = currentApplicationContext();
                    var execution = currentAUT.cwd + "\\" + currentAUT.name + ".exe.csexe"
                    return execution;
                }
    
                function logTestNameToCocoReport(currentTestCase, execution) {
                    var testExecutionName = currentTestCase.substr(currentTestCase.lastIndexOf('\\') + 1);
                    var file = File.open(execution, "a");
                    file.write("*" + testExecutionName + "\n");
                    file.close();
                }
    
                function logTestResultsToCocoReport(testInfo, execution){
    
                    var currentAUT = currentApplicationContext();
    
                    // wait until AUT shuts down
                    while (currentAUT.isRunning)
                      snooze(5);
    
                    // collect test result summary and status
                    var positive = testInfo.resultCount("passes");
                    var negative = testInfo.resultCount("fails") + testInfo.resultCount("errors") +
                        testInfo.resultCount("fatals");
                    var msg = "TEST RESULTS - Passed: " + positive +  " | " + "Failed/Errored/Fatal: " + negative;
                    var status = negative == 0 ? "PASSED" : "FAILED";
    
                    // output results and status to Coco execution report file
                    var file = File.open(execution, "a");
                    file.write("<html><body>" + msg + "</body></html>\n");
                    file.write("!" + status + "\n")
                    file.close();
                }

    A Python version of this code is:

                import re
    
                def getExecutionPath():
                    currentAUT = currentApplicationContext()
                    execution = "%(currAUTPath)s\\%(currAUTName)s.exe.csexe" % {"currAUTPath" : currentAUT.cwd, "currAUTName" : currentAUT.name}
                    return execution
    
                def logTestNameToCocoReport(currentTestCase, execution):
                    testExecutionName = re.search(r'[^\\]\w*$', currentTestCase)
                    testExecutionName = testExecutionName.group(0)
                    file = open(execution, "a")
                    file.write("*" + testExecutionName + "\n")
                    file.close()
    
                def logTestResultsToCocoReport(testInfo, execution):
                    currentAUT = currentApplicationContext()
                    # wait until AUT shuts down
                    while (currentAUT.isRunning):
                        snooze(5)
    
                    # collect test result summary and status
                    positive = testInfo.resultCount("passes")
                    negative = testInfo.resultCount("fails") + testInfo.resultCount("errors") + testInfo.resultCount("fatals")
                    msg = "TEST RESULTS - Passed: %(positive)s  |  Failed/Errored/Fatal: %(negative)s" % {'positive': positive, 'negative': negative}
                    if negative == 0:
                        status = "PASSED"
                    else:
                        status = "FAILED"
    
                    # output results and status to Coco execution report file
                    file = open(execution, "a")
                    file.write("<html><body>" + msg + "</body></html>\n")
                    file.write("!" + status + "\n")
                    file.close()
  2. Add the following function calls after startApplication() in the main test script:
                execution = getExecutionPath();
    
                logTestNameToCocoReport(squishinfo.testCase, execution);

    In Python:

                execution = getExecutionPath()
    
                logTestNameToCocoReport(squishinfo.testCase, execution)
  3. At the end of your script, after closing the AUT (for example with steps clicking File > Exit), call the following function:
                logTestResultsToCocoReport(test, execution);

    In Python:

                logTestResultsToCocoReport(test, execution)
  4. In the event your AUT closes unexpectedly, or a script error occurs, incorporating a try, catch, finally ensures your results still output to the Coco report file.

Your main test script should be similar to the following:

    source(findFile("scripts","squishCocoLogging.JS"))

    function main()
    {
        startApplication("addressbook");
        execution = getExecutionPath();
        logTestNameToCocoReport(squishinfo.testCase, execution);

        try {
            // body of script
        }
        catch(e) {
           test.fail('An unexpected error occurred', e.message)
        }
        finally {
           logTestResultsToCocoReport(test, execution)
        }
    }

Python version:

    source(findFile("scripts","squishCocoLogging.py"))

    def main():
        startApplication("addressbook")
        execution = getExecutionPath()
        logTestNameToCocoReport(squishinfo.testCase, execution)

        try:
            try:
                # body of script
            except Exception, e:
                test.fail("test failed: ", e)
        finally:
            logTestResultsToCocoReport(test,execution)

© 2022 The Qt Company Ltd. Documentation contributions included herein are the copyrights of their respective owners. The documentation provided herein is licensed under the terms of the GNU Free Documentation License version 1.3 as published by the Free Software Foundation. Qt and respective logos are trademarks of The Qt Company Ltd. in Finland and/or other countries worldwide. All other trademarks are property of their respective owners.