3. Test Suites

A test suite is a collection of tests. QMTest can run an entire test suite at once, so by grouping tests together in a test suite, you make it easier to run a number of tests at once. A single test can be a member of more than one test suite. A test suite can contain other test suites; the total set of tests in a test suite includes both those tests included directly and those tests included as part of another test suite. Every test suite has a name, following the same conventions given above for tests.

One use of test suites is to provide groups of tests that are run in different situations. For example, the nightly test suite might consist of those tests that should be run automatically every night, while the checkin test suite might consist of those tests that have to pass before any changes are made to the target application.

3.1. Implicit Test Suites

Section 1, “Tests” explains how you may arrange tests in a tree hierarchy, using a period (.) as the path separator in test names. QMTest defines an implicit test suite for each directory. The name of these implicit test suites is the same as the name of the directory. The implicit test suite corresponding to a directory contains all tests in that directory or its subdirectories.

Let us create some more tests, but this time within their respective subdirectories:

> qmtest create --id=dir1.one -a expression='True' test python.ExecTest
> qmtest create --id=dir1.two -a expression='False' test python.ExecTest
> qmtest create --id=dir2.one -a expression='True' test python.ExecTest
> qmtest create --id=dir2.dir3.one -a expression='False' test python.ExecTest
> qmtest create --id=dir2.dir3.two -a expression='False' test python.ExecTest
      

This will create five new tests, together with three directories:

> qmtest ls -lR
directory                 dir1
test      python.ExecTest dir1.one
test      python.ExecTest dir1.two
directory                 dir2
directory                 dir2.dir3
test      python.ExecTest dir2.dir3.one
test      python.ExecTest dir2.dir3.two
test      python.ExecTest dir2.one
      

These directories are implicit suites, i.e. it is possible to only address tests within them:

> qmtest ls -lR dir2
directory                 dir2
directory                 dir2.dir3
test      python.ExecTest dir2.dir3.one
test      python.ExecTest dir2.dir3.two
test      python.ExecTest dir2.one
      

The suite named "." (a single period) is the implicit test suite corresponding to the root directory in the test database. This suite therefore contains all tests in the database. For example, the command

> qmtest run .
    

is equivalent to:

> qmtest run
    

Both commands run all tests in the database.

3.2. Explicit Test Suites

Explicit test suites are a means to refer to a set of tests for the purpose of executing them together, no matter how they are organized in the test database. For example, you may want to run a subset of all tests nightly. To do this, you create a suite, listing all tests (as well as contained suites) explicitely:

> qmtest create --id=nightly suite explicit_suite.ExplicitSuite(test_ids="['dir1.one', 'dir2.one']",
                                                                suite_ids="['dir2.dir3']")
> qmtest ls -l
directory                              dir1
directory                              dir2
suite     explicit_suite.ExplicitSuite nightly
> qmtest ls -l nightly
test      python.ExecTest dir1.one
directory                 dir2.dir3
test      python.ExecTest dir2.one