Table of Contents
This section presents the concepts that underlie QMTest's design. By understanding these concepts, you will be able to better understand how QMTest works. In addition, you will find it easier to extend QMTest to new application domains.
The central principle underlying the design of QMTest is that the problem of testing can be divided into a domain-dependent problem and a domain-independent problem. The domain-dependent problem is deciding what to test and how to test it. For example, should a database be tested by performing unit tests on the C code that makes up the database, or by performing integration tests using SQL queries? How should the output of a query asking for a set of records be compared to expected output? Does the order in which records are presented matter? These are questions that only someone who understands the application domain can answer.
The domain-independent part of the problem is managing the creation of tests, executing the tests, and displaying the results for users. For example, how does a user create a new test? How are tests stored? Should failing tests be reported to the user, even if the failure was expected? These questions are independent of the application domain; they are just as relevant for compiler tests as they are for database tests.
QMTest is intended to solve the domain-independent part of the problem and to offer a convenient, powerful, and flexible interface for solving the domain-dependent problem. QMTest is both a complete application, in that it can be used “out of the box” to handle many testing domains, and infrastructure, in that it can be extended to handle other domains.
Throughout this chapter we will use the qmtest application with a variety of parameters and options. For a full description of the qmtest please refer to the command-line reference.
The following commands create a simple test database in the current working directory. This test database will be used throughout the following sections of this tutorial.
>mkdir tdb>cd tdb>qmtest create-tdb
A test checks for the correct behavior of the target application. What constitutes correct behavior will vary depending on the application domain. For example, correct behavior for a database might mean that it is able to retrieve records correctly while correct behavior for a compiler might mean that it generates correct object code from input source code.
Every test has a name that uniquely identifies the test, within a
given test database. Test
names must be composed entirely of lowercase letters, numbers, the
“_” character, and the “.” character. You can
think of test names like file names. The “.” character takes
the place of “/”; it allows you to place a test in a
particular directory. For example, the test name
a.b.c names a test named c in
the directory a.b. The directory
a.b is a subdirectory of the directory
a.
Every test is an instance of some test class. The test class
dictates how the test is run, what constitutes success, and what
constitutes failure. For example, the
command.ExecTest class that comes with
QMTest executes the target application and
looks at its output. The test passes if the actual output exactly matches
the expected output.
The arguments to the test parameterize the test; they are what make
two instances of the same test class different from each other. For
example, the arguments to command.ExecTest indicate
which application to run, what command-line arguments to provide, and what
output is expected.
The python.ExecTest class is similar to
command.ExecTest, but, instead of executing a
command using the system shell, it evaluates an expression in the Python
programming language. The test passes if (and only if) the expression is true.
In Python, the expressions True and False are
literals.
The following creates two trivial tests, python_pass
and python_fail:
>qmtest create --id=python_pass -a expression='True' test python.ExecTest>qmtest create --id=python_fail -a expression='False' test python.ExecTest
The first test will always pass while the second will always fail.
The qmtest ls command will show the content of the test database:
> qmtest ls
python_fail
python_pass
Similar to the Unix ls command, the -l option
can be used to provide a detailed listing, with kind ("test", "resource", or "suite"),
extension class, and id:
> qmtest ls -l
test python.ExecTest python_fail
test python.ExecTest python_pass
To run one or more tests, use the qmtest run command:
>qmtest run
Each invocation of the qmtest run command is a single test run, and produces a single set of test results and statistics. Specify as arguments the names of tests and test suites to run. Even if you specify a test more than once, either directly or by incorporation in a test suite, QMTest runs it only once.
If you wish to run all tests in the test database, use the
implicit test suite . (a single period; see Section 3.1, “Implicit Test Suites”), or omit all IDs from the
command line.
QMTest can run tests in multiple concurrent threads of execution or on multiple remote hosts. See the documentation for the run command for details.
How the output of the qmtest run command should be interpreted will be discussed in Section 2, “Test Results”.
QMTest can avoid running one test (a "dependent test") when some other test (a "prerequisite test") has a particular outcome.
Suppose that you have a test database with a very simple test that can be run very quickly, and a very complex test that takes hours to run. You know that if the simple test fails, then there is no chance that the complex test will pass. In that case, you could make the simple test a prerequisite of the complex test. Then, when you run both tests, QMTest will run the simple test first. If it fails, the complex test will not be run at all.
Alternatively, suppose that you have a very comprehensive test that tests ten features of your software. You also have ten separate tests, one for each feature. The comprehensive test can be run in one minute; runnning the separate tests takes two minutes each. So, you want to run the comprehensive test first; if it passes, there is no need to run the individual tests. However, if the comprehensive test fails, you may want to run the single tests to isolate the problem. In this case, each of the simple tests would have the comprehensive test as a prerequisite, indicating that the simple test should be run only if the comprehensive test fails.
If you explicitly run just the dependent test, QMTest will not run the prerequisite test automatically. In other words, prerequisites are an optimization; when running both the prerequisite and the dependent test, QMTest will run them in the order you've implied, and can omit the dependent test if it is not useful. But, QMTest will not automatically force you to run the prerequisite tests when you only want to run the dependent test.
Because prerequisite tests are not run unless you ask for them, the dependent test should not depend in any way on the prerequisite test. Otherwise, users will see different test outcomes when they run the dependent test by itself. In other words, each test should stand alone; the order in which tests are run should not affect their outcomes.
Given one or more input test names and test suite names, QMTest employs the following procedure to determine which tests and resources to run and the order in which they are run.
QMTest resolves test names and test suite names. Test suites are expanded into the tests they contain. Since test suites may contain other test suites, this process is repeated until all test suites have been expanded. The result is a set of tests that are to be run.
QMTest computes a schedule for running the tests to be run such that a test's prerequisites are run before the test itself is run. Prerequisites not included in the test run are ignored. Outside of this condition, the order in which tests are run is undefined.
If QMTest is invoked to run tests in parallel or distributed across several targets, the tests are distributed among them as well. QMTest does not guarantee that a test's prerequisites are run on the same target, though. On each target, tests are assigned to the next available concurrent process or thread.
QMTest determines the required resources for the tests to be run. If several tests require the same resource, QMTest attempts to run all of the tests on the same target. In this case, the resource is set up and cleaned up only once. In some cases, QMTest may schedule the tests on multiple targets; in that case, the resource is set up and cleaned up once on each target.
In the following cases, a test or resource will not be executed, even though it is included in the set of tests enumerated above:
A test specifies for each of its prerequisite tests an expected outcome. If the prerequisite is included in the test run and the actual outcome of the prerequisite test is different from the expected outcome, the test is not run. Instead, it is given an UNTESTED outcome.
If a test's prerequisite is not included in the test run, that prerequisite is ignored.
If a setup function for one of the resources required by a test fails, the test is given an UNTESTED outcome.
The cleanup function of a resource is run after the last test that requires that resource, whether or not that test was run. The cleanup function is run even if the setup function failed.