This section examines some examples of different test environments that are currently available.
The simplest ad hoc test environments
are just collections of a number
of different test commands. Unix shell scripts and Windows batch
files provide one way to collect these commands together. Scripting
languages (such as Perl, Python, and Tcl) can also be used. Failure
of a test written for these environments can often be detected by the
value of each program's return code, with nonzero
generally indicating a problem. Output is usually simple: the test
fail, with the
error code. Single tests can be run by simply invoking them with the
same arguments that the shell script uses.
Some aspects of testing that become more difficult if you use shell scripts and batch files are:
Checking for deadlocks when a test is not ending properly. This can be done by using a separate watchdog process to provide timeouts for tests, but a program can do this more efficiently by using threads.
Coordinating which regular expression patterns to look for in the output from each particular test. With a more general language, this information can be stored in the same place as each test definition.
Gathering statistics about different parts of each test—for example, startup times and response times. A program can offer a better structure for running different parts of tests separately.
Generating reports in HTML. This is just plain tedious using shell ...