Verification, Validation, and Testing of Engineered Systems

5.4 BLACK BOX—HIGH-VOLUME TESTING

5.4.1 Automatic Random Testing

Purpose

Automatic random (or statistical or stochastic) testing is based on the concept of automatically injecting very large quantities of random inputs into a system in order to test its behavior. This approach is the opposite of using predetermined and manually selected tests.

Rationale

The motivation for conducting random testing stems from the fact that it offers the ability to test the system against a very large and, often unexpected, range of system tests generated automatically and with limited investment. The use of broad test samples assesses the stability and reliability of the system by mimicking, to a large measure, its behavior over a long period of time. On the other hand, random testing in its purest application is somewhat risky, due to the lack of a reliable test oracle (i.e., specifiable failure output values). Without such a test oracle, one could miss finding discrepancies in the specification and can assure finding only obvious faults such as system crashes or certain error conditions. Another concern about this method is that we may need to be careful to restrict the random test data generation to only external conditions that could possibly occur. Otherwise, we would waste valuable time and resources evaluating test results or making system improvements that make no sense.

Method

The objective of automatic random testing is to evaluate system performance under unexpected conditions over ...