Verification, Validation, and Testing of Engineered Systems

7.2 OBTAINING SYSTEM QUALITY DATA

7.2.1 Quality Data Acquisition

Usually, system quality data may be obtained from the following sources: (1) available databases of past projects, (2) measuring quality information during a specific project and (3) eliciting information from domain experts. Once raw data have been obtained, a method to aggregate and harmonize it must be selected in order to come up with unambiguous and clear quality data.

Delphi Process

The purpose of eliciting data from experts is to bridge the gap between available database records and required information. Cooke (1991) provides an extensive survey and a critical examination of the literature on the use of expert opinion in scientific inquiry and policy making. The elicitation, representation and use of expert opinion have become increasingly important since advancing technology requires more and more complex decisions. Cooke considers how expert opinion is being used today, how an expert’s uncertainty is represented, how people reason with uncertainty, how the quality and usefulness of the expert opinion can be assessed and how the views of several experts might be combined. Loveridge (2002) expands on Cooke’s seminal work and covers topics such as the selection of people for expert committees, as this is much more critical than is generally appreciated. Vose (2008) describes modeling techniques based on various probability distributions (e.g., triangular, beta).

Eliciting data from experts can be a difficult ...