Methods and Applications of Statistics in Clinical Trials, Volume 2: Planning, Analysis, and Inferential Methods

Chapter 26

Multiple Evaluators

Jason J. Z. Liao and Robert C. Capen

26.1 Introduction

In medical and other related sciences, clinical or experimental measurements usually serve as a basis for diagnostic, prognostic, therapeutic, and performance evaluations. The measurement can be from multiple systems, processes, instruments, methods, raters, and so forth, but for the sake of simplicity, we refer to them as “evaluators” throughout this article. As technology continues to advance, new methods/instruments for diagnostic, prognostic, therapeutic, and performance evaluations become available. Before a new method or a new instrument is adopted for use in measuring a variable of interest, its agreement relative to other similar evaluators needs to be assessed. Measurements of agreement are needed to assess the reliability of multiple raters (or the same rater over time) in a randomized clinical trial (RCT). These measurements of agreement can be used for assessing the reliability of the inclusion criteria for entry into an RCT, validating surrogate end points in a study, determining that the important outcome measurements are interchangeable among the evaluators, and so on. An agreement study involving multiple evaluators can happen in all phases of drug development or other medical-related experimental settings. A typical design for an agreement study involves sampling n subjects from the population, sampling d evaluators from a population of evaluators, and obtaining r replicate determinations/evaluator. ...