Strategic Modelling and Business Dynamics: A Feedback Systems Approach

10.8. Summary of Confidence Building Tests

This chapter has introduced a variety of tests that can help build confidence in models. The tests fall into the three broad categories shown in Figure 10.22. There are tests of behaviour, tests of structure and tests of learning. None of these tests proves that a model is valid in the sense of being the best possible representation of the problem situation. But taken in combination, they demonstrate to modellers and clients that a given model is fit for purpose and of adequate quality.

Figure 10.22. Opportunities for building confidence in models

Tests of behaviour include the visual fit between simulated and actual data in terms of the magnitude, shape, periodicity and phasing of trajectories. Good visual fit is a common and effective way to build confidence in a model because it is a criterion people readily understand, even if they are not modellers. Statistical fit is a similar but more formal test where goodness-of-fit is measured numerically in terms of mean absolute error (MAE), mean square error (MSE) and the Theil inequality statistics. Of course, fit alone does not prove a model is 'correct', it simply shows the model is capable of replicating behaviour from the real world.

Tests of structure evaluate the quality of a model according to criteria that experienced modellers recognise and follow. The tests are like those a craftsman ...