System Dynamics Applications / 2 / 2.5 / 2.5.1

2.5.1 Chicken OR Egg? Understanding The Problem Before Simulating, OR Simulating The Problem To Understand It?

The mental models we hold of complex-world problems are often unique, and because we would feel vulnerable if were to have them made public, they often remain untested. It is a human foible that we hold on closely to our mental models even when we suspect that if they were to be subjected to scrutiny they may not stand up. For example, the implications of policies implemented by heads of government as advocated by public-policy advisers may not be realised for many years, such as at the time a key figure writes his or her memoirs. Yet despite this, policies and strategies with the most far-reaching consequences are frequently made with little real testing of the mental models upon which they were built.

This is particularly the case in economic, social and socio-technical problems where there is scope for interpreting what is observed in many different ways. By comparison, in physical problems there is limited scope for variations in interpretation. Physical problems, such as those addressed by engineers, are generally well-understood because they are governed by physical laws: the double pendulum problem is one such example. Frequently, they are generally simpler than the problems government policy makers or managers face. Jay W. Forrester (1975: 63) made this point when reflecting upon the challenges facing those in the engineering profession compared with their management profession counterparts:

Even the simplest social systems of practical interest lie in the range of tenth to hundredth order (the total number of integrations within the interconnected feedback loops). Almost all their transfer functions (policies) contain important non-linearities; and their principal interacting feedback loops may total ten or more. Our present social systems have been constructed on the basis of experience and intuition. Yet it is the rare engineer whose intuition and experience, when he is presented with the structure and parameters of a second-order linear feedback system, will permit him to estimate its dynamic characteristics correctly. It is safe to say that no one can correctly anticipate by inspection and intuition the behaviour of a fourth-order system, given a random selection of coefficients. How far then must the manager be from a high performance and efficient system when he tries to master the construction of a hundredth-order system with extreme nonlinearity? The result is that same as that which the engineer would encounter if he were to design a physical system of comparable complexity by trial and error. To achieve a semblance of stability he would of necessity give up all but the lowest performance.

Sterman (2000: 37) observes:

… our mental models are dynamically deficient, that is, they omit feedbacks, time delays [and consequences of system response], accumulations and nonlinearities … simulation is the only practical way to test these models, noting that complexity of [both the real world and] our mental models vastly exceeds our capacity to understand their implications.

Further, Sterman (2000: 37) adds:

… conceptual models, typically expressed as causal diagrams, are too large and complex to simulate mentally… without simulation, even the best conceptual models can only be tested and improved by relying on learning feedback through the real world … this feedback is very slow and often rendered ineffective by dynamic complexity, time delays, inadequate and ambiguous feedback, poor reasoning skills, defensive reaction, and the costs of experimentation … in these circumstances, simulation becomes the only reliable way to test hypotheses and evaluate the likely effects of policies.

Of course there is a danger here of misinterpreting what is observed by way of behaviour produced by a model during simulation. Indeed, some would argue against modelling and simulation. One contrary view is expressed this way: “If you do not fully understand a problem, you cannot build a model of it or simulate it!” This introduces the chicken and egg conundrum. Can we build models to test what we know about a particular problem situation, as Sterman suggests, or are we unable to build models until we fully understand the problem we face, as the contrary argument suggests.

The contrary view that we must understand the problem first has some validity. It is based in a line of argument which suggests that despite being well-intentioned, even capable and competent modellers can draw the wrong conclusions: ill-informed or incompetent modellers would do much worse. Building models that look right because they apparently replicate observed real-world behaviour but, in fact, are wrong can produce erroneous understanding.

The argument which suggests that full understanding of complex problems must exist before problems can be modelled and simulated without error is intended to protect clients of modelling against the interpretations and prophecies of those who erroneously interpret the behaviour of their models because they do not really understand them.

One important and strong argument for building system dynamics models of complex (dynamic) problems is to enhance understanding. However, to achieve valid understanding and derive valid learning outcomes from modelling and simulation, the models must satisfy important criteria. The models must be both necessary and sufficient representations of the real world to be useful and to inform valid conclusions: to do this they must be error-free.

At this point it is essential to make it clear that, quite conceivably, we might build an error-free model that has very little utility. We might build another model that presents concepts that challenge our thinking even though the model may be incomplete or strictly incorrect.

Testing is essential to identify errors in the model, but every test must be based on an expectation of invoking a particular response from the model. Further, through developing and applying tests with the intent of verifying that a given model actually behaves as intended, we test our own detailed understanding of the inextricable link between model structure and behaviour. For example, we might design an extreme-value test of our model (such as providing an input of 1,000,000 when the normal range of values might be between 10 and 100), but we must fully understand the implications of applying the test and be able to explain why the model behaves as it does when the test is applied. Similarly we must be able to explain the counter-intuitive responses or unstable transient behaviours of the model. In another example, we might change the initial conditions by a very small amount only to find that the model behaves very differently. In the former example, we might have discovered that the model is insensitive to a particular input variable regardless of the value of that input. The latter might be an indication that the model can behave in a chaotic manner: an example of this is the double pendulum in which a change of less than one degree of angular displacement of either of the two pendulums can produce vastly different behaviour over time.

Traditionally, building models of complex problems and conducting simulations to investigate how they behave over time has been left to experts in mathematics and operations research. It has been implicit that high levels of mathematical skill were needed.

However, that has been changing for more than two decades. The advent of relatively cheap but powerful digital computers at the desktop has produced previously unimaginable opportunities for expert and intelligent non-expert alike to experiment with complex problems through modelling and simulation.

Without rigour and discipline through every stage of modelling, meaningless or erroneous models can result. Risks of this happening will be small if models are progressively verified. Again, this is a matter of applying discipline.

An approach to model building which relies on defining modules of system dynamics structure, assembling and comprehensively testing demands considerable discipline. Building and assembling modules and progressively testing models are powerful and engaging activities that lead to understanding the linkage of structural elements, and the causal relationship between structure and behaviour produced by the model as a consequence.

Through a modular approach to model building, it becomes possible to build models of highly complex problems and conduct powerful simulations to provide unprecedented insights. The questions to focus on are … ‘how are these insights produced through the building of models, testing models and running simulations, and how can these insights be enhanced?’

Of course, if you are unable to define the systemic structure, or if you lack understanding of what drives the observed real-world behaviour over time, it is not possible to build models correctly and to conduct meaningful simulations.