Simulation-Based Training – Beneath The Shroud

Introduction

Simulation has been used throughout history as a means of gaining insight into the behaviour of a real-world system. Initially, simulation was utilised as an analytical tool, providing a means to understand, optimise, or predict behaviour of systems of interest. Application areas such as building, planning, logistics, and management have used simulation techniques to support understanding of either current or proposed systems. For example, early simulation within the shipbuilding industry took the form of scaled physical models evaluated within a scaled ocean. These physical models were highly accurate replications to include the actual wood, rigging, ballast weights, and armament. Once validated as sea worthy the physical models became the “blueprint” for the actual ship.

Theoretical problems such as solving simultaneous linear equations were simulated using Monte Carlo methods [1], which used statistical sampling to estimate a desired result. Other types of problems utilised different mathematical techniques. With the advent of computers, symbolic models can now be developed that represent entities or objects and provide for the inclusion of stimuli. As computer technology improved, more application domains embraced simulation.

Over the last decade Instructional System designers have also been increasing their use of simulation to satisfy training requirements. Initially simulation was used to represent the system they were studying, thereby allowing students to “interact” within the basic parameters to understand better the system operational responses to stimuli or variance of the input variables. Today, training systems use simulated environments to immerse the student in an operational domain. Visual systems developers have added a perceptual realism by providing enabling technologies, which visually present a common virtual environment to the students. Visuals systems today allow each participant to experience a simulated environment as they would the real world. Entities can now be added through models or connected manned simulators as well as actual “live” equipment. Finally, systems are now able to interact over great distances with the advent of a common communication and architectural framework.

Given these advances in modern technology coupled with the shrinking defence budgets, the defence forces of many countries not only depend on simulation, but have mandated simulation as the enabling technology for training systems that span individual task training to mission rehearsal [2]. The use of simulation at all levels of the vertical training vector from individual tasks to large-scale joint exercises is now common practice. Given the increasing role which simulation can play with the advent of standards, technology and acceptance, knowledge about the domain called simulation is essential. As defence forces worldwide become more dependent on simulation to augment, if not replace, real-world training, simulation becomes a direct contributor to the overall preparedness of our forces. With that in mind, this article’s aim is to bridge the gap between the training manager, user and engineer so that a common understanding of the science of simulation and its current application space can be forged.

Therefore, this paper focuses on:

• the current state of the science of simulation;
• the legitimacy of simulation to support training;
• a description of simulation-based training systems; and
• an evaluation of areas of risk and associated problems, and future research.

Background

“Simulation has always been a difficult issue to analyse because it is surrounded by a semantic quagmire, and obscured by a miasma of emotion, over-claims, and flawed analysis, unilluminated by dependable statistics on costs or effectiveness” [3]. Penned in 1994 within a study on the utility of simulation for training, much of the voiced and perceived frustration of then is still true today. As noted then, the miasma in part is self-generated and will take the combined effort of the simulation community to eliminate. But a path through the quagmire addressed in the cited quote may be a reasonable goal of this article.

The semantic quagmire itself is mainly the result of different facets of the simulation community creating their own vocabulary in reference to identical concepts and constructs. Terms such as, Synthetic Battlespace; Common Synthetic Environment; Distributed Interactive Simulation (DIS); Live, Virtual, and Constructive entities; and Virtual Reality, serve to describe the application space. Terms such as aggregation (disaggregation, deaggregation); fidelity; resolution; real-time; latency; bandwidth; interoperability; temporal, spatial and behaviour relationships; interconnectivity, and High Level Architecture (HLA), characterise development aspects of the simulation which are necessary enabling principles for a synthetic environment. Finally, the academic community refers to simulation in terms of mathematical, qualitative, quantitative, knowledge-based, intelligent, rule-based, physical and symbolic models; deterministic versus stochastic natures; predictive, prescriptive and descriptive systems; discrete-event state change versus continuous state change world views, validation and verification; abstraction; and reductionism, to name a few. Furthermore, there is a tendency to use the terms modelling and simulation interchangeably. Finally, with the expansion of the application space, development processes and engineering domain of simulation, there is pressure to coin new terminology and descriptions for each effort so that it can be differentiated from previous undertaking and explanations. No wonder the quagmire around simulation appears to be incomprehensible.

In addition to the semantic quagmire, there is another contributing factor to the somewhat negative view of simulation’s overall effectiveness - the lack of a common conceptual framework for understanding the utility and capability of simulation as a science and tool. More specifically, there is a void of established, published principles for simulation development. Current simulation development is similar to software engineering in the 1970s – it is both an art form [4] as well as a creative activity [5]. The quagmire is further exasperated by the fact that many simulation systems are tendered within a procurement system uninitiated in the science of simulation, for a customer that is not able to articulate training needs into system requirements. These systems are then developed in absence of trained simulationists utilising development processes that are neither capable nor intended to provide the mechanism for reducing the real world order into a non-chaotic computer-hosted symbolic emulation. Thus, given the confusion caused by the lack of clarity and precision associated with the science of simulation, the term quagmire is a justifiable metaphor. Furthermore, this confusion (coupled with lack of tools, measures and simulationists), introduces a level of developmental risk over any simulation enterprise.

The impact of this lack of scientific focus on simulation methods can be felt by looking at a cross section of the major simulation-based “training” systems under development today. Many of these programs are having difficulty meeting the capabilities expected by the user community, let alone schedules and budgets. Within the community, examples of current simulation development programs that have good marks are unfortunately rare. This is not because the companies cannot perform. The two main reasons for the problems facing simulation development appear to be shifting requirements and a procurement system that has not changed to accommodate the complexities of simulation systems as opposed to complex software systems.

Companies that have been successful have adopted numerous tools, feedback mechanisms and engineering processes to ensure that the development activity is continuously monitored, measured and documented. Risk Assessment programs coupled with an Integrated Product Team (IPT) concept (a collaborative enterprise between the customer and the developer), have been utilised to close the gap between conceptual expectations and the delivered reality. As these lessons are being learned, it is becoming clear that simulation system development, regardless of the application domain, is more than software engineering. Therefore, existing procurement practices, development processes and engineering approaches must also evolve to the level required to take a complex real-world environment and produce an abstract simulation-based training system

As software development evolved from an art form into an engineering discipline now founded on academic principles, tools, and professionals; so must simulation development go through the same metamorphosis to become a legitimate engineering science and academic discipline. The combined efforts of the academic, development and user community must support and somewhat drive this transition if simulation-based training systems are ultimately to provide the utility required at an acceptable cost. Currently, the shroud of confusion inhibits simulation from realising its full potential.

Simulation foundations

Much of the current application space and literature refers to the field of simulation as Modelling and Simulation (M&S). Therefore, it is important to differentiate between the two components. Furthermore, to begin to remove the shroud surrounding this area and to provide a basis for discussing simulation-based training, we must also review the fundamental and sometimes conflicting points of view and scientific principles that in effect legitimise the use of M&S as a training and analysis technique.

To establish a common reference for this article and before attempting to embrace simulation for any activity, two fundamental questions must be thoroughly understood from both a technique and tool perspective:

What is modelling?

What is simulation?

What is modelling?

Modelling as defined in the US Department of Defence, A Glossary of Modelling and Simulation Terms for Distributed Interactive Simulation (DIS), is the “application of a standard, rigorous, structured methodology to create and validate a physical, mathematical, or otherwise logical representation of a system, entity, phenomenon, or process”.

Modelling, as it relates to training environments refers to the actual reduction and system development process associated with replicating a real-world environment by a set of interactive symbolic models. The result of this process is an abstraction of the real-world system defined by a conceptual model or set of models. The conceptual model constitutes the allowable range of entity interactions and behaviour characteristics as they pertain to the represented training environment. Therefore the question “What is modelling?” refers to the real-world reduction and system development process.

For the purpose of discussion we shall adopt the following composite definition.

Modelling, the technique, is the reduction of a real-world phenomenon of interest into a construct or set of constructs called symbolic models, which captures specific system behaviours. These symbolic models which emulate the real system are defined by a set of dependent and independent variables that are influenced by a set of stimuli which are governed by a set of constraints. The end effort is targeted at a particular computer system and designed in a specific computer language.

The modelling activity is a difficult challenge for any professional and requires a combined set of skills in problem formulation, system reduction, modelling techniques, experimental design and other specific skills relative to the application space under study. It must also be noted that two modellers of the same system given the same constraints will probably approach the effort differently. Thus while the resulting efforts may have some commonality, the actual representation may differ in modelling techniques, approach and fidelity. Therefore, the need for a uniform approach to modelling will help guide the overall effort to a successful conclusion.

What is simulation?

The term simulation is both the resultant system and the activity of executing the system to gather information about or replicate the real-world phenomenon. As a starting point, simulation is defined in The American Heritage Dictionary as “an imitation”. The Macquarie Dictionary characterises simulation as “pretending or feigning”. Both further define simulation as “the assumption of a false appearance”. In reality, there is a very fine line between the validated imitation and the believable sham.

The Institute of Electrical and Electronics Engineers (IEEE) Standard Dictionary of Electrical and Electronics Terms provides a more focused technical definition with respect to software: “Simulation (Software). The representation of selected characteristics of the behaviour of one physical or abstract system by another system.”.

A particular simulation, the tool or end product, is an abstracted model or group of interactive models that have been developed to approximate a specified system for a particular purpose, (that is training, planning, analysis, entertainment and so on). The “system” can be a simulation of a natural system (that is biological, ecological, geological, astrophysical), a hypothetical system or environment (that is synthetic environments, interactive games, virtual worlds or systems), and intelligent systems (that is human behaviour, and reasoning).

Computer simulation typically includes the execution of the full set of symbolic models, which comprise the referent system over a specified time period. As a simulation executes, data is captured so that some type of inference can be made about the real world or referent system. Depending on the modelling techniques and types of independent variables, issues dealing with time advancement, initial system loading, number of replications and randomness, and so on, must be resolved. The following definition is therefore proposed to define the tool and activity.

Simulation, the tool, is a set of interactive symbolic models that have been developed to represent specific behaviour(s) of a real-world system or phenomena. Simulation, the activity, is the exercising of these models over a defined time frame to provide an emulation of the real-world phenomena given a set of variables of interest as bounded by the initial assumptions, conditions, constraints, and stimuli, which characterised the original real-world system.

Although simulation has been called a “problem-solving process” [6], that description no longer accurately depicts the application space that now embraces simulation. In fact, the US Defence Science Board characterises simulation as:

“Everything is simulation except combat.”

Although defined in terms of military training, it is clear that the user community, in general, is equating simulation with the absence of the trained-for reality (combat) and in so doing, has implied that simulation can emulate that reality with sufficient accuracy as to diminish the need for real-world experience. More specifically, the characterisation implies that until the reality of combat serves to train the combatants, simulation will have to serve as the transitional mechanism from which these combatants gain their experience and confidence.

In order to establish a working taxonomy for referencing M&S for the rest of this article, the bounds of the term must be set. First, a simulation is implied to mean a set of symbolic models hosted on a computer utilising one of the representation techniques. Second, a simulation-based training system refers to a distributed interactive computer simulation within a specified training environment. Finally, the use of the term simulation as the broader description is meant to include the entire development process and resultant final product as specified in a modelling and simulation project unless further clarified.

Simulation perspectives

Simulation spans many disciplines, which individually have their own perspective, bodies of knowledge and terms by which they characterise simulation. As stated earlier, part of the reason for the semantic quagmire surrounding simulation is the lack of a focused strategy, science and policy from which to embrace the underlying science of simulation. Therefore, given the proposed working definitions of simulation, a common conceptual framework is also required to further remove the perceived quagmire.

Today, within the simulation community, organisations such as the (international) Simulation Interoperabilty Standards Organisation (SISO) [7] exist to provide an open forum that promotes cooperation and exchange of ideas, and to facilitate the standards development process and practices for simulation with respect to interoperable training systems. Policy centres such as the US Defence Modelling and Simulation Office (DMSO) [8] and the Australian Defence Simulation Office (ADSO) have been established to provide a community focus for collaborative development of standards and processes for the simulation community. Research centres such as Australia’s Defence Science and Technology Organisation (DSTO) and the US Institute for Defence Analysis (IDA), in collaboration with other scientific centres world wide, constitute a sound engineering community that provides continued research and technical advancements.

As a result of these different communities and their viewpoints, a common reference point for simulation is starting to emerge in the form of a multi-faceted architectural definition. For example, as shown in Figure 1, the US C4ISR Architectural Framework [9] provides a valuable reference point for describing the different architectural aspects of a simulation system.

Figure . Architectural Relationships.

These aspects, when oriented to simulation systems, allow for defining and representing a simulation process and the resultant system in terms of operational, technical and system characteristics. These different aspects define their requirements in terms of interactions and functional relationships with each other, thus resulting in a structured analysis of the intended simulation system. Although not a complete simulation methodology, this multifaceted approach does provide a starting point for further collaboration.

Operational aspect

Ultimately, a training system must satisfy the training needs of the target audience. By representing the user’s needs in terms of an operational architecture, the user and the developer have a tool to identify and capture training requirements. The operational aspect defines the system in terms of activities, supporting data, rules (policy and doctrine), qualitative and quantitative influences, constraints, levels of entity control and interactions required to meet the user’s needs. Behavioural characteristics must be identified to provide both entity interaction and operational realism. If the system is intended to interoperate with other systems, communications functions must be established to allow for information to be processed and disseminated without degradation of the simulation environment. Finally, if the simulation system is intended to provide a real-time virtual environment thus allowing for interaction within that environment, temporal, spatial, and behavioural issues must also be resolved to ensure that interoperability is achieved among the components comprising the overall training environment.

Technical aspect

Technically, the resultant simulation system is intended to be accurate within a particular environment for a specified range of influences, assumptions, and information and is focused at a particular level of fidelity and resolution that dictates the granularity of the resulting information, interactions and levels of control.

The technical aspect of the architecture is intended to capture the development guidelines, standards, and criteria that if implemented correctly will meet the requirements for system services, interfaces, and interoperability issues demanded for the proposed simulation system. The technical architecture is therefore the architectural view, which translates the operational needs into system standards and requirements through the implementation of specifications and processes.

System aspect

The simulation system must eventually reside on a specified computer system. Depending on the requirement, the simulation system may act as a stand-alone tool on a single system or may be distributed over a network (LAN/WAN), requiring communication lines, interfaces, hardware, and protocols to support the proposed simulation traffic. Interconnectivity issues of bandwidth and latency must be accounted for to insure that the physical architecture does not impact the operation requirements. Therefore, the intent of the system view is to depict the physical connections, networks, interfaces, and other platforms required to satisfy the operational requirements. As mentioned before, the intent of this view is to present a set of physical resources that, in fact, meet the operational requirements as well as the governing technical specifications and standards.

Simulation principles

The engineering activities associated with creating a simulation system culminate in a set of models that are intended to represent the behaviour of the modelled component over a range of defined stimuli. Once a simulation system is completed, decisions as to simulation techniques, information processing, fidelity, resolution, world view, timing and model representation have all been made and incorporated into the final product. An understanding of the process for capturing the real-world system and representing it as a set of symbolic models is essential in order to understand the complexity of a specific simulation development process.

Real-world reduction

The process or formalism of reducing reality into a symbolic model for the purpose of simulating a real-world system has been referred to as abstraction. The basic principle from which simulation depends on for validation as a scientific technique is that the real world can, in fact, be studied by reducing it into parts and studying those parts in isolation. Some argue that the reduction process itself obscures the basic underlying relationships and that in their isolation, they are poor imitations of the combined reality and therefore provide no meaningful value [10]. For example, to attempt to model traffic so that control mechanisms can be placed intended to actually expedite flow, the modeller will quickly run into the dilemma of how to model an individual bus, train, car, taxi, and so on. The modeller will soon realise that actually it is the drivers that are being “controlled” and that they are affected by weather, traffic, time of day, sunlight and so on. These drivers manifest different traits depending on whether they are residents or tourists, old or young, experienced or beginning drivers, men or women. Ultimately, a driver’s actions, reactions and attitudes, are influenced by the composite of these factors coupled with each driver’s cognitive and motor skills. Where does the modeller draw the line on the influences to be included? Clearly, the warning from the anti-reductionists is that the degree of deliberation on what to model and how to represent the interactions of both qualitative and quantitative influences will ultimately determine the success of the undertaking.

Although reductionism may have been more readily acceptable in past years, the increasing complexity of today’s systems and in conjunction with an expanding application space lend credence to the anti-reduction supporters. Given this increase in complexity, the reduction of the underlying reality and influences of these systems is much harder to isolate and consequently replicate in the desired isolation or environment. On the other hand, some aspect of reduction must be plausible if we are to understand our world at all.

Given the concerns above, the abstraction process defined by Zeigler [11] provides a formal mechanism, when expanded, for deriving operational, technical and system requirements through a reduction process. The real-world system is reduced to a set of influences or stimuli, bounded by a set of constraints and characterised by a range of allowable interactions that are then transformed into a set of symbolic models that when executed attempt to replicate the real-world system they represent.

Although the process of abstraction may be well articulated theoretically, the actual implementation process can be somewhat daunting. Defining and bounding a simulation system can prove to be both elusive and expensive. Not only must the final system meet a defined training need, the process must take into account an imposed level of realism. Mission-creep, in the form of operational support requirements, tends to expand the nature of the simulation system as these new training requirements are assimilated into a system that was not initially designed to support them. Therefore, it is imperative that prior to development, a firm understanding of the intended use of the simulation training environment is quantified and bounded in order to select the corresponding level of fidelity and resolution as well as the appropriate modelling approach for the different influences and interactions.

The selection of the correct set of modelling techniques directly impacts the success of the overall simulation. More specifically, the level of fidelity and resolution of the underlying modelling constructs directly impact not only the technical and system architectures but act as the main cost drivers for the system itself. Issues of interaction, timing, and information typing must be resolved before interfaces, communication issues and system architectures can be resolved. Capturing the customer’s or intended user’s requirements is part of the answer. Unfortunately, in most cases the user will not understand the difference between quantitative and qualitative information. Moreover, the user may not even understand, let alone be able to articulate, the actual training domain to be emulated in terms transferable into simulation. Without an understanding of the types of modelling techniques and their utility, decisions are made based on uncertainty. This then leads to the exclusion of system attributes and influences that may later be determined to be essential to the training system.

Therefore, to successfully abstract a simulation system from a need to a solution, regardless of the intended use, a formalised process must be established and adhered to. A mechanism for extracting user requirements and translating them into model formalisation is also required to support the overall development process. A formal process is critical to the military training community if the entire life cycle of the equipment is to meet current and future expectations and be cost-effective. To that end the abstraction process is an essential initial component of the overall development methodology.

Over the past ten years, two schools of thought have emerged that attempt to define the simulation process. The basic difference is grounded in the definition of the true nature of the effort.

Zeigler, recognises that simulation is a unique development process from which imitation of a real-world system is identified and created. As noted in his seminal work, Zeigler described a hierarchy process that allows for the reduction of a real-world system into a computer resident replication. His approach includes five levels of abstraction.

• The Real System is the actual system or source of observable data and or environment of interest.
• The Experimental Frame a set of limited circumstances (interactions) under which the real system may be observed or manipulated (or interacted with).
• The Base Model is a reference model capable of accounting for all of the stimulus, transactions and interactions of the real system in terms of perceived behaviour.
• The Lumped Model is a reduced version of the base or reference model that is still valid for the defined areas of interest in the Experimental Frame.
• The Computer Model is a computer-hosted set of models that implements the Lumped Model, utilising a particular computer language and computer architecture.

The goal of the process is portrayed symbolically in the equation:

real system (data) ≈ model-generated (data)

Obviously the desired result of the process is for the simulation to be an exact imitation of the real system as it relates to the training requirements. In practical terms though, the more complex the system, the less likely that goal can be achieved. Accordingly, the degree of validity of both the process and the resulting model can be described by three increasingly coupled relationships to the real world:

• Replicatively Valid – model data matches real system.
• Predictively Valid – predicts data from real system.
• Structurally Valid – emulates the behaviour of the system in producing the data.

Validation of simulation-based training systems may require all three levels of validity depending on the training effectiveness and performance measures to be met.

The second school of thought approaches simulation development as an extension of the Software Engineering paradigm in which a set of formal requirements is implemented through a structured process into the desired system. Missing from this standard approach are the steps required to create a formal requirements definition for the system as accounted for in the abstraction process in terms of real-world realism. In reality, both approaches must be combined to account for the uniqueness of simulation system development.

Simulation “world views”

Although the abstraction process provides a theoretical framework for reducing the real world into a computer model, the actual abstraction activities and assumptions of fidelity, influences and constraints must be based on a broad perspective or ‘world view’ [12]. The world view is based on the relationship of the dependent variables with time. Typically, there are two world views:

• discrete-event state change, and
• continuous state change.

Discrete-event modelling

Discrete-event modelling refers to analysis of a system at discrete-change intervals. These intervals are determined by either a change of state of an entity or allowable transaction. Most publications [13] further reduce the world view of discrete-event modelling into either of the following:

• Event Scheduling,
• Activity Scanning, and
• Process Interaction.

Event scheduling systems process events based on the event’s occurrence in time without any consideration of conditions of order or hierarchy. For example, the interaction of two entities will cause an event to be scheduled at that point in time.

Activity scanning systems process events based on both a schedule and applicable conditions. Typically, meeting all of the conditions for execution triggers the activity. Until then the scheduled activity is placed in a queue. By monitoring the queue activity, conclusions can then be drawn as to the efficiency of a process.

Process interaction systems represent processes that are linked together in a definite pattern and may compete for entities as well as resources. An example of this type of system is training resource allocation modelling with respect to curriculum processes. The entities – students – must go through a complete process constrained by resources and throughput requirements.

Real-time, interactive training systems with respect to the training audience are discrete-event simulations. These systems tend to be event scheduling in nature and process entity interactions through a set of conflict resolution rules, which determine the outcome of the event as it applies to the entities and the simulation environment. Within the simulation, entities interact at finite or discrete points in time and it is the interaction that causes the state change. In this respect, time advancement can be viewed as dependent on the interactions within the simulation system. For example, when a force is engaged by another force in the simulated environment, it is the actual engagement interactions that are time tagged and ordered so that conflict resolution or state changes can be resolved. From this it is determined what happened to each entity (that is, fired on, hit, spotted) and what is the next state (that is, dead, missed, wounded, and so on) of the entities will be. Therefore, discrete-event simulations are characterised as state changes, which occur at distinct points in the simulation time line. As events are recorded, the system remains in that state until the next event. Thus it is possible to review a discrete event faster than actual or real-time since the state change is all that is recorded.

Continuous modelling

Continuous modelling results in a system where time is the independent variable. “Entities interactions” are represented by a set of equations based on one of the following:

• Explicit algebraic or functional forms [eg., y (x,t)].
• A set of difference equations [eg., yt+1 ayt + but].
• A set of differential equations [eg., dy/dt = (x,t)].

Continuous simulation has also been referred to as quantitative simulation meaning that the resulting system typically provides a set of values over a particular interval of time (t). Although the final simulation may be mathematically eloquent, it is difficult to understand and correlate to real-world behaviour. Mathematical models are typically present in training systems, but their results are usually presented in a form of perceptual stimulus.

Modelling techniques

The entities themselves comprise models that incorporate desired attributes and characteristics within a set of logical and physical constraints. The underlying modelling techniques can be mathematical models, artificial intelligence (AI) models, knowledge-based (expert system – rule-based) models, behaviour models, and graphical models. Part of the abstraction process is to not only determine the world view but also to determine modelling approaches for each of the components of the simulation. Initially, decisions as to the variables and their form will help determine what type of data is to be passed between models and consequently what types of information is presented to the training audience. In a real-time interactive environment, for example, data between entities may be a new position in terms of coordinates, but the information presented to the student may be that the entity, such as a tank, may be on a main road as displayed on the system map. Therefore, it is the format of the information and the processing requirements that ultimately determine the types of models that are to be developed. Additionally, considerations must be given to system constraints as well. The more fidelity or details represented by a model, the more internal processing time is required to formulate a specific action or response.

Simulation-based training

Simulation, as used by the military for the purpose of training, poses a unique set of challenges. To a certain extent, when simulation is meant to encompass distributed interactive training environments, the application space exceeds traditional simulation paradigms. The basic natures of simulations in the past were described as:

• descriptive – describing or explaining the world;
• prescriptive – describing optimal solutions to problems; and
• predictive - describing or forecasting future states.

A training system may comprise simulations that do all three in the context of a combined training system. Yet in reality, an interactive training system as presented to the student is not any of the three traditional natures. In fact, training has created a fourth nature or paradigm:

emulative – representative of and responsive as the real world.

Through emulation we are attempting to replicate an interactive dynamic environment. The degree of difficulty in developing these systems is directly related to the type of training, the degree of interactions and the level of human involvement. The risk of failure to provide the required level of training has led to the development of training systems that utilise the latest technology as a means of defence against such a claim. Unfortunately, lessons are starting to emerge that demonstrate money itself does not guarantee a valid training system or process.

By looking at the full range of training systems, a hierarchy of complexity and consequence of failure due to invalid simulation environments can be derived. From the Defence community (as well as others) we have a need for training individuals in their tasks to training forces for mission-oriented exercises.

Distributed interactive simulation (DIS)

DIS has come to be known as a set of protocols, which enable heterogeneous simulations, simulators and semi-automated forces to operate over a network (LAN/WAN) in a common synthetic environment. More importantly though DIS is also a specific simulation paradigm, which requires stringent operation, technical and system standards in order to effectively create the required training environment. This environment poses the most difficult challenge of the simulation-based training paradigm. For training to be considered realistic, virtual entities projected in the training environment must be constrained in such a manner as to be:

• behaviourally correct – act and respond as the real-world entity;
• temporally correct – respond within the correct time (real-time); and
• spatially correct – appear at the proper location as determined by the mathematical, dimensional and symbolic characteristics.

The DIS community refers to the correct inclusion of these constraints as enabling interoperability. Communication issues such as protocols, latency, bandwidth, networks and message management comprise the other aspect of DIS commonly referred to as interconnectivity. Interoperability (realistic simulated interactions) and interconnectivity (communications architecture, management and control) comprise the two essential components required in any distributed training or gaming environment. Each component contributes to the overall satisfaction of a realistic real-time training or Synthetic Environment (SE).

The anatomy of a DIS training system

The current efforts as expressed in terms of time and money being spent to define a common simulation environment, is a clear statement of the intent to continue to use simulation as an enabling technology for training. Virtual or synthetic environments provide a realistic “world”, which students may interact in and with. Simulation is an attractive alternative to using real weapons, equipment and environments in the face of shrinking budgets, complex systems, and environmental concerns. Given the complex nature of a SE (Figure 2) which must deal with interoperability and interconnectivity issues, tools for abstracting that environment into a valid computer generated set of models are essential.

The complexity of issues to be addressed is increased, given that an interactive system with man-in-the-loop elements must appear to operate in real-time, and that the responses to action, activities and interactions must be temporally, spatially and behaviourally correct. Entities or objects operating and interacting in the SE must react as they would in the real world.

Interoperability represents the desired goal when an interactive SE is developed for training. But interoperability itself does not provide a total validation of the system. Further validation comes from the inability of the user to distinguish the interactions of the simulated environment from the real world. Turing [14] defined a validation process that simply stated that a system could be validated if, when presented data from both the real and simulation system, experts could not distinguish a difference. Simulation-based training attempts to augment, and in some cases replace, the real world with an emulation. A logical extrapolation of the Turing Test would be to present students to knowledgable experts and see if they can determine how they were trained. Obviously training standards for measuring effectiveness and performance would also be required.

By applying the Turing Test to training, interoperability becomes a requirement supported by performance measures for defining behavioural, temporal and spatial requirements that ensure students cannot tell if the entity and environment they are interacting with is computer-generated or real. Real in this sense, though, still requires that information sent from the real entity such as an aircraft, be transformed and represented in the SE exactly as in the real world. Interoperability is impacted by the two main components of a SE, the communication network and the training systems that included simulations, systems, and equipment, which produce the interactions.

A perfect network in the training world is one that has infinite bandwidth and zero latency. Of course regardless of the improvements under way, networks are far from that goal. Therefore, interoperability and interconnectivity issues must be resolved through optimisation of the simulation functional, communication and operational architectures in order to provide the best environment for training. The Joint Technical Architecture (JTA) has been developed to begin the process of standardising interfaces in the three architectural domains, while the High Level Architecture (HLA) addresses more specifically the simulation unique requirements. Nonetheless, the more detailed the environment in conjunction with increasing interactions and entity counts the more difficult it is to keep the system in real-time.

Pool table analogy

Modelling a pool table can provide an example of the levels of complexities accounted for in a distributed interactive training system [15]. Given fifteen identically round balls that are under constant motion and equal velocity, the SE must depict the exact location, direction, speed, and actions of every ball known as the current state at some time t. As described the pool balls are basically “dumb” three-dimensional entities that have changing states based on interactions with each other and their environment. If this is the real-world model, as we go through the abstraction or reduction process, the computer model becomes more complex than possibly first realised. Let us add the following assumptions:

• individual balls are no longer autonomous;
• the desired aim is to avoid collision among same “forces”; and
• rules for engagement must be followed.

Furthermore, we add a level of network complexity by distributing the balls along different machines. The pool table is artificially generated and is static except that the rails have no pockets and when encountered repel the ball. Finally, to add a level of control, let us assume that they are divided into groups of three and attempt to manoeuvre together. Finally, let us add in operators that are controlling the movements of one set of three semi-automated balls.

It becomes pretty clear that each entity must be able to detect the location of all other balls, and that each set of three balls must communicate state information to coordinate their activities.

Given that this is an analogy of a training system, let us call the existing balls a red force and add fifteen more balls as a blue force and fifteen more balls as a white force. The white force is neutral to the blue force but not to the red force. The blue and red forces both have different rules of engagement and doctrine. Red and blue forces have an operational chain-of-command for the purpose of information and intelligence gathering and command and control.

We have now generated a bounded playing field – the table, and have populated that field with semi-automated entities – pool balls. No longer is it adequate to just process entity state information but also future states based on current information about the entire playing field. Entities must not only determine their current state but also the state of others from which it must then plan a next or future state.

In order for the system to execute, a time system must be adopted. Given that there are real individuals in the loop, they must perceive the activities as occurring instantaneously or in real-time. The real-time requirement adds a level of complexity to the processing (interoperability) and communications (interconnectivity) requirements. To meet the timing issues, information must be gathered and then passed utilising modelling techniques that do not contribute to the degradation of entity behaviour or interaction time.

Although this is a simplistic look at the problems facing the creation of complex real-time distributed training systems, the analogy is also appropriate for the interoperability issues as well. The pool balls must occupy the dimensional space on the table as expected. They cannot fly, or pass through each other. There cannot be one super ball that destroys all the others unless that was part of the desired characteristics of that particular ball. Each ball must appear, behave and respond as the real-world ball for the simulation to remain valid with respect to the real world.

Fidelity versus resolution

Specifying a distributed interactive training system is a difficult challenge. Although the abstraction process cited earlier presents a conceptual framework for the reduction of the real world into a computer abstraction, this process is not the total solution. At odds with the abstraction process is the absence of an evaluation or reduction step targeted at the issue of how much depth in realism must the artificial reality accommodate.

Fidelity accounts for the not only the level of detail of the underlying models of both influences and object interaction within the training environment, but also how accurately their behaviour and impact are to be represented. Fidelity evaluation must also determine what interactions are valid. For example a computer or symbolic model may represent an aircraft as a point in space with direction and velocity, while another model may include mathematical models of engines, flaps and aerodynamics. Obviously, fidelity requirements are directly linked to the level of precision required for the training and the type of information to be supplied. The trade-off for fidelity is the amount of processing power needed to aggregate the information into a “whole” entity.

At the system level, fidelity impacts on a training system in two ways. First, fidelity considerations determine the level of information that must be captured, analysed, and processed so that the system may appear real for its intended purpose. The results of this process impact on the interoperability and interconnectivity or communications architecture of the proposed system. Secondly, analysis must determine the required perceptual level of fidelity the student is to be emersed into. While it may seem heuristically correct that a student will be better trained in a flight simulator than on a PC-based program, the reality is that the research community has yet to validate this perception.

To determine fidelity, careful consideration must be given to the degree of relative importance or criticalness of the required motor or cognitive skills in conjunction with the overall focus of the training. Validation of fidelity is a difficult challenge, which requires the development and application of a set of performance and effectiveness measures. Many factors contribute to effective training and since fidelity is one of the major cost drivers of training systems, decisions, which ultimately define the overall system, should reflect careful consideration of the available alternatives that meet the training measures of effectiveness and performance. Unfortunately, training effectiveness and performance standards do not exist which allow designers to determine a level of system and perceptual fidelity based on a training requirement.

Resolution, on the other hand, has been defined as the level of direct control or the implementation within the simulation of operational command and control (C2). Determining the level of resolution is directly attributable to the training audience for a particular training event and impacted by the overall entity composition – live, virtual, constructed. Resolution for entity control typically only goes down for two levels in the hierarchical C2 structure. Therefore, the target audience of the training exercise directly impacts the composition of the models and the control level and information presentation and processing of the interactions between the entities. Although each model is independently represented in the training environment, it may only have the resolution of a four-ship flight or a company of tanks. Entity resolution is specified based on the required information processing level of entity interactions and thereby provides the basis for determining appropriate (next state) responses.

Decisions that imply a level of fidelity and resolution account for the major cost of a simulation-based training system. Unfortunately, once these decisions are made, the determined levels of fidelity and resolution are then built into the system which can ultimately impact robustness, transportability as well as interoperability between systems.

The utility of simulation for training

It is somewhat difficult to measure the utility of a particular training system. In the past, studies have been produced that equate the cost, for example, of firing real missiles as opposed to virtual ones. In reality though, given that a training system is the appropriate imitation of the real system or environment, other benefits can be derived.

Simulation is cost-effective and can be measured by comparing the life cycle costs of the real system to that of the training system. Typically training equipment can be procured, managed, and utilised at cost savings over the real equipment.

Readiness as measured by the ability of an entity such as an individual or unit to execute its mission at the highest degree of competency, is the obvious goal of training. Simulation not only provides training on demand, it can capture metrics that can then be used to calibrate not only collective training but individual weaknesses for future enforcement. Individuals can then train without the need to include others until competency can be obtained.

Repeatability as it applies to simulation describes the ability to replicate and train on a given requirement within the same SE. Repeatability is essential for verifying student performance. By establishing measures of performance and effectiveness, features and data capturing techniques can be included to not only provide a basis for feed back, but also allow the training system to automatically concentrate on weaknesses for a particular student.

Failure, although not a desired characteristic of training, no longer carries the consequence associated with the real equipment. Aircraft that crash in a training environment allow the students the luxury of reviewing their mistakes. Overall, simulation/simulators reduce risk, safety hazards, and preserve the useful life of the real equipment.

Simulation environments allow for critical, dangerous, and potentially disastrous training components to be accommodated, where as in the real environment, safety and/or security issues preclude their inclusion.

Simulation provides a mechanism for including the capability of actual and anticipated stimuli or conditions, circumstances and reactions for interactive evaluation.

New technology and approaches can be introduced into the training environment for the purpose of familiarisation, analysis, and cost effectiveness.

New acquisition strategies as well as prototype systems can be presented at tenders to help define and present a particular approach to a set of requirements thus allowing for visualisation and conceptualisation of the proposed solution.

Challenges to be considered

While simulation-based training systems provide a credible alternative to training on the real equipment, very few will argue that there is no difference between a simulator and an airborne F-18. Obviously there needs to be both an acceptance for the limitations of simulation and a trade-off to balance the need for equipment familiarisation and training validation.

Measures must also be taken to insure that negative training does not take place. Military students will ultimately become soldiers, airmen and sailors who must depend on the real equipment and not the simulation. When the simulator and simulation match the real-world equipment and environment the risk of negative transference is diminished.

Finally, the accurate representation of the systems directly represents the interpretation of the training results. Systems, weapons, weather, visibility, terrain, sensors, doctrine, reactions, and so on must be captured through a methodically sound process and included at the correct level of fidelity and resolution for optimal training effectiveness.

Motivation as an independent variable

Probably the most basic factor that can impact on the need for higher fidelity and resolution incorporated in a training system is the motivation of the individuals being trained coupled with the consequence of failing at the real-world task. Recent examples have demonstrated that motivated individuals can receive a high degree of training transfer on low fidelity and resolutions systems. Although motivation can not be “procured” in a training system nor mandated in a training environment, it is a key independent variable that should be considered before a final solution is determined. The possibility of spending hundreds of millions of dollars for a simulator when a PC-based flight simulation package achieves a large percentage of the desired results cannot be discarded as inconsequential.

The evolution of change

The technology, policies and standards that impact the simulation industry are constantly under change. It is a difficult challenge to keep abreast of these evolving developments. Whether it is the High Level Architecture (HLA) [16] or new programs such as OneSAF (One Semi-Auto-mated Force) or Distributed Mission Training (DMT), to fail to keep up with the evolution of the technical and policy influences that drive the industry is to fail overall. The impact of not having a clear vision of the future evolution of these forces is to produce a legacy system from its date of conception. Part of the solutions is to identify central points of contact within Defence and liaison with their appropriate counterparts. Probably more essential is the investment in a well trained simulation community which includes mangers, users, and developers. Both Defence contractors and Defence program managers need a foundation of knowledge in order to effectively measure the true utility of any training system. Procurements strategies are shifting from specifying Statements of Work (SOW) – requirements document, towards a Statement of Objective (SOO) – a functional objective document. Clearly a move is under way to shift the burden of defining the optimal solution from the buyer to the contractor.

Future research

To facilitate the cost-effective use of simulation as well as further reduce the shroud of confusion, three areas of research seem to be most prevalent. from these issues more focus can be brought to bear on establishing a common body of knowledge for future simulationists and meet the needs and expectations of the user community.

Simulation development process

Modelling and simulation as defined in this paper requires a more stringent process than Software/Systems Engineering accounts for. Philosophically, a simulation system must go through a reduction process before system requirements can be defined. Although touched on in this paper, more extensive research into capturing training requirements in terms of qualitative influences and quantitative measures must be conducted. Other techniques such as Influence Diagrams, the Analytical Hierarchy Process (AHP), and Domain Engineering must be evaluated as a potential candidate tools for inclusion into a Simulation Development Process. Assumption testing and sensitivity analysis must also be accounted for prior to the creation of the training system performance and effectiveness specification.

Training metrics

Central to the development of training systems that are cost effective and provide the required level of training is the development of training metrics. Whereas, their operational, physical, and functional requirements define real systems, training systems must first be defined in terms of training transfer, fidelity and resolution which then translate into operational, technical and system requirements. Training and learning, consists of acquiring motor and cognitive skills sufficient to accomplish the defined task given a specified condition and derived standard. Measures of Effectiveness (MOE) are derived to measure the qualitative requirements of the task such as body position or fatigue, while Measures of Performance (MOP) capture the quantitative aspects such as reaction speed. The combination of the two when documented over the range of training tasks determines the underlying architecture required of the system. In most cases, it is a desired benefit to have the training system embedded in the operational hardware, but other decisions as to the supporting functional and operation architectures can be produced more cost effectively when produced from a training-needs orientation rather than a system orientation.

Simulation-based training learning theory

Although learning theory is in itself a difficult issue to tackle, the training community must come to grips with a plausible theory in order to measure training transfer, effectiveness, and retention. While traditional learning theory investigates how the mind acquires and files information that ultimately dictates actions, the training community must step back and determine the best way to present a simulation environment with respect to fidelity and resolution that best presents the desired training tasks. Inroads into this area require investigation coupled with cost-effective training environments, which incorporate the most effective learning techniques.

Conclusion

In the final analysis, it is obvious that simulation-based training plays a vital role in training defence forces and that, with the continued downward pressure on the Defence budget and personnel numbers, those within the Defence Force need to be well prepared for any future eventuality. Nonetheless, simulation as a science has a way to go before it is truly able to reach its full potential. The fine line between the valid imitation and sham is always present and only through the rigours of a stringent development methodology can the sham be avoided.

To illustrate this point, a mentor and friend from the United States defence community recently sent an article to me, which had been published by the Defence Systems Daily titled Mutant marsupials take up arms against Australian Air Force. This article described how kangaroos when “buzzed” by “hotshot Aussies” in low flight during a simulation, initially scattered and then fired on the aircraft with Stinger Missiles. A follow-up article, again from the Defence Systems Daily, clarified the issue. The clarification stated that although the developers at DSTO used the same code as the Stinger detachment within the Modular Semi-Automated Force (ModSAF) software, they had not yet modified the weapons code and the mutant marsupials did not fire missiles but rather the default object – large multi-coloured beachballs. I responded to my friend that although in simulation anything is possible, regardless whether it was missiles or beachballs, clearly there is a warning for pilots. When and if they encounter the same “hostile” environment, leaving the kangaroos alone might prove to be a smart survival technique. But in conclusion, this story illustrates a valid point for deliberation when comparing training systems with entertainment systems.

Simulation systems intended to emulate the real-world environment or system for training in the Defence Force utilise the same underlying technology as distributed interactive entertainment games. As presented earlier, attaining a desired level of realism in an emulation of a real-world environment, is only one omitted influence, missed assumption or misinterpreted requirement away from not meeting expectations. Typically the impact of incorrect representation or omission of required interaction details is not as obvious as beachballs being fired at aircraft. In reality the presence of incorrect behaviour can manifest itself in many different areas of a simulation. Small inconsistent behaviour interactions are far more subtle and difficult to capture and their impact difficult to analyse in terms of degradation of training effectiveness over the course of a specific simulation. A distributed interactive training system relies on technical and perceptual accuracy in order to provide the capability of reaching a defined level of training competency within the emulated environment.

By comparison though, a game rewards competency without recognition of training requirements yet its base of reality must also embrace the same technical and perceptual accuracy. Games seek to motivate the player through higher scores and access to more complexity. But in actuality both of these applications areas have similarities that require consideration. The training and entertainment community both strive to produce a virtual environment intended to immerse the operator or player in an artificial world as defined by the observable environment, allowable interactions and defined attributes. While each community has different motivating factors and measures of success, both the entertainment and training community have insights that if shared, could benefit each other. Each has the goal to immerse the participant in an interactive artificially created world, which responds to real-time stimuli. Each has a goal of realism. Each provides training whether it is to conquer space aliens, super humans or the enemy and requires a conditioned response on the part of the participant to succeed. Clearly, the focus, motivation and mindset of the individuals emersed in a simulation environment ultimately determine the reality of the applicable domain – game or training. As the fictional character Ender [17] came to realise, the two domains when combined with appropriate insulation from the real world and outside influences, can be merged so that the game becomes the reality.

“Real. Not a game. Ender’s mind was to tired too cope with it all. They weren’t just points of light in the air, they were real ships that he had fought with and real ships he had destroyed. And a real world he had blasted into oblivion.”

References

Author

Michael L. Darby is a lecturer in the School of Electrical and Computer Engineering at Curtin University of Technology in Perth, Western Australia with a principal focus on Modelling and Simulation research. Mr. Darby is also the principal Director and Chief Scientist for North Star IT Solutions Pty. Ltd. Mr. Darby has over 20 years of practical simulation experience in training, analysis, and process engineering.