Demonstrating the Benefits of Simulation in a Military Environment

Introduction

Simulation, in its widest sense, is generally understood to be “an imitation of reality” [1, p.868]. Simulation technology has been used extensively as a problem-solving technique in many different spheres of human endeavor for many centuries, including design, construction, decision support, and in various specialised domains, including the military. More recently, computer-based simulation techniques have been used extensively in the class of applications known as decision support systems. This paper addresses the application of simulation in the military environment.

The modern military environment is characterised by budgets that are shrinking in real terms, while weapon systems and vehicles are increasingly expensive to procure and operate. In addition, safety and environmental issues have meant that realistic training using actual systems is harder to achieve. In an effort to overcome these obstacles, most modern military forces have embraced simulation as an effective and cost-efficient means of preparing and training for military operations.

While there is now a well-developed military simulation industry, and strong anecdotal evidence of the benefits of simulations abounds, solid evidence of benefits, and monetary savings accruing, is problematic. Most military forces have embarked on a programme to introduce simulation technology into many facets of training and operations, without the evidence of clearly proven benefits of doing so. This has made it hard to justify both capital and operating expenditure.

This paper provides a basis for demonstrating the benefits of simulation in a military environment. The paper:

• provides background information on current initiatives to define the benefits of simulation in military forces,
• suggests a possible methodology or framework for measuring the benefits of simulation, and
• provides brief details of existing evidence of the benefits, as realized by some military organisations.

Military applications of simulation

There are three main categories of simulation systems applicable to the military domain, as follows.

Constructive simulation

This branch of simulation involves computer models that represent the actions of people and/or equipment. This category includes the so-called “war games” systems, which model tactical engagements in real time. Possible applications of constructive simulation include:

• Tactical decision training. Staff are trained in military subjects such as tactics and staff procedures using war games such as JANUS and Tac Ops (Constructive simulations developed by the US Army and US Marine Corps respectively), and procedural trainers, essentially CPX (command-post exercise) drivers. Both individual and collective training is possible, without the need to deploy formations and units into the field for FTXs (field-training exercises), or, in fact, without the need to deploy individuals or headquarters into the field for traditional TEWTs (tactical exercises without troops).
• Operational analysis. Simulations are used to conduct trials of proposed force structures and weapon systems prior to development or acquisition of such systems.
• Command and control decision support. Constructive simulations can be used as part of, or in conjunction with, command and control systems, for the following purposes [2]:
• Situational awareness, including terrain analysis and visualisation, and analysis of possible enemy courses of action (COAs).
• Design and analysis of own COAs.
• Logistics planning and feasibility checks.
• Mission planning and rehearsal and After Action Review (AAR).

Virtual simulation

This involves real people engaged with “human-in-the-loop” simulators, usually representing operator interaction with individual weapons systems, vehicles or aircraft.

Live simulation

The actions of real people (troops) and/or real platforms/equipment are connected into the represented environments. The real troops frequently utilise weapons engagement simulators; for example, lasers which indicate whether a “kill” is achieved when a weapon is fired (currently applicable to direct fire weapons only).

Integration of simulation categories

Modern distributed simulation technology is enabling the integration of all the above categories. This allows a “battle lab” capability, which is able to provide a variety of capabilities, including training, command and control decision support, and operational analysis (to model the effects of new organisations or weapon systems). An integrated system of various types of simulations is often termed a “synthetic environment” (SE).

Assessing the benefits of simulation—recent and current initiatives

A major open source of information on assessment of the benefits of simulation is the US Defense Modelling and Simulation Office (DMSO) web site at URL http://www.dmso.mil. The DMSO sponsors a Special Interest Area (SIA) known as impact assessment (IA). The objectives of the IA activity are as follows:

• “…to provide DoD-wide visibility into studies of the impact of M&S;
• to assist the DMSO in identifying best practices for determining and reporting data on the utility of M&S use (e.g., empirical return-on-investment and cost-effectiveness data);
• to assist the DMSO in determining productive applications of M&S tools and for making M&S investment decisions.” [3]

Somewhat dauntingly, [3] cautions:

“The lack of a single document or collection of documents outlining common practices and procedures for decision makers looking for assistance when making M&S decisions and investments adds to an already complex problem. Additionally, few methods exist for answering questions about potential M&S use and benefits. Finally, no formal processes exist for the collection of empirical data. Therefore, answering questions regarding the effectiveness and the cost of using M&S is nearly impossible.”

In spite of this apparent pessimism, the DMSO site contains many valuable documents that assist in developing a framework for the demonstration of benefits.

The foundation of IA in US DoD is encapsulated in the DoD M&S Master Plan [4]. Objective 6 (Share the Benefits of M&S) states:

“The optimal use of M&S across the Department of Defense will not occur unless the positive (and negative) impacts and cost-effectiveness of M&S are documented and communicated. The DoD Components must educate potential user communities on the existing and expected benefits of M&S employment so that they may make informed investment decisions.” [4, p. 2-8]

Further, sub-objective 6-1 (Quantify the Benefits of M&S) states:

“Achieving the DoD M&S vision requires more than just providing technical capabilities. Users must be convinced that M&S support of their operations is both operationally effective and cost effective. Thus, it will be necessary to analyze and demonstrate the use of M&S to support specific functional needs. Quantitative measures of the benefits that clearly demonstrate the impact of M&S must be developed. The results will be disseminated to the Department of Defense, Congress, other government agencies, and industry.” [4, p. 4-27]

US Institute for defense analyses (IDA) report

A key document that at least partially satisfies sub-objective 6-1 is a 1996 US IDA report entitled The Utility of Modelling and Simulation in the US Department of Defense: Initial Data [5]. A perceived shortcoming in the conduct of the study is summarised by the following statement:

“While no formal assessment could be accomplished based on the relatively small amount of information gained in a limited period of time, an informal meta-analysis - an analysis of other organizations’ analyses - is included in this report. Meta-analysis has two serious shortcomings: it is based on information often a year or more old, and it is based on information typically biased towards the positive.” [5, p. ES-1]

The findings of the study are briefly summarised as follows:

• There are many applications of M&S to acquisition. Twenty case studies of Target Interaction, Lethality and Vulnerability showed a 30-to-1 return on investments in M&S support for milestone decisions and the Cost and Operational Effectiveness Analysis process. The Army Missile Systems Command reported a total of over $320 million in cost avoidance or savings from ten case studies.
• Training applications of M&S were commonly used and the results were positive. Reporting was thorough on individual skills training, including both cognitive and psychomotor skills. Cognitive skills trainers, typically computer-aided instruction, paid for themselves in five years or less. Psychomotor skills trainers, for example, flight simulators, driver trainers, conduct of fire trainers, and maintenance trainers, were all shown to be cost effective when properly mixed with training on the real equipment. At this level, analysts have well-established theories and experimental methods for conducting analysis. The same is not true for unit training, particularly of high echelon units. The high cost of a joint or combined exercise precludes the repeated, controlled experiments necessary to gather meaningful data on the benefits of additional learning trials. However, multi-million dollar savings are reported when comparing computer-assisted command-post exercises to field-training exercises.
• Although M&S are used extensively in analysis, few reports documenting benefits were found by the Task Force. The effect of M&S on analysis, while real, is problematic to quantify.
• Across functional areas, measures of effectiveness (MOEs) were not universally agreed upon. Analytic frameworks, including MOEs, need to be developed and applied consistently throughout DoD. Frameworks should be unique to each functional area and perhaps even functional sub-areas.
• A formal reporting mechanism does not exist for gathering information, nor do the methodologies exist for objectively assessing the value of using M&S.

In terms of use of simulations in training, [5, p. 9] makes the following observations:

“Individual training is supported most often by stand-alone simulators. These simulators range from simple devices (such as rifle marksmanship trainers) to more complex devices (such as maintenance simulators, tank gunnery simulators, and flight simulators). Simpson et al. [1995] drew these general conclusions about the effectiveness and cost of such simulators:

‘… in aggregate, simulators provide significant beneficial transfer from simulator to aircraft at a median operating cost of about one-tenth of an aircraft…Because of their scope, the body of studies probably provides the strongest case for the value of any type of simulation.

Students trained using maintenance simulators perform about as well as those trained with actual equipment, but simulators cost a fraction…of the equipment…where time to train was reported, training with simulators took…less time than with actual equipment.’ ”

Bearing in mind the caution about the possibility of information “being biased towards the positive”, the report does provide at least some evidence of the benefits of simulation.

US Army benefits initiative (BI)

The US Army has also embarked on a program to demonstrate the benefits of simulation. The program is described at [6] and [7]. The program was initiated as a result of a Deputy Chief of Staff (Operations) Directive which:

“ … directed AMSO [Army M&S Office] to coordinate the Army’s efforts to develop and implement a definable process by which quantitative and qualitative M&S benefits are captured enabling our better understanding, justification, and use of M&S across the Army.

… AMSO will develop a comprehensive M&S benefits methodology (or methodologies) which will provide a factual basis to answer M&S benefits questions.” [6]

Phase 1 of the program developed a framework from which the US Army can assess, and in part quantify, the benefits of simulation across multiple usage domains. The only apparent application of the framework to date is a Phase 2 report that evaluates contenders for high echelon-level constructive simulations (war games). The framework is described in the next section of this paper.

A framework for assessment of simulation benefits

A well-known text on Information Systems states:

“ ... the success of an information system should not be measured only by its efficiency in terms of minimising costs, time, and the use of information resources. Success should also be measured by the effectiveness of information technology in supporting an organisation's business strategies, enabling its business processes, enhancing its organisational structures and culture, and increasing the customer and business value of the enterprise.” [8, pp. 30-31]

So, an initial question is: what do we want to measure—efficiency (particularly monetary savings) or effectiveness, or both? Also, benefits can be either quantifiable or non-quantifiable. In the case of efficiencies, benefits can normally be quantified (for example, in monetary or time-saving terms), but for effectiveness, quantification is usually more difficult. Further, additional benefits of simulation that are often quoted involve enhanced safety and/or risk reduction. Again, such benefits are difficult to quantify.

According to [6], benefits of simulation can be characterized using four basic criteria, namely: better?; cheaper?; faster?; or only way?. The definitions are:

• Better. The quality of the product or the quality of the process employed is improved through the application of M&S.
• Cheaper. The total cost of the product or process is reduced through the application of M&S.
• Faster. The period of time from task initiation to task completion is reduced through the application of M&S.
• Only way. M&S provides the only means of accomplishing an event that would otherwise be impractical, dangerous, or prohibitively expensive.

As first suggested by Elmaghraby [9], simulation applications can be decomposed into five basic “functions”, or uses of simulation, namely: Aid to Communicating; Aid to Experimenting; Aid to Predicting; Aid to Thinking; and Aid to Training/Instructing. These five “functions”, or uses of M&S, are defined as follows:

• Aid to communicating. The use of M&S in helping to visualize concepts, make ideas more comprehensible, illustrate findings, or demonstrate important cause and effect relationships.
• Aid to experimenting. The use of M&S to plan, rehearse, augment or conduct controlled experiments in situations where direct experiments would be dangerous, impractical, or prohibitive in cost.
• Aid to predicting. The use of M&S to predict the behaviour characteristics of the modelled entity. Entities include, but are not limited to the environment, processes, and physical systems.
• Aid to thinking. The use of M&S to help organize and sort out hazy concepts and inconsistencies.
• Aid to training/instructing. The use of M&S as training and instruction aids.

The US Army BI [6] combines these two taxonomies to produce a framework for benefit assessment as shown at Figure 1.

At this time, training is generally the biggest application of simulation in military organizations. Based on [6], suggested measures of effectiveness (MOE) for the training function are outlined in the next section. Measures could also be derived for the other functions as required.

Suggested measures for effectiveness (MOE) for simulations used in training

Better

The quality of the product or the quality of the process employed is improved through the application M&S:

• Improvement in task proficiency measured by average/qualification scores.
• Simulation realism (creating a realistic number of effects with fidelity).
• Total assessment of trainee progress.
• Increase in performance.
• Reductions in individual remedial training.
• Reduction in number of live rounds fired to qualify.
• Increase in the number of first time qualifiers.
• Reduction in time to complete task.
• Improvement in task proficiency.
• % of skills realistically replicated (supported).
• % of all skills replicated (supported).
• % of organic C4I systems energized.
• % of sensors replicated.
• % of realistic sensor replication.
• Increased number of environmental conditions included.
• Increased number of threats included in operational environment.
• Amount of data stored and accessed.
• Additional number of support hardware and man-in-the-loop interactions supported.
• Increase in user confidence.
• Number of negative environmental affects such as noise, blast, contamination, manoeuvre damage to fragile ecosystems avoided.
• Number of additional (distributed) training/exercise locations included in training event.
• Increased number of training events included in training scenario.
• % increase in accomplishing required training/training event(s).
• % increase in productive training time (reduced dead time).

Cheaper

The total cost of the product or process is reduced through the application of M&S:

• Reduction in travel expenses attributed to distributed training, measured in current year dollars.
• Cost savings of using new methods.
• Reduction of personnel required to do task.
• Cost savings associated with completing tasks early.
• Number of physical mock-ups or prototypes that are required.
• Reduction in the cost of achieving a defined level of proficiency.
• Reduction in the number of rounds to qualify (cost savings).
• Reduced demand on management, facilities, and personnel resources (that is, less time, number, amount, and so on).
• % reduction in, or number of FTXs avoided (reduction in operational tempo by reducing the need for FTX training).

Faster

The period of time from task initiation to task completion is reduced through the application of M&S:

• Reduction in the time required to bring students up to standards, measured in days.
• Cost savings for fewer training days.
• Reduction in time to train.
• Reduction in time to complete task.
• Reduction in set-up or preparation time.

Only way

Models and simulations provide the only means of accomplishing an event that would otherwise be either impractical or prohibitively expensive:

• Number of training events that could not otherwise be supported.
• Additional simulated training environments that could not be captured in FTX (that is, nuclear, biological, chemical, environmental extremes, and so on).
• Increased number of functions that can be dispersed/distributed.
• Increased number of areas addressed that are difficult/impossible to physically test due to cost/time/manpower or risk.

Existing evidence of the benefits of simulation

In many situations, it may be more cost effective to utilise evidence provided by other military organizations than to embark on an in-house program of measuring benefits as per the framework of the preceding section. This section outlines brief details of quantitative and non-quantitative evidence of the benefits of simulation, as reported by US military organizations; it is not intended to be comprehensive.

In additional to the benefits information provided by [3], another US document containing useful information is a US National Training Systems Association (NTSA) article entitled Why Use Simulation?—Return on Investment [10]. Brief extracts from this report that provide evidence of the benefits of simulation are as follows.

Effectiveness

In accordance with US Federal Aviation Administration rules, simulator training alone qualifies a pilot to fly a new airliner for the first time on a revenue flight.

Several studies, relating to the use of simulation in lieu of live fire, indicate that performance with simulation is at least equal to live fire training, but that cost is lower. Soldiers with MACS (Multipurpose Arcade Combat Simulator) training expended fewer rounds during live-fire qualifications and fewer soldiers failed to qualify as compared to those trained using traditional methods. Several studies with the Squad Engagement Training System (SETS) have shown positive transfer from SETS to live fire. Training with the Indoor Simulated Marksmanship Trainer (ISMT) has been demonstrated to benefit live-fire performance.

Efficiency

Studies indicate the relative cost of military flight simulators is 10% of actual equipment when both actual equipment and simulators are already in the inventory. If the simulators must be both procured and maintained the figure is 33%. (These figures could be extrapolated to other forms of virtual simulation, provided appropriate margins for error are defined.)

Tank training. The cost-effectiveness of simulators for tank training is well documented:

• Cost to operate an actual tank is estimated at $75 per mile.
• Cost to operate a Tank Driver Trainer simulator is $2.50 per mile.
• The US Army estimated it saved $2.5M training 2 200 soldiers as tank drivers (period unknown). That is a saving of $1,136 per soldier, which equates to about 15 hours of training per soldier.
• In tank gunnery, the introduction of the Conduct of Fire Trainer reduced the annual expenditure of ammunition from 134 to 100 rounds per tank and improved marksmanship. This resulted in an annual cost avoidance of approximately $29M.

Gunnery training. The Precision Gunnery Training System (PGTS), a trainer for TOW and Dragon missiles, has been demonstrated to be cost-effective, and permits training that would otherwise cost several hundred million dollars per year if actual missiles were used. Table 1 summarises the relative costs.

• “The true test of the training effectiveness of the PGTS-TOW occurred during the Persian Gulf War. The Naval Air Warfare Center Training Systems Division took a number of PGTS Field Tactical Trainers for the TOW to Saudi Arabia and conducted intensive training in close proximity to hostile forces. The Marine Corps TOW anti-armor gunners described the training as being ‘just like the real thing.’ Marines experiencing their first combat indicated that they felt extremely confident to function in the situation. This was attributed, in a large part, to the confidence they had gained by repeated practice with the PGTS TOW trainers shortly before the ground war started. One USMC TOW crew destroyed ten tanks/armored vehicles out of ten missiles fired. Overall, the percentage of first hits by personnel trained on the PGTS exceeded those not trained on the PGTS.” [10]

Risk reduction

In 1989, a B-1 on a routine training mission from Dyess AFB suffered a landing gear malfunction. The nose wheel would not go down. The B-1 was refuelled and stayed aloft for five hours while experts tried to determine the cause. A nose gear-up landing profile was flown in the B-1 simulator, and the data from this simulation was used when the aircraft had to make a landing at Edwards AFB with no nose gear. The landing was successful.

The contribution of simulation and training devices has played a significant role in naval aviation safety. In the 1950s the accident rate for naval aircraft was 20 per 100 000 hours. By 1989 the rate had fallen to 2.39 per 100 000 hours.

Conclusions

Most modern military forces have embraced simulation as an effective and affordable means of preparing and training for military operations. While there is anecdotal evidence of the benefits of simulations, solid evidence of benefits, and monetary savings accruing, is lacking. Many organisations have embarked on programs to introduce simulation technology into many facets of training and operations, without the evidence of clearly proven benefits of doing so. This has made it hard to justify both capital and operating expenditure.

Within the US DoD, initiatives to demonstrate the benefits of simulation are now in progress. These include the US DMSO Impact Assessment (IA) and US Army Benefits Initiative (BI). Open source information available on these programs indicates that progress is being made, in what is a challenging problem domain.

Consideration of the benefits of simulation should include both efficiency (particularly monetary savings) and effectiveness measures. Benefits can also be either quantifiable or non-quantifiable. Further, some perceived benefits of simulation involve enhanced safety and/or risk reduction, which are difficult to quantify. Documents available in the public domain do provide both quantitative and qualitative evidence of the benefits of simulation. These are usually expressed on a case-by-case or system-by-system basis.

The US Army has created a framework for defining benefits of simulation that could be refined for adoption by other military forces. This is based on the following criteria: better?; cheaper?; faster?; only way?.

Measures of effectiveness can then be defined for each simulation domain (for example, training and instruction) grouped together under these criteria.

In some situations, reliance on external evidence may be more practicable and cost effective than establishment of an in-house measurement program. Although somewhat limited, evidence relating to simulation benefits in military organizations does exist, particularly emanating from the US.

References

Author

David Wilton is a Territorial Force officer of the NZ Army Simulation Centre at Linton Camp, NZ. His civilian employment is as a senior lecturer at the Institute of Information and Mathematical Sciences of Massey University, Albany, NZ. He has a BSc degree in physics from the Royal Military College of Australia, a postgraduate diploma in computer science from Victoria University of Wellington, NZ, and an MSc in Information Technology from the University of New South Wales, Australia. Email: D.R.Wilton@Massey.ac.nz, Telephone: 64-9-4140800 ext 9594, Facsimile: 64-4-4418181.