Assessing The Effectiveness Of Simulation-Based Counter-IED Training

Introduction

Explosive hazards, such as Improvised Explosive Devices (IED) represent a major threat to Coalition forces on overseas operations. For instance, they are responsible for an increasing number of casualties [1–3]. Countering this threat can be challenging, as the tactics employed by insurgents and opposing forces are highly adaptive.

Training is an important way of addressing the threat posed by IED. In order to respond to the adaptive techniques used by insurgents, Coalition counter-IED training also needs to be flexible and adaptive in order to best prepare military personnel to detect and respond to explosive hazards [4–5].

One way the USA and other countries have addressed this training challenge is through the use of simulation-based training [3]. It is perceived to be a safe, cost-effective means of providing a wide range of customizable and repeatable training opportunities. Consequently, the use of simulation as a complement to existing forms of training is increasing in the military [6].

An aspect of simulation that is growing in popularity as a training method is desktop-based simulation. This includes commercial off the shelf computer games originally designed for entertainment purposes, which have been re-purposed as training products, as well as products specifically designed for training [7–9]. Compared to larger-scale simulations, desktop based simulations are cheaper. In addition, they are perceived to generate higher levels of trainee engagement and motivation, especially among younger trainees [8,10].

It is vital that simulation provides effective training; otherwise there is a risk that trainees will not learn the necessary skills [11]. In addition, if simulation-based training is ineffective compared to traditional forms of training, the potential cost benefits of its use are negated. The challenges of designing, conducting, and assessing simulation-based training are considered by human systems integration practitioners (see for example, [12]), to be a human factors issue, alongside more conventional human factors areas such as human factors engineering and ergonomics.

While definitions of ‘effectiveness’ vary, a common framework for evaluating training effectiveness is Kirkpatrick’s model [13]. This contains four levels, 1) trainee reaction, 2) learning during training, 3) improved performance following training, and 4) improved organizational outcomes following training. This model is useful for assessing the effectiveness of training, as it acknowledges the need to look beyond trainee evaluations or learning during training, and consider in addition the extent to which there is transfer of training.

Despite the perceived benefits of simulation-based training, evidence on their effectiveness in military contexts is lacking [4,9]. In part this is due to the challenges of conducting empirical evaluations of military training. As noted by several researchers, evaluations are frequently conducted in conjunction with military training courses. As a consequence, military training objectives take precedence over research objectives, and the experimenters’ control over the design and conduct of the activity is lessened [14,15].

Only a small amount of research has examined simulation-based training for explosive hazard detection, either for vehicle crews or dismounted combatants [2,5,10]. All of these studies used the desktop simulation Virtual Battlespace 2 (VBS2), which was also used in this study. These studies have generally been conducted using the first two levels of the Kirkpatrick [13] model.

For instance, in [10], US Marine Corps soldiers completed a series of vehicle convoy scenarios in VBS2, including reacting to an IED. Training outcomes were measured through knowledge tests, surveys, and a live activity, with baseline measures taken prior to training. A control group completed conventional training comprising “sandbox” training and rehearsals using a static vehicle. The former involved using small objects to represent vehicles and military personnel in model terrain, to rehearse and discuss actions in response to IED events. Following training, the experimental group’s attitudes towards simulation were significantly more positive, and participants’ self-ratings of performance significantly improved, but this did not translate into improved knowledge or better performance in the live activity. This study used the first three levels of the Kirkpatrick [13] model, although evidence of improved performance—level 3—was not found.

In [2], VBS2 was used as the simulation environment for vehicle crew counter-IED trainer for the US Army. This was developed to complement existing methods of counter-IED training, such as PowerPoint slide presentations, static displays, and field exercises using inert IEDs. The researchers perceived that the addition of VBS2 training provided greater levels of immersion, realism, and variety of training scenarios in comparison to existing training methods. User feedback on the VBS2 system was positive, although no empirical evaluations were conducted. This study assessed training effectiveness at the first level of the Kirkpatrick [13] model only.

In [5], the effectiveness of VBS2 training and conventional sandbox training was compared. Personnel from the Canadian Army completed IED training using either VBS2 or sandbox. Training outcomes were measured through a knowledge test, self-reported ratings of individual and team confidence, and Subject Matter Expert (SME) assessment of team performance during an activity conducted in VBS2. The study showed that levels of improvement in knowledge and self-reported ratings of confidence were greater following VBS2 training than sandbox training. This represents evaluation at levels 1 and 2 of the Kirkpatrick [13] model. Results of the SME assessment—corresponding to level 3 of the Kirkpatrick model—were not reported.

These results are generally consistent with other studies of military simulation-based training, in terms of both the way in which effectiveness was evaluated, and the study findings. For instance, in [16], six studies of simulation-based training in the military are reviewed. In two studies, simulation-based training was rated as more effective than conventional forms of training. In one, conventional training was rated more highly than simulation-based training. In the remaining three studies, either there was no difference between simulation-based training and conventional training, or no comparisons were made. In addition, [17] reviewed a number of studies of military simulation-based training, and concluded that the evidence was inconclusive, in part due to methodological limitations. In contrast, a meta-analysis reported in [18] found that a number of military training simulations, including a combat vehicle simulation, had a significant outcome on self-reported transfer of training.

There are three broad conclusions that can be drawn from these studies. Firstly, there is a lack of research examining the effectiveness of simulation-based counter-IED training. Secondly, where such studies have been conducted, there is a tendency to use as indicators of effectiveness the lower levels of the Kirkpatrick [13] model, such as trainee ratings or performance during training, rather than the higher elements such as performance and organizational improvements after training. Finally, where performance following training has been assessed (see, for example, [5,10]), these results are either not reported, or show no significant difference compared to conventional forms of training.

Simulation-based training is a relatively new form of training within the Australian Army, compared to more traditional forms such as Rehearsal of Concept (ROC) drills, also known as sandbox drills. However, the Australian Army has recently introduced simulation-based training using VBS2 for vehicle crews [19]. Anecdotal evidence from instructors of training outcomes including positive responses from students and improved student performance in training courses have been reported [20], which led to a request for a formal study. This paper describes the study, which aimed to compare the effectiveness of simulation-based training conducting using VBS2, and traditional training conducted through ROC drill, for vehicle crews in the Australian Army.

Method

The study was a longitudinal study, conducted over 12 days. It used a naturalistic crossover design, where participants completed both types of training methods: ROC drill, and simulation-based. Training effectiveness was measured in two ways:

• Through questionnaires measuring participants’ attitudes towards the usability and training effectiveness of ROC drill and simulation-based training; and
• By comparing the assessment outcomes from a field exercise for participants conducting ROC drill on the day of assessment with the assessment outcomes for participants receiving simulation-based training on the day of assessment.
• These two measures corresponded to the first and third levels of the Kirkpatrick model [13]. Due to the way the course and assessment were structured, it was not possible to test at the second level—performance improvement during training.

The study received approval from an ethical review panel, and was conducted in accordance with Australian guidelines on human research [21].

Participants

Participants in this study were 16 enlisted personnel from the Royal Australian Corps of Transport (RACT) of the Australian Army, who were enrolled in a promotion course. Nine participants were of the rank Lance Corporal and the remaining seven were Private soldiers. Fourteen participants were male, and two were female. All participants gave informed consent to participate.

Participants’ ages ranged from 21 to 49 years, with a mean age of 33 years. Based on data from all 16 participants, the average length of military service was 7.1 years, with an average length of experience driving military vehicles of 6.2 years. Eight of the participants had served overseas in areas of operation including East Timor, the Solomon Islands, Iraq, and Afghanistan. Six of the participants had previously used VBS2 during recruit training, in barracks, or for pre-deployment training.

Materials

VBS2 is a first-person perspective desktop computer simulation, produced by Bohemia Interactive Simulations. Unlike computer games produced for entertainment purposes, VBS2 is intended to be, and marketed as, a training product [22]. Key differences to games produced for entertainment are that VBS2 has more accurate modeling of weapons effects and the effects of injury and fatigue. The injury and fatigue models mean that a player’s avatar experiences performance degradation as a result of wounding or sustained physical exertion; these are perceived to make VBS2 more realistic and hence more suited for training [23].

As this study was conducted in conjunction with an existing training course, the researchers had no influence over the choice of VBS2 as the simulation environment. However, the use of VBS2 in this study is consistent with its use in the Australian Army for driver training and counter-IED training [20]. Figure 1 shows a screenshot from VBS2, showing the first-person perspective view of a driver of an in-service Australian Army vehicle. To the left of the figure, an Australian Army soldier stands next to an IED.

ROC drill involves using model vehicles to represent a vehicle convoy. Terrain is represented through use of objects such as soil, rocks, or string, as shown in Figure 2.

Measures

Multiple measures and sources of data, both quantitative and qualitative, were used in order to gain a more comprehensive understanding of the phenomena of interest [24,25]. A summary of the day/s on which each measure was completed is shown in Table 1.

With the exception of the summative assessment, which was an existing part of the course, the frequency with which measures were administered was a balance between maximizing the amount of data collected and minimizing the impost on participants. At this stage of the course, the participants’ daily routine encompassed over 14 hours of scheduled activities plus overnight picket duty, with the expectation that they would complete additional written work and reading as required outside formal instruction times. Hence, there was a strong desire from the research team and the course staff to minimise additional workload generated by completing measures.

The background questionnaires measured basic demographic details such as age, rank, years of military driving experience, and previous experience with VBS2.

Usability was measured in two ways, firstly through an existing usability measure, the System Usability Scale (SUS) [26,27], and secondly through a training effectiveness questionnaire, which was developed by the researchers based on the Unified Theory of Acceptance and Use of Technology [28].

The SUS contains 10 questions, such as “I think that I would like to use this system frequently”, and “I thought the system was easy to use”, which are responded to on a 5-point Likert scale, with ‘strongly agree’ and ‘strongly disagree’ as the anchors [26,27]. A percentage score is obtained, with higher scores indicating higher usability levels. For this study, references to ‘the system’ were replaced with references to VBS2.

The SUS provides a single score measure of usability, as a percentage, which can be useful in quickly conveying changes across time; however it was not intended for the individual decomposition of questions, which thus reduces its ability to provide more specific guidance on respondents perceptions of a system.

The SUS was used to assess VBS2 only, and not ROC drill. ROC drill is a frequently used training tool, which most, if not all, participants would have used previously. In contrast, most participants were unlikely to have used VBS2, given the relative recency of its use as an instructional tool. Hence, changes in overall usability were predicted for VBS2 but not for ROC drill. On this basis, and in light of the need to minimise additional workload on the participants, ROC drill was not assessed through the SUS. However, it was assessed using the training questionnaire.

The training questionnaire provided the following decomposition of usability perceptions: attitudes towards computers in general, effort expectancy, performance expectancy, behavioural intention, and social influence [28]. The questionnaire was responded to on a five-point Likert scale, with either ‘strongly satisfied’ and ‘strongly dissatisfied’ or ‘strongly agree’ and ‘strongly disagree’ as the anchors.

Three versions of the questionnaire were developed. There were slight changes to wording and tenses of some questions between versions, to reflect the fact that participants’ exposure to ROC drill and VBS2 would increase over the course of the study. For instance, the first questionnaire asked participants to respond to the statement “Using this training method will improve my ability to learn”. This was changed to “Using this training method improves my ability to learn” in the second version, and “Using this training method has improved my ability to learn” in the third version.

Two self-efficacy questionnaires were used, one based on the General Self-Efficacy (GSE) scale [29], and the other based on the Computer Self-Efficacy (CSE) scale [30]. The GSE scale contained 10 items that provided a global view of self-efficacy. It included statements such as: “I can always manage to solve difficult problems if I try hard” and “I can usually handle whatever comes my way”. Participants responded on a four-point scale, rating the extent to which these items were true of them (1 = Not at all true, 2 = Hardly true, 3 = Moderately true, 4 = Exactly true).

It has been argued, (see, for example, [31]) that the concept of self-efficacy is domain specific. To address this critique, training method specific self-efficacy measures were developed. These were developed using the CSE, which has previously shown that that computer self-efficacy is linked to participant assessment outcomes [32].

The specific self-efficacy measures based on the CSE included items that were scored the same as for the general items. These measures were specific to each training method. Statements for the ROC drill version included “I feel confident planning my travel route using the ROC drill procedure” and “I feel I can adequately task section members using the ROC drill method”. Statements for VBS2 included “I feel confident in understanding the keyboard functions within VBS2” and “I feel confident logging into VBS2”. The VBS2 version contained 10 items, and the ROC drill version contained 6 items.

The Repertory Grid Technique (RGT) originated from the psychological study of personality and assumed that objects we interact with are made up of a collection of similarity or difference dimensions referred to as personal constructs. While these personal constructs tell us something about the individual, they also reveal something about the object and its attributes. From a design perspective, it is the differences/similarities of the “object” that is of interest to the researcher and RGT is a formalised method for extracting these personal constructs with minimal researcher influence to prevent bias [33].

While the RGT is normally conducted as an individual process, it was modified during the trial to facilitate group discussion. The first step of the RGT activity involved presenting three objects to participants to practice the technique. The objects car, train and donkey were written up on the whiteboard and participants were asked to think of one way in which two of these objects were similar to each other, but different from the third. An example, “the car and train both have wheels, while the donkey has legs” was provided by the facilitator and participants were then asked to present any additional responses. Answers were talked through and written on the whiteboard and once participants had exhausted their findings and acknowledged a good understanding of the task, they were asked to look at the list of training methods used to support their training requirements. This list was provided on their worksheets and participants were asked to include any further training methods they might have experienced. It was requested that individuals rank order these training methods, with a one indicating the method they used most often. The facilitator then presented the following three training methods: VBS2, ROC drill and Classroom training with PowerPoint and asked participants to again write down how two of these objects might be similar to each other, but different from the third. Five to ten minutes was provided to complete the task and participants were then asked to substitute the Classroom training with PowerPoint with their highest ranked alternate training method and repeat the similarity/differences task. The facilitator then asked participants to identify the three comparisons they thought were most significant and invited each participant to share one of these comparisons and their personal constructs with the group. Findings were talked through as a group and the RGT activity was complete when participants had no further examples they wished to share.

The summative assessment was the formal evaluation of the participants’ performance on the training course. It comprised a number of assessment criteria relating to the participant’s ability to command and control the convoy in the field. Each criterion was assessed as Yes, No, or Not Applicable. Certain criteria were identified as essential; failure to complete one or more of these resulted in a rating of Not Yet Competent (NYC). Failure to complete three or more of the remaining criteria also resulted in a rating of NYC, otherwise the participant was assessed as Competent. While the lack of granularity in this assessment is acknowledged, this was an area where the research requirements were subordinate to the course requirements; the researchers did not have the ability to modify this assessment or introduce additional direct measures of performance.

The study team members attended the whole course, observing the participants as they experienced the variety of training methods in the class room, the simulation suite and in the field. These observations provided contextual understanding that supported interpretation of the data.

Procedure

The study was conducted over 12 days at an Australian Army base while the participants were undertaking a training course on command and control of vehicle convoys. As part of the course, participants had been divided into two 8-person sections, with the roles of section commander and second-in-command (2IC) rotating among participants on a daily basis. During the study, questionnaires, measures, and assessments were conducted as per the schedule in Table 1. Due to the course requirements, there were instances where participants were unavailable to complete questionnaires or measures. In the results section, N is reported for all results.

On Day 1, participants completed familiarization training on VBS2. This was self-paced, completed individually, and introduced participants to the keyboard and mouse commands required to interact in the virtual environment (such as walking, using weapons, and entering vehicles).

From Days 2–10, each morning, one section took part in simulation-based training, while the other section took part in ROC drill. The type of training received by each section alternated from day to day—for example, a section receiving simulation-based training on one day would conduct ROC drill the next day, and simulation-based training the following day.

During VBS2 training, participants were grouped in pairs, with one participant acting as vehicle driver, and the other as co-driver. Each pair was seated at adjacent computers, with the driver’s computer equipped with a Logitech® steering wheel and foot pedals. Within the VBS2 simulation environment, each pair crewed an in-service Australian Army vehicle. Figure 3 shows a pair of computers used for VBS2 training (note that Australian vehicles are right-hand drive).

The objective during each VBS2 training session was to drive in convoy to a designated location. The section commander or 2IC briefed the section using the same procedure that would be used prior to a live convoy operation; the route was identified, along with relevant information such as recent insurgent activity, and actions to be taken in response to any incidents.

After this brief, the convoy drove in the virtual environment. During the scenario, an explosive hazard such as an IED was encountered, which the section was required to respond to using established tactics, techniques and procedures. The nature of the explosive hazard varied from day to day, and was determined by the staff member supervising the training.

ROC drill training was delivered by the participant acting as section commander. He or she moved the model vehicles through the simulated terrain, with section members discussing and rehearsing responses to encountering an explosive hazard.

From Days 2–10, following the morning’s training session, an assessment activity was conducted each afternoon. The assessment took place on a military training range. Each section drove in a convoy of four in-service military vehicles, with two participants per vehicle, travelling on a predefined route towards an identified end point. The two sections followed different routes, and had different endpoints, so they did not travel together.

At some stage during this drive, the section encountered a potential explosive hazard situation. The nature of the situation varied, and was determined by the course staff in advance. In general the complexity of the scenario increased from day to day. On encountering the explosive hazard, the section was required to respond appropriately. In-service personal weapons, blank ammunition, smoke grenades and signal flares were used to enhance the realism of this activity.

Throughout the activity, the participant acting in the role of section commander was observed and formally assessed. This was carried out by military personnel from the course staff, using the competency-based summative assessment. Any participant who was assessed as NYC was reassessed at a later stage of the course. This procedure was repeated over the duration of the study until all participants had acted in the role of section commander and had been assessed. It is acknowledged that this means that the participants assessed towards the latter stages of the course may have acquired more experience than participants assessed earlier in the course. The timing of the assessment schedule was outside the researchers’ control. However, it is unlikely to have affected results, as the assessment scenarios towards the end of the course tended to be more difficult than the earlier scenarios. This was done, in part, to balance out any advantage participants may have gained through being assessed later in the course.

The summative assessment was the only external assessment of participants’ performance. Although the researchers were permitted to watch the assessment activity, due to safety requirements they had to remain at a distance. In addition, they were not able to listen to radio communications. Hence, while the researchers were able to gain an overview of the activity, they were unable to observe in sufficient detail in order to conduct their own objective performance assessments.

Results

An alpha level of 0.05 was used for all significance testing. Exact probability values are reported, except where p < 0.001. Effect sizes are described using criteria outlined in [34]. Error bars in figures indicate the Standard Error of the Mean.

Quantitative measures

Results from the summative assessment data were based on N = 16, where eight participants received VBS2 training on the day of assessment, and eight participants received ROC drill training on the day of assessment. Only one participant was found NYC following assessment. This participant, who was subsequently found Competent on a later assessment, had taken part in VBS2 training on the day of assessment. Testing using Fisher’s exact test indicated that the difference in assessment outcomes as a function of type of training received on the day of assessment was not statistically significant (p = 0.99). When the individual criteria on the assessment were examined, there were no apparent trends or differences as a function of the type of training received on the day of assessment, hence no further analyses were conducted on the summative assessment data.

SUS scores at each test time are shown in Figure 4. N = 14 at Day 1, and N = 15 at Day 11. A repeated measures t-test showed that the decrease in scores from Day 1 to Day 11 approached levels of statistical significance, t (13) = 1.92, p = 0.08, with a moderate effect size (d = 0.59).

Results from the performance expectancy dimension of the training questionnaire are shown in Figure 5. The results are based on N = 16 for ROC drill Day 1, VBS2 Days 1, and 11, and N = 15 for ROC drill Days 3 and 11 and VBS2 Day 3. Responses have been collapsed into three categories: Agree (combining ‘Strongly Agree’ and ‘Agree’), Neutral, and Disagree (combining ‘Strongly Disagree’ and ‘Disagree’).

The figure shows that the majority of participants agreed with statements relating to ROC drill and to VBS2, although there were some minor fluctuations over time. A 2×3 Chi-square analysis conducted on the number of ‘agree’ responses showed no significant differences, χ² (2) = 0.04, p = 0.98.

Across all dimensions, the pattern of findings was similar: the majority of participants responded positively towards both ROC drill and VBS2, with only minor fluctuations over time. Chi square analyses conducted on the remaining four dimensions showed no significant differences, with p values ranging from 0.60 to 0.97.

As the VBS2 and ROC drill self-efficacy questionnaires contained a different number of questions, scores were converted to percentages to enable comparisons. The general and specific self-efficacy scores for each test time (Day 3, Day 11) and each simulation type (VBS2, ROC drill) are shown in Figure 6. N = 15 for each test time and simulation type, except ROC drill Day 11, where N = 16.

A 2×2×2 repeated measures ANOVA conducted on these data showed a significant effect of training type, F (1,15) = 18.66, p < 0.001, and significant interactions between training type and time, F (1,15) = 7.74, p = 0.01, and training type and self-efficacy type, F (1,15) = 21.58, p < 0.001. The interaction of time, training type, and self-efficacy type approached levels of significance, F (1,15) = 3.75, p = 0.07.

Post-hoc testing using paired samples t-tests confirmed that levels of general self-efficacy did not differ significantly as a function of test time of training type (p values ranged from 0.379 to 0.900). However, there were significant differences between levels of specific self-efficacy for VBS2 and ROC drill at both Day 3, t (15) = 5.44, p < 0.001, and Day 11, t (15) = 3.44, p = 0.004. Effect sizes were very large and large, respectively, (d = 2 and d = 1.32).

Qualitative measures

Results from the repertory grid activity were derived from 16 participants. During the group discussion, participants highlighted a range of advantages and disadvantages of ROC drill and VBS2 as well as more general comments about the mix of training methods. The advantages of ROC drill included its ease of use, the ability to use it anywhere, and that it is well established. The disadvantages include the difficulties in representing complex training scenarios, and limited immersion. Training delivered using VBS2 was seen as more structured and provided feedback. It also provided greater immersion. However, it requires more resources. VBS2 was considered a valuable supplement to ROC drill and field exercise. The participants also indicated that computer based simulation should supplement but not replace ROC drills and field exercises, and identified areas where each could be used effectively. For instance, ROC drill could be used to learn procedures, with VBS2 used to rehearse them.

Discussion

In order to address the risks and challenges IEDs pose to Coalition forces on overseas operations, the use of desktop-based simulation has been proposed as one way to provide effective training. This is an area where only a small number of studies have been conducted [2,4,10]. Of these studies, the majority [2,4] relied on subjective trainee reactions, rather than examining improved performance following training. In [10], performance following simulation-based training was not found to be significantly better than performance following conventional training.

This study aimed to build on previous research and compare the effectiveness of simulation-based and traditional counter-IED training by:

• Measuring participant attitudes towards the usability and training effectiveness of ROC drill and simulation-based training, and
• Comparing assessment outcomes.

These two measures corresponded to the first and third levels of the Kirkpatrick [13] model of assessing training effectiveness. Using multiple levels of this model provides a more thorough examination of training effectiveness than simply relying on attitudes and reactions alone.

Overall, the results from this study suggest that participants are more negative towards simulation-based training than they are towards conventional training yet, the two types of training did not result in different assessment outcomes. There are a number of explanations and interpretations for these findings, and some methodological limitations. These are discussed in detail in the following paragraphs.

The decline in usability scores towards VBS2 may be due to changes in the way it was used prior to each questionnaire administration. On Day 1, when the SUS and usability questionnaires were first completed, participants had completed individual, self-paced familiarization on VBS2. In comparison, on subsequent days when the questionnaires were administered, participants had been undertaking collective training, with more complex tasks being performed. This highlights the need for repeated measures of usability, as first impressions may not persist.

In addition, the decline in specific—but not general—self-efficacy scores resulted in part from a small number of participants consistently recording strong negative views and questionnaire responses towards VBS2. These participants found using computer-based training challenging and frustrating, and their responses became more negative across the study. This was observed anecdotally by the researchers as well as being reflected in the questionnaires. Given the small sample size, strong negative responses from a minority of participants had marked effect on means. While their views do not reflect those of the majority, they are valid, and counter the belief prevalent among some researchers, (see, for example, [8,9]) that desktop simulations almost universally produce high levels of trainee engagement and motivation.

The decline in specific, but not general, scores is also consistent with previous research suggesting that self-efficacy is domain specific (see, for example, [31]). This indicates that while participants were generally confident in their abilities, and confident in their ability to use ROC drill, they lacked confidence in using VBS2. It is difficult to determine if this is simply a function of less exposure to VBS2 than ROC drill, or a function of VBS2’s potentially greater complexity, or if there is another contributing factor. In addition, it is difficult to determine if this difference would have decreased over time. This is an area for future research to address. While the modified version of the CSE questionnaire used in this study was not validated, the minor changes are unlikely to have affected results. The questionnaire continues to be used in research into the effectiveness of simulation, and one area that may be addressed in future research is a formal validation of the questionnaire.

The RGT identified strengths and weaknesses of each training method, and areas where each could be effectively used. Comments on the areas where VBS2 is most effective for training have been made by participants in previous studies (see, for example, [35]), but there does not appear to have been any empirical examination of whether or not VBS2 is more appropriate for teaching some types of skills above others. This is an area for future research to address.

A lack of significant differences between assessment outcomes, which is in line with previous research (see, for example, [10]), does not necessarily mean that the two types of training were equally effective. Rather, it reflects that as a consequence of the training schedule and performance measures utilized in this study, no differences could be detected. The competency-based scale had no granularity; students were either Competent or they were not, with no degrees in between. While it is possible that assessors biased their ratings so that neither training method appeared superior, this is considered unlikely; the assessment staff had limited or no involvement in the conduct of ROC drill and VBS2 training, used clearly defined criteria in conducting their assessment, and were experienced military personnel whose focus was on ensuring that participants were fairly assessed.

A stronger contributor to the lack of significant differences in assessment outcomes is likely to be the fact that all students received both ROC drill training and VBS2 training. This meant that it was difficult to quantify the relative contribution of each type of training. As discussed earlier in the paper, the way that training was conducted, and the way that performance on the live activity was assessed were not within the researchers’ purview to shape, given that the course outcomes affected students’ promotion prospects, and data collection was at all times subordinate to course requirements. This is one of the challenges of conducting research in conjunction with military training courses, as previous researchers have identified [14,15]. However, a follow-up series of experiments are currently being planned. These will be conducted as standalone activities, rather than in conjunction with military training. Consequently, data collection requirements will be paramount, which will allow the efficacy of particular training methods to be assessed.

In conclusion, this study aimed to compare the effectiveness of simulation-based and conventional counter-IED training. It was unable to conclusively demonstrate that one form of training was more effective than another, due to a number of factors including methodological limitations and the challenges inherent in conducting experimentation in conjunction with military training courses. However, it has identified a number of areas for future research that will help identify how simulation can best be used for counter-IED training.

References

Authors

Susannah Whitney holds a PhD in Psychology from the University of Queensland, and is a Cognitive Scientist at DSTO, a position she has held since 2007. E-mail:susannah.whitney@dsto.defence.gov.au Phone +61 8 7389 4641, Fax +61 8 7389 4193

Justin Fidock holds a PhD in Information Systems from RMIT University and a Master’s degree in Psychology from the University of South Australia. He has been employed at DSTO since 1988, and is currently a Senior Cognitive Scientist.

Nicole Ferguson has an undergraduate degree in Psychology and 20 years experience as Australian Army soldier. She began work at DSTO in 2010 as a Cognitive Scientist.

.