Volume 15, Number 2, July 2012
Technology-Based Training System Design: A Generic Model Of Training Technology Selection, Design And Implementation
- * Human Factors Group, Monash Injury Research Institute, Monash University, VIC 3800, Australia.
- ** Learning Systems Analysis, Hampton, VIC 3188, Australia.
Abstract
Technology remains the key driver for training within military systems. As rapid advances in capability continue, the range of technologies available to training system designers is likely to increase significantly; however, there is no universally accepted, theoretically underpinned guidance for selecting, designing, and implementing new training technologies and the process of linking technologies to training requirements remains ambiguous. Based on a review of the training and human factors literature, a generic model of training technology selection, design, and implementation is presented. The model provides training system designers with guidance on the selection, design, and implementation of new technologies for future training requirements. Each component of the model is discussed with reference to the training system design and evaluation literature. In closing, a series of pertinent future lines of inquiry are discussed.
Introduction
Over the past three decades, advanced technologies such as simulators [1], virtual reality (VR) systems [2], and serious games [3] have been used extensively to train operators in a range of domains, including the military, aviation, healthcare, rail, road transport, and process control. The range of systems on offer is such that training system designers face difficult choices over which technologies to use, and further, how chosen technologies are best designed and implemented in order to maximise training benefit. In addition, rapid advances in technological capability mean that the training technologies on offer are continuously evolving, and the technology available to support training now is likely to be very different to that available in even the very near future.
Despite the traditionally technology-centric approach to training system design in most domains, exactly how best to link training technologies to training objectives remains ambiguous [4]. It is widely acknowledged that a common problem associated with training technology selection, design, and implementation is a lack of consideration of training theory [1,4–7], and there is currently no universally accepted guidance regarding the selection, design, and implementation of training technologies for a particular training objective. When this is coupled with the vast range of training technologies on offer, and continual advances in technological capability, the difficulties associated with designing, developing, and implementing new technology-based training systems become starkly manifest.
The aim of this article is to provide guidance on the selection of training technologies for future military training systems. We present, based on an exhaustive review and synthesis of the training and human factors literature, a model of training technology selection, design, and implementation. Each component of the model is discussed with reference to the appropriate training system design and evaluation literature. In closing, pertinent future lines of inquiry, including validation of the model, are urged.
A model of training technology selection, design and implementation
After many years of developing and studying training systems, we are still unable to reliably link training technologies to training objectives [4]. Although it is clear from the training literature that the selection, design, and implementation of training technologies should be based on a range of important factors, a lack of theoretically underpinned, universally accepted guidance is apparent. The majority of technology-based training system design guidance presented in the academic literature focuses on specific issues in isolation, such as performance measurement [8], simulation [6], and fidelity [9], and it is notable that none deal specifically with training technology selection and implementation. Further, of the guidelines focussing on the overall process of training system design and implementation, the majority are either presented at a high level of granularity or are concerned with one form of training technology only. Salas et al [10], for example, present a set of generic guidelines for the design, delivery and evaluation of training systems, which, although giving useful pointers, do not make explicit reference to the factors affecting technology selection, design and implementation.
Based on our review, and the need for guidance on training technology selection and the ensuing design and implementation of training systems, a model for training technology selection, design and implementation is presented in Figure 1. The model describes the most appropriate process, based on a synthesis of the training literature, when designing new training systems that require the selection and implementation of new training technologies. The model therefore focuses on the fundamental issues that require consideration before, during, and after training technology selection, based on the assumption that a particular training need has been identified.
The model is decomposed into the following four components: 1) training program definition, 2) training technology identification, evaluation and selection, 3) design, test and refine training program, and 4) delivery and evaluation of training program. A summary of each of the steps included in the model, including discussion on the fundamental issues requiring consideration, is given below.
Phase 1 - training program definition
1. Training Needs Analysis (TNA). TNA represents the first step in the design and delivery of any training program and is used to identify where training is needed, what it is that needs to be trained, and who it is that needs to be trained [10]. As such, the outputs of TNA are critical considerations when selecting appropriate training technologies and designing and implementing training systems. Despite its widely acknowledged importance, there is only limited discussion of TNA in the academic literature, and most agree that TNA remains largely art rather than science [7]. It is, however, generally accepted that TNA efforts should comprise three forms of analysis: organisational analysis, task analysis, and person analysis (see, for example, [7,10–12]).
Many training programs often fail due to organisational constraints and conflicts [10], and various studies have demonstrated the importance of the organisation in the effectiveness of training programs (see, for example, [13,14]). The first step of TNA, Organisational analysis, therefore considers the system wide components that are likely to affect the delivery of training programs [12], including the relationships between training objectives and organisational goals, resources, constraints, and support for training transfer [7]. Further, psychosocial characteristics, such as safety culture and manager/supervisor/peer support are analysed.
The task analysis component involves analysing the tasks to be trained so that learning objectives and training content can be specified [10]. It involves describing the task(s) in question and identifying the competencies required to perform them successfully, which are normally expressed as Knowledge, Skills and Attitudes (KSAs). The output describes the work functions to be performed, the conditions under which they will be performed, and the KSAs required for performing them [10]. Salas et al [10] recommend that the task analysis component be broken down into three sub-components: task description, task specification, and task requirements. The task description part involves describing the work functions and the resources required to undertake the job [10]. The task specification step involves describing the tasks that need to be performed and the range of conditions under which they are performed [10] and the task requirements component involves identifying and describing the KSAs required to complete the task(s) safely and efficiently.
Various formal methods exist for the task analysis phase, including task analysis approaches such as Hierarchical Task Analysis (HTA) [15], and cognitive task analysis approaches, such as the Critical Decision Method (CDM) [16]. Further, Salas and colleagues [10] have clearly specified how task analysis outputs should be presented (for example, as KSAs) and how task analysis outputs inform training program design.
The third and final TNA phase, person analysis, focuses on who needs to be trained and what training is required by each trainee [10]. Differing roles, abilities and levels of experience and competence mean that, within complex sociotechnical systems, different individuals are likely to require different levels and/or forms of training, and without an understanding of differing training requirements, training can be ill-defined, and targeted at the wrong level and/or at the wrong individuals [12]. Person analysis seeks to determine whether or not trainees have the requisite competencies (that is, KSAs) for the task or job in question [10].
Overall, despite their importance, the literature is generally vague regarding the role of TNA outputs in the selection and implementation of training technologies. It is clear, however, that there should be a link between TNA outputs and training technology selection, design, and implementation. Organisational constraints, such as resources available (such as finances and personnel) have a large bearing on the technology that can be used and also how it is configured. The tasks to be trained are also important, since the task(s), KSAs, and contextual conditions need to be catered for by the training technology used. The training tasks and scenarios also have significant implications for factors such as the level of fidelity used, the training scenarios used, part versus whole training judgements, and the performance measures and level and timing of feedback used. Finally, trainee characteristics, such as existing competencies, motivation to learn, and attitudes, are also important considerations.
2. Clearly define and specify training aims and objectives. The aims and objectives of the overall training program play a key role in the selection, design and implementation of training technologies. Following the TNA phase, it is important to clearly define and specify exactly what the aims and objectives of the training program are, including what the aims and objectives are, what the desired outputs are, and what competencies (that is, KSAs) are to be developed and maintained as a result of the training program. Clearly defining and specifying the aims and objectives of the training program is critical since it allows the program to be designed appropriately, and informs the selection of an appropriate technology to deliver the training required.
3. Identify the organisational, task, and trainee characteristics and constraints that are likely to affect training program design, delivery and evaluation. Using the TNA outputs, the next step involves identifying the organisational, task, and trainee characteristics and constraints that are likely to affect the design, delivery and evaluation of the training program. In particular, those factors that influence the technology used to deliver the training should be identified and clearly stated. Examples of these factors include available budget (including for initial purchase, set up, and running costs), housing capability, pressures from higher management, the KSAs to be trained, organisational culture, existing training programs and policies, existing trainee KSAs, and trainee motivation to learn, to name only a few.
4. Define training program content. Defining the content of the training program is a key step in the selection and implementation of the appropriate training technology. Training content in this sense refers not only to what activities trainees have to engage in whilst completing the training, but also the scenarios to be used, the KSAs to be trained, the instructional strategies, the feedback required, and the measures of trainee performance to be used. All play a pivotal role in the selection, design, and implementation of the technology used to deliver the training.
Key issues to consider during training program content definition which influence the training technology used include defining the scenarios to be used, the level of fidelity of the training system, choosing to use either part or whole task training, determining what feedback is required and how trainee performance will be measured.
The concept of fidelity is a fundamental issue in training system design, and has been identified as having a strong link to transfer of training [17]. Fidelity refers to how similar a training situation is relative to the operational situation [18]. In a seminal work on simulation fidelity in training system design, Hays and Singer argued that simulation fidelity is “…a two dimensional measurement … in terms of: (1) the physical characteristics, for example, visual, spatial, kinaesthetic, etc.; and (2) the functional characteristics, for example, the informational, and stimulus and response options of the training situation” [36, p. 50]. With regard to training technology selection, design and implementation, the fundamental issue is what level of fidelity is required to maximise the effectiveness of the training system.
Various dimensions of fidelity exist; for example, Liu et al [9] and de Winter et al [18] refer to the plethora of dimensions cited in the literature in the context of training systems. Liu et al [9] point out that most dimensions of fidelity are not mutually exclusive and that a high degree of overlap exists between them. Much research has focussed on the issue, in particular with regard to the level of fidelity required to maximise the level of training transfer achieved by a particular training system. Historically, simulator designers have been preoccupied with achieving maximum fidelity levels, and various authors discuss the widespread but erroneous belief commonly held by training system designers that higher levels of fidelity automatically lead to higher levels of training transfer (see, for example, [1,5,17]). Dahlstrom et al [5], for example, describe how the assumed relationship between maximum fidelity and maximum training transfer currently prevails in the aviation, maritime, nuclear power, medicine, and military domains.
Although there is some evidence that for acquisition of higher-level skills specifically, higher realism may be advantageous, and also that limited fidelity may restrict the range of tasks or sub-tasks that can be trained in a simulator [19], most research indicates that more is not always necessarily better [20]. Evidence suggests, for example, that, depending on the tasks being trained, low fidelity systems can lead to greater transfer of training. Salas et al [20], for example, discuss a range of studies undertaken in the aviation domain that demonstrate a failure of transfer from high fidelity simulations (see, for example, Taylor et al, 1991, 1993; Lintern et al, 1989, 1991; all cited in [20]). Indeed, the evidence from flight simulation training evaluation suggests that higher levels of fidelity have little or even no effect on skill transfer, and that reductions in fidelity can actually improve training (Caird, 1996; cited in [5]). Liu et al [9] concluded that adding more fidelity, particularly in the latter stages of training, produces minimal gain in transfer of training.
The question regarding what the appropriate level of fidelity for a particular training system should be is a multi-dimensional one that requires careful consideration of a range of factors, including the task(s) and KSAs being trained, the training/learning objectives being pursued, and the trainees and their existing skill levels. It is also important to note that different levels of fidelity may be appropriate for different stages and aspects of skill development and high fidelity is not necessarily always best. As Liu et al [9] point out, the answer to this question should always be ‘it depends’. Notwithstanding this, it is generally agreed that high fidelity neither guarantees training transfer nor is required in all cases (for example [1,5,9]). Indeed, training strategy, rather than fidelity level, has been found to be more important than fidelity with regard to training effectiveness [21]. Wickens and Hollands [22] point out that, rather than strive for maximum fidelity, researchers should focus on understanding which components of the training should be made similar to the target task.
Another important training content consideration is whether trainees should practice tasks in their entirety (whole task training), or whether the task should be decomposed and practiced in its component parts (part task training). Part task training involves splitting a task into sub-tasks for presentation to the learner [23], which allows trainees to practice component tasks rather than the whole task, the main advantage being that sub-tasks are easier to learn [24]. Whole task training, on the other hand, involves presenting a complete task to learners so that they are able to practice the task as a single unit [23].
Much akin to fidelity, the issue of whether to use part or whole task training is highly dependent upon the task(s) being trained and the existing skill sets of trainees. According to Lee et al [25] whole task training is more efficient for discrete, continuous tasks such as changing gear whilst operating the vehicle during driving. For serial tasks (where the whole task consists of individual, discrete tasks), however, part task training is more appropriate, particularly since the discrete tasks may represent part of other serial tasks as well. Teague et al [23] suggest that, if the training task can be approximated and understood as a whole, then it should be trained using a whole-task training approach; however, they also urge that the selection of one approach over the other is dependent upon a range of interacting factors, including the trainees themselves and the task being trained.
One of the main attractions of using advanced technologies, such as simulation, for training is the ability to automate large portions of performance measurement, and the provision of guidance and feedback. This not only reduces instructor workload but also provides a tailored training experience [8]. Performance measurement in a training context refers to the process through which behaviours observed within the training environment are translated into a summary statement of trainee performance, which in turn provides the foundation for subsequent performance evaluation [8]. Despite its importance, little attention has been given to the identification of desirable characteristics of performance measures (Johnson et al., 2004; cited in [8]), and one effect of the rapid increases in technological capability has been a tendency to resort to ad hoc measurement strategies [8].
Mangos and Johnston [8] recommend that, in order to ensure validity, training system designers should specify a model of response processes and ensure that the measures adopted are capable of capturing key responses as specified by the model. For criterion relevance, criterion contamination and deficiency should be minimised. To ensure reliability, a model of behavioural consistency across scenario contexts should be hypothesised. To ensure measurement invariance, performance measures that provide meaningful, interpretable metrics across different performance contexts are required. Ensuring objectivity and minimising intrusiveness involves developing measurement strategies that do not interfere with performance. Finally, diagnosticity is achieved via the meaningful aggregation of raw performance data into useful indices for performance assessment. Mangos and Johnston [8] also offer an additional set of general principles for the effective application of performance measures. These include that training system designers should ensure that performance measures are developed based on sound underpinning theory, that measurement affordances should be considered and exploited, and that the usefulness of the measures adopted for evaluating training effectiveness should be ensured.
The provision of appropriate feedback and guidance during training are also critical to the overall levels of training transfer achieved. Increasingly more emphasis is being placed on the role, timing, and form of feedback delivered as part of technology based training. The provision of real-time performance feedback is an advantage of training technologies such as simulators and VR, however, there is concern that the potential learning that is to be gained through such feedback may not be realised or may be negated if the feedback is delivered at inappropriate times, such as when task demands are high. Tailoring feedback to meet the information processing needs of the trainee so as not to compromise learning is therefore a promising avenue of future development for training simulators.
5. Define training technology requirements. The outputs derived from the first four steps should be used to define a series of training technology requirements. On the basis of the first four activities, training program designers should now have a clear understanding of the proposed training program, including the aims and objectives, the tasks to be trained, the constraints likely to influence program design and delivery and the required content, including fidelity, feedback and performance measurement requirements. This information should be used to define the requirements of the technology to be used to deliver the training. Example training technology requirements include, but are not limited to: cost, task type, KSAs to be trained, fidelity and validity levels required, efficacy/validation evidence, instructional strategies required, scenarios required, inputs and outputs, performance measures required, feedback and guidance required, and part versus whole task training requirement.
Phase 2 - training technology identification, evaluation and selection
1. Identify candidate technologies. Once the technology requirements are clearly defined, they should next be used to drive the identification of a range of suitable, candidate training technologies. Potential training technologies should be selected or rejected based on an initial judgement against the technology requirement criteria. As a corollary, a list of candidate training technologies that could potentially be used for the training program in question is derived.
2. Evaluate candidate technologies. The chosen candidate technologies should be evaluated against a series of training technology selection criteria. Example training technology selection criteria, along with the proposed origins of benchmarks to use in technology evaluations, are presented in Table 1.
3. Select training technology. The appropriate training technology should be selected based on the evaluation of the candidate training technologies.
Phase 3 - design, test and refine training program
1. Purchase and implement technology and develop content. Once the appropriate technology has been chosen, purchased and implemented, the training content (such as scenarios, tasks, feedback mechanisms, and performance measures) should be developed based on the outputs of the training program definition phase.
2. Define evaluation process. Although often ignored, it is essential that organisations evaluate their training programs, and the selection of training technologies should be based, to some extent at least, on evidence of its efficacy in previous training systems. It is imperative first of all that the evaluation process is clearly defined in terms of what it is that is to be evaluated (such as training transfer levels and TER) and how it is to be evaluated. Salas et al [10] suggest that during this stage the experimental plan and location should be clearly defined.
Training evaluation involves collecting data to determine whether a particular training program achieved its training objectives in terms of improved job performance [27]. Perhaps explaining its limited uptake, training evaluation is typically difficult to organise and conduct, resource intensive, expensive, requires high levels of expertise for study design and data evaluation [4] and often generates negative results [10]. Further, evaluation often has to take place in the field during real world job performance which is often difficult to organise.
Assessing the quality of training involves focussing on three main criteria: training time, transfer of training, and skill retention. The most important of the three is the level of training transfer achieved [22]; the ultimate goal of training is, after all, for trainees to transfer what was learned to the actual real world operational environment [17]. The concept is therefore concerned with how well training ‘transfers’ to the real world operational environment; it refers to the extent to which the KSAs trained by a particular training program are applied, generalised and maintained, over time, within the job environment in question (Baldwin and Ford, 1988; cited in [7]).
| Criteria | Description | Benchmark source |
|---|---|---|
| Cost (initial and running) | Refers to the financial cost associated with purchasing, setting up, and running the training technology in question. | Organisational analysis component of TNA output |
| Housing requirements | Refers to the housing requirements associated with the training technology in question. | Organisational analysis component of TNA output |
| Maintenance requirements | Refers to the resources required to maintain the training system throughout its lifecycle | Organisational analysis component of TNA output |
| Resources required to set up | Refers to the resources (other than financial) required to set up the training technology in question, including personnel and time required. | Organisational analysis component of TNA output |
| Wow factor | Refers to the wow factor associated with the training technology (e.g. high for high fidelity, full motion simulators, low for PC-based desktop training applications | Organisational analysis component of TNA output |
| Fidelity | Refers to the physical and functional fidelity (and others if deemed appropriate, such as motion fidelity) capability of the training technology when compared to the physical and functional fidelity requirements of the training program. | Person and Task analysis components of TNA output |
| Validity | Refers to the overall validity of the training technology. | Person, Task and Organisational analysis components of TNA output |
| Flexibility | Refers to the flexibility of the training technology in terms of the ability to modify and add and remove training scenarios, the ability to train other tasks and KSAs, the ability to modify performance measurement and feedback, the ability to modify housing and input and output devices, and the ability to upgrade the technology as and when appropriate. | Person, Task and Organisational analysis components of TNA output |
| Efficacy (validation study evidence) | Refers to the extent to which there is supportive evaluation and validation evidence concerning the training technology presented in the literature. | Person, Task and Organisational analysis components of TNA output |
| Ability to train KSAs | Refers to the capability of the training technology for training the task(s) and associated KSAs required. | Task analysis components of TNA output KSA specification |
| Ability to engage trainee | Refers to the ability of the training technology to engage and motivate the trainee to learn. | Person analysis component of TNA |
| Performance measurement capability | Refers to the trainee performance measurement capabilities of the training technology and also the extent to which these can be modified or to which new performance measurement dimensions can be added. | Person and Task analysis components of TNA output Training program content specification |
| Feedback capability | Refers to the capability of the training technology to provide feedback to trainees during and post training. Feedback capability also considers the technologies capability to generate feedback for training instructors to deliver to trainees. | Person and Task analysis components of TNA output Training program content specification |
| Part versus whole task training capability | Refers to the capability of the training technology to cater for part and/or whole task training. | Person and Task analysis components of TNA output Training program content specification |
Successful transfer of training is said to occur when there is “effective and continuing application, by trainees to their jobs, of the knowledge and skills gained in training, both on and off the job” (Broad and Newstrom, 1992, p. 6; cited in [26]).
Various models of training transfer are presented in the literature (for example [28–31]). Although these models differ, it is generally agreed that training transfer is a complex issue, and that various factors contribute to the level of transfer achieved. For example, Burke and Hutchins [32] suggest that there are three primary factors that influence levels of training transfer: learner characteristics (for example, trainee personality, ability and motivation), intervention design and delivery (for example, TNA, learning goals, content relevance, instructional strategies and methods adopted, self management strategies and technological support for transfer) and the work environment (for example, transfer climate, supervisor/peer support, opportunity to perform and accountability). Similarly, Baldwin and Ford [28] proposed a tripartite model of training transfer that decomposes the concept into training inputs, training outputs, and conditions of transfer.
When a major purpose of purchasing a technology is for training, it is desirable to examine its effectiveness objectively. For example, transfer of simulation training to subsequent real-world performance can be measured using the Transfer Effectiveness Ratio (TER) and the Training Cost Ratio (TCR). The most widely used measure of training effectiveness in the literature is the TER. Originally conceived by Professor Stan Roscoe [33] the TER compares the relative efficiency of the training technology with a conventional method such as training in the aircraft. Wickens and Hollands [22] also discuss the Training Cost Ratio (TCR), which represents the time and/or cost of simulation-based training compared to real-world training. Wickens and Hollands [22] suggest that it is important to consider both of these measures together when weighing the benefits of training systems. They also suggest that training program cost effectiveness can be assessed by multiplying the TER by the TCR. If the product is greater than 1, then the program is cost effective. If the product is less than 1, then the program is not cost effective. Wickens and Hollands point out that this measure can be used to determine when training should stop and transfer to the real world task should begin (the amount of training at which TER × TCR = 1 is the point at which the training program ceases to be cost effective, although it is noted that training programs may still be useful after this point if training is safer than real world task performance). Wiggins and O’Hare [34] also discuss the Cost Effectiveness Ratio (CER), which is used to determine the value of simulation-based training (Farmer et al, 1999; cited in [34]).
Another popular approach used for training evaluation purposes is Kirkpatrick’s [37] typology [7,10,38]. Kirkpatrick’s model of training evaluation outlines four levels of training outcomes: reaction, learning, behaviour and results [38]. Level 1, reactions, involves assessing trainee reactions to the training program, including reactions to the training, assessing judgements on the quality and relevance of the training [38], satisfaction with the training and instructor, and also trainee ratings of course materials and effectiveness of training delivery [10]. Level 2, learning, involves assessment of the learning that has taken place during the course of the training [38] and involves examining the extent to which trainees have acquired the KSAs being trained. Level 3, behaviour, involves assessing the extent to which the KSAs trained are applied on the job [38] and level 4, results, involves assessing the impact that the training has in terms of organisational objectives [38]. Factors assessed include reduced turnover, reduced costs, improved efficiency, and improved quality [10]. Tannenbaum et al [31] present an extended version of Kirkpatrick’s typology which includes six categories of training effectiveness. These include:
- reactions;
- attitude change;
- learning;
- training performance (behaviour I);
- job performance (behaviour II); and
- results/Organisational effectiveness.
3. Conduct pilot run and refine as appropriate. Once the training content is developed, pilot testing of the training program and technology should begin. This involves an iterative process of testing, refining and testing the training program and technology.
Phase 4 - deliver and evaluate training program
1. Deliver training. Once all testing and redevelopment is complete, the training program can be delivered.
2. Support training transfer. Supporting training transfer is a critical factor in the success or failure of training programs. Often, despite the best intentions of trainees to apply newly learned skills on the job, limitations in the post-training environment interfere with the actual level of transfer achieved [12]. Factors such as trainee self efficacy, career motivation and opportunity to perform trained tasks on the job (Ford et al., 1992, cited in [2]) and the level of support provided by peers, supervisors and the organisation itself have all been cited as key elements in the level of training transfer achieved. Other factors thought to be important include the time between training and on the job performance (Ford et al, 1992; cited in [7]), the use of intervention strategies to enhance the probability of transfer (Brinkerhoff and Montesino, 1995; cited in [7]) and informal reinforcement of transfer activities (Smith-Jentsch et al, 2000; cited in [7]).
3. Evaluate training program effectiveness (i.e. training transfer, TER, TCR). Once the process is clearly defined, the training program evaluation can commence. It is important that the outcomes derived from the training program evaluation are used to inform the selection and implementation of technologies for future training programs.
4. Document findings and outcomes. Documentation and dissemination of the training outcomes and evaluations is a key final step in the process. It is also important that outcomes and findings are fed back into the process to inform the development and evaluation of future training systems and technologies.
Conclusions
One of the major problems associated with training system selection, design, and implementation is a lack of consideration of training theory [1,4–7]. Although various models of the training system design process exist, there is a general lack of theoretically underpinned guidance for fitting technologies to training requirements [4]. The aim of this article was to outline a proposed model of training technology selection, design, and implementation. The model provides, based on a synthesis of the training and human factors literature, an overview of the steps which should be taken when designing training systems in which advanced technologies are being used to deliver the training required. Whilst we acknowledge that specific guidance regarding the issues identified can only be provided on a case-by-case basis (in particular, detailed TNA should be conducted before decisions can be made on factors such as fidelity level, whole versus part task training and performance measurement, feedback and guidance), we contend that the model provides a useful framework for ensuring that appropriate candidate training technologies are identified, evaluated, selected and implemented based on a comprehensive understanding of the training requirement in question.
Some of the fundamental issues requiring consideration throughout the steps defined in the model were discussed, including the level of fidelity required, the issue of part versus whole task training, trainee performance measurement and feedback, and evaluating training transfer. Although guidance was given, it is concluded generally that the literature on each of the issues discussed is limited. For example, with regard to the level of fidelity required to maximise training system effectiveness, the literature is vast and is divided into two camps; the first arguing that a high fidelity level is better for training (for example [35]), and the second arguing that training transfer is better augmented through the use of low fidelity systems or even deliberate deviations from reality (for example [18]). Whilst there is support for the notion that high fidelity simulation leads to enhanced transfer of training in some areas (for example [19]), a growing body of evidence indicates that more is not always necessarily better (for example [17,20]), and it is well known that there is not always a linear relationship between fidelity level and the level of training transfer achieved. More, therefore, is not necessarily better. What is clear is that the level of fidelity required is ultimately contingent on various factors, including the task itself, the KSAs to be trained, the trainees, the training aims and objectives, and the training content. The importance of investigating and clarifying these factors before selecting and designing training technologies is therefore paramount.
The model presented brings about a range of pressing lines of future inquiry. First and foremost is the application and validation of the model through training system design case studies. Whilst small test case studies were used during development of the model, retrospective and prospective applications of the model are required. The authors have recently applied the model for training system design purposes; however, it is not currently possible to discuss these findings further. Second, comparison of the outputs of applications of the model with current training system design processes is required. Third, one of the notable omissions from the training literature is a full, valid, and detailed taxonomy of training transfer. It is the authors’ view that such a taxonomy or classification scheme would reduce uncertainty regarding the concept and would form a basis for assessment. It is therefore recommended that a training transfer taxonomy or classification scheme be developed.
In closing, it has been clear, for many years, that advanced training technologies could potentially revolutionise military training programs. Despite this, exactly how to fit training technologies to training needs still remains unclear. Without structured and valid guidance on how to select, design and implement training technologies, the likelihood that the full potential of training technologies such as simulation is reached is reduced. To ensure that the full potential of advanced training technologies is reached, applications of the model presented in this article during future technology-based training system design efforts are urged.
Acknowledgement
This research was funded by the Land Operations Division of the Australian Defence Science and Technology Organisation (DSTO).
References
[1] Hancock, P.A. (2009). The future of simulation. In D. Vincenzi, J. Wise, M. Mouloua, and P.A. Hancock (Eds.). Human factors in simulation and training. (pp. 169-186), CRC Press, Taylor and Francis, Boca Raton, FL.
[2] Wilson, J. R. (1997). Virtual environments and ergonomics: needs and opportunities. Ergonomics, 40:10, pp. 1057–1077.
[3] Stone, R. (2009). Serious games: virtual reality’s second coming? Virtual reality, 13:1, pp. 1–2.
[4] Darken, R.P. (2009). Identifying the limits of training system effectiveness through taxonomies of human performance. Theoretical Issues in Ergonomics Science, 10:3, pp. 231–243.
[5] Dahlstrom, N., Dekker, S., van Winsen, R., and Nyce, J. (2008). Fidelity and validity of simulator training. Theoretical Issues in Ergonomics Science, iFirst, p. 1–10.
[6] Salas, E. and Burke, C. S. (2002). Simulation for training is effective when…. Quality and Safety in Healthcare, 11, pp. 119–120.
[7] Salas, E. and Cannon-Bowers, J.A. (2001). The science of training: a decade of progress. Annual Review of Psychology, 52, pp. 471–499.
[8] Mangos, P.M., and Johnston, J.H. (2008). Performance measurement issues and guidelines for adaptive, simulation-based training. In D. Vincenzi, J. Wise, M. Mouloua, and P.A. Hancock (Eds.). Human factors in simulation and training. (pp. 301-319), CRC Press, Taylor & Francis, Boca Raton, FL.
[9] Liu, D., Macchiarella, N.D., and Vincenzi, D.A. (2008b). Simulation fidelity. In D. Vincenzi, J. Wise, M. Mouloua, and P.A. Hancock (Eds.). Human factors in simulation and training. (pp. 61–74), CRC Press, Taylor & Francis, Boca Raton, FL.
[10] Salas, E., Wilson, K.A., Priest, H.A., and Guthrie, J.W. (2006). Design, delivery, and evaluation of training systems. In G. Salvendy (Ed.) Handbook of Human Factors and Ergonomics, 3rd Edition, John Wiley and Sons.
[11] Goldstein, I. L. (1980). Training in work organisations. Annual Review of Psychology, 31, pp. 229–272.
[12] Tannenbaum, S.I, and Yukl, G. (1992). Training and development in work organisations. Annual Review of Psychology, 43, pp. 399–441.
[13] Rouiller, J.Z., Goldstein, I.L. (1993). The relationship between organisational transfer climate and positive transfer of training. Human Resource Development Quarterly, 4:4, pp. 377–390.
[14] Tracey, J.B., Tannenbaum, S.I., and Kavanagh, M.J. (1995). Applying trained skills to the job: the importance of the work environment. Journal of Applied Psychology, 80, pp. 239-252.
[15] Stanton, N.A. (2006). Hierarchical task analysis: Developments, applications, and extensions. Applied Ergonomics, 37:1, pp. 55-79.
[16] Klein, G., Calderwood, R, and McGregor, D. (1989). Critical decision method for eliciting knowledge. IEEE Transactions on Systems, Man and Cybernetics, 19:3, pp. 462–472.
[17] Liu, D., Blickensderfer, E., Macchiarella, E., and Vincenzi, D. A. (2008a). Transfer of training. In D. Vincenzi, J. Wise, M. Mouloua, and P.A. Hancock (Eds.). Human factors in simulation and training. (pp. 49–60), CRC Press, Taylor & Francis, Boca Raton, FL.
[18] de Winter, J.C.F., Wieringa, P.A., Kuipers, J., Mulder, J.A., and Mulder, M. (2007). Violations and errors during simulation-based driver training. Ergonomics, 50, pp. 138–158.
[19] Pardillo, J.M. (2007). Report on the effectiveness of simulators as educational tool. Humanist Project Reference: GUPM-071015T1-DA2.
[20] Salas, E., Bowers, C.A., Prince, C. (1998). On simulation and training in aviation. The International Journal of Aviation Psychology, 8:3, pp. 195–196.
[21] Stewart. J.E., Johnson, D.M., and Howse, W.R. (2008). Fidelity requirements for Army aviation training devices: issues and answers. U.S. Army Research Institute for the Behavioural and Social Sciences, Research Report 1887.
[22] Wickens, C.D. and Hollands, J.G. (2000) Engineering Psychology and Human Performance, New Jersey: Prentice-Hall.
[23] Teague, R.C., Gittelman, S.S., and Park, O. (1994). A review of the literature on part task and whole task training and context dependency. United States Army Research Institute, United States Army Institute for the Behavioural and Social Sciences, Technical Report 1010, AD-A285 954.
[24] Helander, M. (1995). A guide to the ergonomics of manufacturing. Taylor & Francis, London, UK.
[25] Lee, T.D., Schmidt, R.A., and Young, D.E. (2006). Skill learning: conditions of training. In Lee TD, Schmidt RA, Young DE. Skill learning: Conditions of practice. In W. Karwowski, (Ed.), International Encyclopedia of Ergonomics and Human Factors, (pp. 912–916), London, Taylor and Francis, UK.
[26] Weiner, I.B., Freedheim, D.K., Schinka, J.A., Gallagher, M., Velicer, W.F., Goldstein, A.M., Nelson, R.J., and Lerner, R. M. (2003). Handbook of psychology. John Wiley and Sons.
[27] Goldstein, I.L. (1986), Training in organizations: needs assessment development and evaluation, (2nd Edition), Brooks/Cole Publishing Co, Monterey, CA.
[28] Baldwin, T.T., and Ford, J.K. (1988). Transfer of training: a review and directions for future research. Personnel Psychology, 41, pp. 63–105.
[29] Canon-Bowers, J., Tannenbaum, S., Salas, E., and Volpe, C. (1995). Defining competencies and establishing team training requirements. In R.A. Guzzo and E. Salas (Eds.), Team effectiveness and decision making in organizations. (pp. 333–380). San Francisco: Jossey-Bass.
[30] Kozlowski, S.W.J., and Salas, E. (1997). A multi-level organisational systems approach for the implementation and transfer of training. In J.K. Ford, S.W. J. Kozlowski, K. Kraiger, E. Salas, and M. Teachout (Eds.), Improving training effectiveness in work organizations (pp. 247–287). Mahwah, NJ: Lawrence Erlbaum Associates.
[31] Tannenbaum S.I., Cannon-Bowers J.A., Mathieu J.E. (1993). Factors That Influence Training Effectiveness: A Conceptual Model and Longitudinal Analysis. Report 93-011, Orlando, FL: Naval Air Warfare Center.
[32] Burke, L.A., and Hutchins, H.M. (2007). Training transfer: An integrative literature review and implications for future research. Human Resource Development Review, 6 (3), pp. 263–296.
[33] Lee, A.T. (2005). Flight Simulation. Ashgate, Aldershot, UK.
[34] Wiggins, M.W., and O’Hare, D. (2006). Applications of microsimulation in cognitive skills development. In W. Karwowski, (Ed.), International Encyclopedia of Ergonomics and Human Factors, (pp. 3262–3267), London, Taylor and Francis, UK.
[35] Noble, C. (2002). The relationship between fidelity and learning in aviation training and assessment. Journal of Air Transportation, 7, pp. 33–54.
[36] Hays, R.T. and Singer, M.J. (1988). Simulation fidelity in training system design. New York: Springer-Verlag.
[37] Kirkpatrick, D.L. (1976). Evaluation of training. In R. L. Craig (Ed.), Training and development handbook: A guide to human resource development. New York: McGraw Hill.
[38] Bates, R. (2004). A critical analysis of evaluation practice: the Kirkpatrick model and the principle of beneficence. Evaluation and Program Planning, 27:3, pp. 341–347