Volume 15, Number 1, March 2012
Skills Retention In A Complex Battlefield Management System: A Pilot Study
- * Department of Informatics and Systems Engineering, Cranfield University at the Defence Academy of the United Kingdom, Shrivenham, SN6 8LA, UNITED KINGDOM.
Abstract
This paper reports the findings of a pilot study which examined the long-term retention of digital skills in a UK Information Management System (IMS). Empirical evidence suggests that, long periods of non-use of a complex digital system may impact on the retention of the digital skills required to use the system effectively. However, no empirical data was available for a UK system. An experimental trial was conducted with military personnel to investigate skills retention for a UK IMS over a nine-week period.
Background
Military tasks increasingly involve complex digital Information Management Systems (IMS). These systems enable rapid and dispersed operations to be executed successfully [1]. IMS allow military personnel to share information, such as graphic information, messages, operation orders, reports, and Global Positioning System (GPS) information on units. They, therefore, afford the efficient distribution of information and thus generate a clearer common operating picture, allowing military personnel to react faster to information received and to deploy forces and resources more effectively in a time critical environment. At the top end, this capability provides staff officers with the opportunity to plan and make decisions at a faster rate than the opposition and to manage and analyse data entering the system [2]. However, along with these capabilities, complex digital systems place demands on the ability of military personnel to recall and action accurately long sequences of steps required to perform digital tasks successfully. The aim of the present study was to examine skills retention for a UK IMS and investigate whether skills fade occurs.
Definition of skill/digital skill
Military personnel rely on numerous skills in order to carry out tasks they have already been trained on. In the military domain, and indeed in many other contexts, a skill refers to the achievement of a specified criterion level of performance for a task. An individual who successfully executes a task is perceived as having a skill and as demonstrating proficiency at a task [1,3]. Digital skills encompass those required to use software running on a computer. This involves a combination of data entry and the execution of commands through a graphical user interface (GUI).
Digital skills requirements for performance
The military expects its personnel’s skills with complex systems to be sufficiently high to allow their units to execute their tasks, both in routine and emergency situations, when they are deployed. Digital skills are characterized by distinct, multi-step procedures—for example navigating through a series of menus and submenus to set parameters and execute commands. The ability of individuals to recall the required steps and more importantly, the order in which a task’s steps must be performed has been shown to predict performance [2,4,5]. Although there is a motor component to digital skills—for example, moving a pointing device or selecting functions on a touch screen—only a small degree of motor skill is required [1,6].
Skills retention literature
Memory for discrete procedural sequences (involving step-by-step actions) has been found to be perishable over extended periods of non-use [1,2,5,7]. In contrast, greater retention of perceptual-motor skills has been observed over long periods of non use [8,9]. Task complexity is a strong predictor of skills retention, where tasks with the most steps show more pronounced skills fade [10]. This is consistent with what psychologists have observed concerning procedural skills fade since the 1950s [11]. Moreover, steps within a sequence which are not prompted by the equipment or by the previous sequence of steps have also been found to be more difficult to perform [12,13]. However, one study measuring digital skills retention after an eight-week retention interval failed to find an effect of task complexity on retention. This absence of an effect may have been due to poor skill acquisition evidenced by a very low level of initial performance. This would have invalidated the subsequent tests of retention [2].
There is some anecdotal evidence in the UK that the complexity of an IMS can impact on the retention of digital skills required for its use, however this anecdotal evidence is based on a previous UK IMS. Other similar complex digital systems in the US have shown skills decay [14,15]. Non-use of these systems by military personnel can lead to skill deterioration which, in extreme cases, might mean that the skill is no longer functional when needed [16]. Skills fade is particularly salient and problematic in situations where individuals receive initial training of knowledge and skill that they may not be required to use, or exercise, for extended periods of time [17–19]. For the military, this period is usually between training and subsequent deployment. Military operators of complex software-intensive systems have found that their ability to operate these systems and retain currency is subject to decay if they do not experience regular use of these systems [14,15]. Most research has shown that increasing the retention interval reduces skills retention. A meta-analysis conducted on 189 independent data points taken from 53 articles showed a strong correlation between retention interval and skill decay (r = –0.51, p < 0.05) [17].
In addition to task complexity, other factors have been found to affect the retention of information [17,19,20]. Testing has been shown to enhance retention. In a series of experiments, Hagman [21,22] measured the retention of a simple motor task three minutes and 24 hours after acquisition. Participants took part in three blocks of six trials before being tested three minutes and 24 hours after training. Three schedules of training were compared where, in each block, the different groups would have a) alternating study and test trials, b) one study trial followed by five test trials, or c) five study trials followed by one test trial. A study trial involved studying how to perform the motor task at hand. A test trial was a practice test trial which wasn’t scored. Retention after 24 hours showed less forgetting for the second group where one study trial was followed by five test trials in each block. Similarly, Allen, Mahler, and Estes [23] found that the frequency of errors was reduced by 50% when 10 study trials were followed by a test trial compared to when no test trial was completed. Retrieval practice has been shown to enhance retention whereas repeated studying has no effect on retention [24]. The effect of retrieval practice seems to increase with the retention interval [25–26]. As the number of retrieval practice sessions increases, the level of forgetting decreases over time and thus a greater number of skills are retained as knowledge and hence remembered over time.
Given the strong correlation between initial learning and long-term retention, a number of studies have systematically manipulated the amount of initial training to observe its effect on long-term retention. Most studies have compared proficiency training (being able to do the task once correctly) and mastery training (overlearning) (learning beyond one successful performance of the skill). For a procedural task, one study [12] found that mastery training led to better performance than proficiency training, both after one week and five weeks after training. However, the training manipulation (proficiency versus mastery) and the retention interval (one week versus five weeks) did not show a significant interaction. This lack of interaction supports the idea that mastery training has a positive effect on long-term retention but no effect on the decay rate.
Current training in the military is rote-based with a focus on delivering a pre-determined programme of instruction primarily through instructor-led lectures and assisted practical exercises. This is the training method adopted by the military for IMS. Research has shown that a guided discovery-based method of training can enhance adaptability and transfer of digital skills and help military personnel maintain digital readiness [27]. A guided-discovery approach involves giving students minimal instruction and providing them with a series of exercises to work through. This approach has been found to be more effective than other approaches such as unguided exploration, or classroom instruction alone [27–29]. Schaab and Dressel [27] trained digital skills to entry level, enlisted soldiers using either guided demonstration (traditional approach) or constructivist techniques. The constructivist group was given minimal instruction, realistic vignettes to learn and integrate knowledge of the digital system, and a series of group-based practical exercises. In the traditional group, the students followed an instructor who provided a lecture, conducted a demonstration and then provided them with a practical exercise to work through. Emphasis on the conventional training group was placed on how the system operates. Although an equivalent level of performance was observed for both groups, the guided exploration group performed better on a novel practical exercise and reported lower levels of cognitive load. Student-centered active learning approaches such as discovery learning have been shown to produce better retention of information in comparison to traditional teaching methods (such as teaching-centered approaches). However, this research has been conducted with school students in education settings, largely in the area of problem solving (see [30]). To our knowledge, no research has been conducted on the effect of training methods on the retention of digital procedural skills for complex systems.
Few empirical studies have examined skills retention for complex digital systems. One experiment reported extensive skills fade for tasks performed using a vehicle mounted digital information system, which was in use before the current system in the US [6]. Estimates of digital procedural skills fade were provided over a no-practice retention interval for the tasks of creating and sending digital graphical map overlays and reports. The overlay and report tasks represented what are considered as common and important tasks for current and future systems. After a 30-day retention interval, a 52% drop in overlay task performance and a 23% drop in report task performance were observed [6].
Another study examined performance on the current US vehicle mounted digital information system for a sample of officers [2]. Immediately after completing a 16-hour training session the officers received a hands-on skill test and a knowledge test. Performance on both these tests was then re-assessed after an eight week retention interval, where there was no practice of digital skills required to use this system. The assessments at baseline and re-assessment were the same in terms of the tasks presented and their order. However, to avoid a rote response, some task details were changed, since memory of procedural skills was of interest here. A significant drop in performance was observed for the hands-on test but not the knowledge test. Looking at a breakdown of performance at each of a total of 13 skill tasks, the researchers found that performance significantly declined for three of the 13 tasks. Therefore the overall level of reduction in performance was at 10%, although again this decline was statistically significant. One US report questions whether digital skills are highly perishable. In a study by Schaab and Moses all soldiers passed a variant of their final examination three to four months after they had finished their training, with no use of the system during this period. Moreover, two soldiers, who had not used the digital system for approximately one year, passed a variant of their original classroom examination [31].
Goodwin et al’s [2] and Sander’s [6] studies provide evidence of a link between the amount of initial practice and experience with respective digital systems and the extent of digital skills retention. In Sanders’ experiment, none of the soldiers reported having being trained on this system. In contrast, nearly 72% of the officers in Goodwin et al’s. experiment had used the digital system whilst out on operation in either Iraq or Afghanistan. Moreover, the officer’s rated their ability to be at a medium level. It was assumed by Goodwin et al that this reported experience with the IMS on operations may have helped participants form more durable memory traces for the digital procedures required to successfully execute commands and complete tasks. Consistent with this assumption was the finding that participants who reported less practical experience with the IMS performed the worst on the hands-on test and thus suffered greater skills fade [2].
A number of factors moderate digital skills retention. Of the factors discussed above, task complexity (number of steps to be executed), is a key predictor of the amount of skills that can be retained and later recalled after a period of non-use [6]. Since the previous research examining digital skills retention was conducted, the capabilities offered by IMS have expanded as newer versions of these systems have been developed [6,2,31]. Moreover, there is anecdotal evidence for digital skills retention for an IMS in the UK but this is based on a previous IMS. In light of this, the present pilot study extended upon previous research. It aimed to examine the effect of time-since-training on the retention of complex digital skills for two levels of training, in the context of a UK IMS over a nine-week period.
Method
Design
The quasi-experimental design involved participants from two levels of training who were finishing their IMS training programme. The IMS training programme delivered was rote-based and involved instructors delivering a pre-determined programme of instruction primarily through instructor-led lectures and assisted practical exercises. This method of training is the standard method adopted by the military in the UK. Level 1 participants had no prior training on the IMS whereas Level 2 participants had previously completed Level 1 training and were given additional training on planning tools. For each level, the IMS training programme covered specific functions and how to use them in the new IMS system. Level 1 training was delivered to Advanced Signallers and above Vocational Signallers, Watchkeepers and Staff Officers and covered situation awareness. This training looked at the basic use of the IMS. The skills trained for situation awareness included the use of the instant messaging application, and tools and techniques for creating and sending information. Level 2 training was delivered to Staff Officers and some Non-commissioned Officers (NCOs) requiring training on planning tools. In addition, key functions trained at Level 1 were incorporated into the Level 2 training. Therefore the levels of training in this study represent two cohorts whose roles differ. Level 2 participants learn the planning tools but this is not necessary for Level 1 participants. As Level 2 differs from Level 1 because it incorporates planning tools the study examines skills retention for both levels of training.
For both training levels there were three different groups. Level 1 had 26 participants in each group whilst Level 2 had eight (with nine participants in Group 3) (see Table 1). All groups received their initial assessment at the end of their IMS training as a baseline measure (the beginning of the trial). The criterion level of initial performance was set at 80%, the minimum level of performance military personnel are required to reach. All three groups were also assessed at the end of the nine-week trial. Group 1 only included these two measures. Group 2 was additionally assessed after three weeks, whilst Group 3 was assessed at the beginning of the trial, six weeks after the initial assessment, and nine weeks after the initial assessment (the end of the trial).
Participants
A cross section of military personnel not involved in operations took part in the pilot study. The cohort receiving Level 1 training consisted of 78 participants taking part in their initial assessment after training (first assessment of performance). Twenty five participants made up the Level 2 training cohort taking part in the initial assessment.
Measures
The assessments currently used to evaluate knowledge and skills retention at the end of each level of training for the IMS, were used as the measure of performance. The overall score was used to assess the level of performance for each group.
Assessments
The assessments are referred to as either the initial assessments or further assessments. The initial assessments are the assessments that military personnel normally go through after their IMS training. The further assessments are all the other assessments that are specific to this trial (for example, assessments after three, six, or nine weeks) and are the same as the initial assessment. The initial assessment (developed by Subject Matter Experts (SMEs)) was used to measure performance at all assessment points during the nine-week trial. The sequence of tasks remained the same for each assessment within each level. Whilst the sequence of tasks presented in Levels 1 and 2 could be re-ordered, the view of the SMEs was that this would be detrimental, as the sequence is designed to be logical and developmental. Moreover, the sub-tasks within the assessment for Level 2 reflect the participant undertaking an Intelligence Preparation of the Battlefield (IPB) task. Similar to initializing the system at the beginning of the assessment for both levels, Level 2 must be completed in a set sequence in order for it to mirror how this task is completed in the operational environment. The further assessments included a section which requested information on how much time participants had spent using and/or practicing using the IMS since their last assessment.
Procedure
The training instructors were briefed before the start of the trial in order to standardise the assessment process. The researchers went to observe one assessment at each of the three sites to make sure the procedure was carried out according to the requirements. The assessments for both training cohorts (Level 1 and 2) were carried out by military instructors from the course staff. Instructors assessed the performance of their own students. Due to instructor availability and location it was not possible to randomly allocate participants to an instructor (assessor) in order to eliminate any bias due to instructors. Several instructors took part in the training programme for both cohorts (approximately five). For both training levels, the training instructors assigned participants to one of the three experimental groups before participants commenced their training.
In each location, a classroom equipped with 20 laptops running and a training Local Area Network (LAN) was used for the trials. The additional assessments (those occurring after the initial assessment) took place no more than two days before or after each assessment interval.
The instructors were asked not to assist or interfere with the student(s) during the assessment, unless as a matter of Health and Safety. Participants were given a copy of the Quick Reference Guide (QRG) during all assessments for both training levels, since this would be available when out on operations. For Level 1, participants were allowed 80 minutes from the end of their brief to complete each assessment. Participants undertaking the Level 2 assessments for the trial were given 240 minutes from the end of their brief to complete each assessment.
The instructors were asked to check that participant’s placed a tick in the box by the task(s) at which they had difficulty remembering the procedure(s) which needed to be executed or at which errors had been made. The instructors were asked to make sure that if they did assist participants they ticked the corresponding box on the scoring sheet. This procedure was requested by the researchers to allow the quantification of performance.
Participants were asked not to confer on tasks and the instructors ensured this did not occur. The training units involved in the trial were requested not to provide any additional revision training before the assessments. On completion of each assessment the instructor used the system log to identify questions where participants made errors.
Scoring
The training instructors scored the assessments following their specified marking scheme. Marks were allocated to each task. The instructors deducted the appropriate number of marks for tasks where participants had difficulty or made errors.
The instructors were asked to go back to the log of actions for each participant, which was available on the LAN, and verify whether the tasks had been completed correctly. If this wasn’t the case, the instructor was asked to deduct the appropriate number of points on the answer sheet of the participant. Upon completion of each assessment the researchers collated the marks awarded to participants. Each assessment at Level 1 was scored out of 65 and the pass mark was set at 80% (52 marks). For Level 2, the maximum number of marks attainable was 63 and the pass mark was also 80% (50 marks). The pass mark of 80% represents the criterion level of performance set by the military and is thus the standardised operational definition of proficiency.
Results
Attrition rate
Unfortunately a low number of participants completed all the assignments they were assigned to. This was due to the availability of military personnel and logistical issues with calling back personnel to complete the further assessments.
From a total of 78 participants in the Level 1 training cohort, who completed the initial assessment, four completed all the assessments they were assigned to take part in (two participants in Group 1, two in Group 2 and none in Group 3). For the Level 2 training cohort, out of a total of 25 participants 16 completed all the assessments they were assigned to take part in (eight participants in Group 1, five in Group 2 and three in Group 3) (see Table 2). Due to the high participant attrition rate we used two approaches to look at the data: the average scores and the number of assessments failed. Given the low number of participants who completed all assessments, the participants who completed at least two assessments (initial and at least one other assessment) were considered. This decision was made based on the fact that it was possible to carry out a repeated-measures analysis when a participant had taken part in at least two assessments. In addition, information on how much time participants had spent using and/or practicing using the IMS since their last assessment was not collected. This was due to confusion between the initial and further assessments. These assessments were the same apart from the request for information on IMS experience, which was included in the further assessments only.
Level 1
The results for the Level 1 training cohort are presented in Table 3. A total of 18 participants completed at least two assessments: four completed all assessments and 16 completed two of the three assessments they were assigned to take part in. From this group, 12 participants dropped below the specified 80% mark at their first assessment after their initial assessment (which doesn’t include the nine-week assessment of Group 2 as it was not their first assessment after their initial assessment): six participants failed out of a total of nine at the three-week assessment, four participants out of seven failed at the six-week assessment, whilst two out of two failed the nine-week assessment. If we consider all the assessments undertaken by these 18 participants we have a total of 20 assessments (including the nine-week assessment of Group 2). From these, performance on 14 of the assessments was below the 80% mark; six at the three-week assessments (out of nine), four at the six-week assessment (out of seven), and four at the nine-week assessment (out of four).
Paired sample t-tests were conducted to compare the score of participants who completed the first assessment after training with the score for each of the other assessments (three-, six- and nine-week assessments). In addition, Pearson’s correlation coefficient r was calculated as a measure of effect size. This measure is a versatile measure of the strength of an experimental effect [32] and is constrained to lie between 0 (no effect) and 1 (a perfect effect). Cohen [33] suggests the benchmarks of r = 0.10 (small effect), r = 0.30, (medium effect) r = 0.50 (large effect). A paired sample t-test showed that when comparing the score for participants who completed the initial assessment and the three-week assessment a significant drop in the level of performance was observed going from 94% to 60%. This drop was of a large effect size (t(8) = 4.93, p < 0.01, r = 0.87). When comparing the scores for participants who completed the initial assessment and the six-week assessment, there was a significant drop in the level of performance going from an average of 93 % at the initial assessment to 69% at the six-week assessment. This retention interval had a large effect on performance (t(6) = 3.29, p = 0.02, r = 0.80). Finally, the comparison between the scores of participants who did the initial assessment and the nine-week assessment showed a significant drop going from 94% at the initial assessment to 36% at the nine-week assessment point. This drop in performance represented a large effect size (t(3) = 8.41, p = 0.01, r = 0.98). For this cohort, the average score for each of the three further assessments (three-week, six-week, and nine-week) fell below the 80% criterion.
Level 2
The results for Level 2 are presented in Table 4. A total of 19 participants took part in at least two assessments: 16 completed all assessments and an additional three participants who completed two assessments were also included in this section. From this total, seven participants dropped below the criterion at their first assessment after their initial assessment (which doesn’t include the nine-week assessment of Group 2 and Group 3 as it was not their first assessment after the initial assessment): three participants failed out of a total of seven at the three-week assessment, two participants out of four failed at the six-week assessment, whilst two out of eight failed the nine-week assessment point. If we consider all the assessments undertaken by these 19 participants we have a total of 27 assessments (including the nine-week assessment for Group 2 and 3). From these, 12 assessments were below the 80% mark; three (out of seven) at the three-week assessment point, two (out of four) at the six-week assessment, and seven (out of 16) at the nine-week assessment.
A paired sample t-test showed that when comparing the scores for participants who completed the initial assessment and the three-week assessments a significant drop in scores was observed going from 98% to 83% (t(6) = 3.4, p = 0.015, r = 0.81) and this observed drop in performance was of a large effect size. However, this average score was still just above the 80% criterion. When comparing the scores for participants who did the initial assessment (95%) and the six-week (73%) assessments, no significant drop in scores was observed (t(3) = 2.19, p = 0.12). Finally, the comparison between the scores for participants who did the initial and nine-week assessments showed a significant drop going from 95% at the initial assessment to 77%. The effect size for this observed decline in performance was large (t(15) = 4.14, p = 0.01, r = 0.73). The average scores in Level 2 dropped below criterion for two of the three further assessments (six-week and nine-week).
If we consider the average scores for all further assessments in this pilot study (such as the three average scores for Level 1 and the three average scores from Level 2), five out of six, or 83% of them, fell below the 80% pass mark. Also if we look at the number of first assessments failed during the trial (the first assessment completed after the initial assessment), 11 out of 20 fell below the 80% mark.
Discussion
The aim of this pilot study was to examine skill retention for a UK IMS and investigate whether skills fade occurred. Although the study started with a decent sample size, the rate of attrition was very high. As a consequence the power for this study is low. If we consider general sample size rules of thumb when measuring group differences then a reasonable sample size (with a medium-to-large effect size) would require 30 participants per cell for 80% power. If cell size is reduced, then the sample size should be no lower than seven per cell [34].
The results have shown that more than half of the participants did not pass their first assessment which was taken between three and nine-weeks after their initial assessment (across the two levels of training). If we look at the average scores for the two training cohorts who completed the nine-week trial, the average score fell below the pass mark for all three means in Level 1 whilst in Level 2 the average score fell below the criterion on only two occasions (six-week and nine-week assessment). This result is not surprising if you take into account the fact that the participants who completed training at Level 2 would have previously had the training at Level 1. Therefore, these participants would have had more experience with the system and hence a greater level of performance would be expected.
For Level 1, the trial has shown a significant drop in scores below the criterion at each assessment point (three-week, six-week, and nine-week). This suggests those trained at Level 1 may require more frequent refresher training in order to maintain their level of skills above the criterion. For Level 2, the trial showed that there was a significant difference between the uplift assessment score and the nine-week assessment score. The average score for the six-week assessment dropped below criterion but this score did not differ significantly from that observed at the initial assessment. The results indicate military personnel trained up to Level 2 maintained performance above the set criterion level for longer. Thus, it may be that those trained to Level 2 require less frequent refresher-training. However, strong conclusions as to the frequency of refresher training required cannot be made due to the high participant attrition rate.
The results indicate that skills fade is a reality when using the UK digital IMS, but the finer details are based on a very small number of observations. In addition, the average scores are influenced by the different number of assessments completed by the different groups (two versus three). In light of this, caveats need to be placed on the data.
General discussion
The pilot study presented here was the first to quantitatively measure the long-term retention of digital skills required for a UK IMS. Caveats have been placed on the data due to the small sample size in each training cohort (Levels 1 and 2) and the subsequent pooling of data, as a result of the significant attrition rates. Although strong conclusions cannot be derived from this data, the data can be viewed as providing some indication of a reduction in digital skills retention after only a few weeks. The results of the present study therefore support the findings of previous research [2,6].
Although a statistically significant and large decline in performance was observed at all assessment points for Level 1 participants and at the nine week assessment points for those at Level 2, the mean scores were of the order of 73–80% at week nine with the criterion being 80%. It follows that this difference is arguably of no practical significance. Therefore, although performance on most occasions fell below the specified level of proficiency, the fact this decrement was very small could be viewed as adding support for Schaab and Moses [31] who question whether digital skills are highly perishable.
Given the low number of participants the data from Group 2 and Group 3 at the nine-week assessment point had to be pooled. Performance on these assessments would have been influenced by time and a testing effect. However, given the effect of testing should have improved performance [25] and [26] one would not have expected the observed decrement in performance.
One problem with the assessments used to measure performance is that performance was based on the number of marks allocated for each task. The assessment displayed the number of marks per task. This is problematic as it was possible for participants to complete the tasks that were worth the most marks first. Although the tasks within the assessments are presented sequentially (especially the initialization of the system), it was possible for participants to focus on the tasks worth more marks and leave some of the tasks worth fewer marks. Such a strategy, if adopted, would affect the validity of the performance data, in addition to the small sample size of the present pilot study.
Future research on IMS should measure the performance of participants at each task so that training instructors can be informed of which tasks participants have greater difficulty performing. Given the only information available in the present pilot study was a score for each task, reflecting the number of marks deducted, it was not possible to determine which steps participants completed. Furthermore, as the number of marks attributed to each question was not perfectly correlated with the number of steps it was not possible to determine the tasks participants found more difficult. Future studies should also consider investigating the mouse movements and decisions made by participants as they complete the IMS assessment. This would also help in determining which steps are most likely to be forgotten. We were unable to record each individual menu choice or movement of the mouse, a limitation which minimized our ability to account for the sources of error and difficulty that lead to the observed decline in performance.
Past research has demonstrated a link between the amount of initial practice and experience with respective digital systems and the extent of digital skills retention. This information was requested but unfortunately was not captured owing to confusion between the initial assessment after IMS training and the further assessments. The content of both assessments were the same except for the request for information regarding experience. This had lead to only the initial assessment being distributed by the training instructors to personnel for assessment.
Another potential limitation that should be acknowledged is the instructor ratings of performance, which may have biased the results. Although the researchers were permitted to watch the assessment activity, they were unable to conduct the assessment as qualified military instructors are required to conduct formal assessments of performance. Also, due to instructor availability it was not possible to randomly allocate participants to an instructor (assessor) in order to eliminate any bias due to instructors.
The training method adopted, and the method adopted to assess performance was not within the researchers’ remit to change, given that the training outcome dictated whether or not military personnel could move on to the next stage of their training programme. This is an unfortunate challenge of conducting naturalistic research with military personnel on training courses (see [35]). Research has shown that a guided discovery approach in contrast to traditional methods of instruction (rote learning based) can enhance adaptability and transfer of digital skills and help military personnel maintain digital readiness in changing environments [27]. Future research should examine whether the way training is delivered impacts on the retention of digital skills learnt. since, to our knowledge, this has not been examined with military personnel and complex digital IMS.
Conclusions
The present pilot study provides the first quantitative analysis of skills retention for a UK IMS. A difference in performance for both training cohorts (Level 1 and Level 2) was observed over a period of nine weeks indicating a decline in performance had occurred for this UK IMS. The trends in the data support previous research that has demonstrated a decline in performance and thus skills fade for digital IMS. However, caveats are placed on the data owing to the small sample size and subsequent power due to high participant attrition rates during the trial. Stronger conclusions cannot be drawn without further research with a larger sample size, ensuring a sufficient number of participants at each assessment point.
Acknowledgement
The work reported here is part-funded by the Human Dimension & Medical Sciences Domain of the UK Ministry of Defence Scientific Research Programme, and was initiated by the Dstl Programme Office.
References
[1] G.A. Goodwin. The Training, Retention, and Assessment of Digital Skills: A Review and Integration of The Literature, (ARI Research Report 1864), US Army Research Institute for the Behavioral and Social Sciences, Arlington, VA, (DTIC No. ADA470707), November 2006.
[2] G.A. Goodwin, B.C. Leibrecht, R.L. Wampler, S.C. Livingston, and J.L. Dyer, Retention of Selected FBCB2 Operating Skills Among Infantry Captains Career Course (ICCC) Students, (ARI Research Report 1872), US Army Research Institute for the Behavioral and Social Sciences, Arlington, VA, July 2007.
[3] C. Stothard and R. Nicholson, Skill Acquisition and Retention in Training: DSTO Support to the Army Ammunition Study, Land Operations Division Electronics and Surveillance Research Laboratory, (DSTO-CR-0218), December 2001.
[4] R.A. Wisher, M.A. Sabol, and J.A. Ellis, Staying Sharp: Retention of Military Knowledge and Skills, (Final Special Report 39). US Army Research Institute for the Behavioural and Social Sciences, Alexandria, VA, July 1999.
[5] W.R. Sanders, Cognitive Psychology Principles for Digital Systems Training, (Research Report 1773), US Army Research Institute for the Behavioral and Social Sciences. Alexandria, VA, 2001.
[6] W.R. Sanders, Digital Procedural Skill Retention for Selected M1A2 Tank Inter-vehicular Information System (IVIS) Tasks, ARI Technical Report 1096, US Army Research Institute for the Behavioral and Social Sciences, Alexandria, VA, (DTIC No. ADA368212), August 1999.
[7] J. Annett, “Trained Skilled Performance”, in A.M. Colley and J.R Beech (eds), Acquisition and Performance of Cognitive Skills, Chichester, Wiley, 1989.
[8] J. Patrick, “Training: Research and Practice”, London, Academic Press, 1992.
[9] R.W. Swezey and R.E. Llaneras, “Models in Training and Instruction”, in G. Salvendy (ed), Handbook of Human Factors, New York, Wiley, pp. 514–577, 1997.
[10] J.L. Shields, S.L. Goldberg, and J.D. Dressel, Retention of Basic Soldering Skills, (Research Report 1225), US Army Research Institute for the Behavioral and Social Sciences, Alexandria, VA, 1979.
[11] J.A. Adams. “Historical Review and Appraisal of Research on the Learning, Retention, and Transfer of Human Motor Skills”, Psychological Bulletin, Vol. 101, No. 1, pp. 41–74, 1987.
[12] S.L. Goldberg, M. Drillings, and J.D. Dressel, Mastery Training: Effects on Skill Retention, (Technical Report 513). Army Research Institute for the Behavioral and Social Sciences, US: Alexandria, VA, 1981.
[13] C.M. Knerr, J.H., Harris, B.K. O’Brien, P.J. Sticha, and S.L. Goldberg, Armor Procedural Skills: Learning and Retention, (ARL Technical Report 621). US Army Research Institute for the Behavioral and Social Sciences, Alexandria, VA, (AD A153 227), 1984.
[14] R.P. Lynch, Lessons Learned: Commanding a Digital Brigade Combat Team, Institute for Defense Analysis-Joint Advanced War Fighting Program, U.S: Alexandria, VA, (DTIC No. AD-A395042), 2001.
[15] J.C. Johnston, B.C. Leibrecht, L.D. Holder, R.S. Coffey, and K.A. Quinkert, Training for Future operations: Digital Leaders’ Transformation Insights, (ARI Special Report 53). US Army Research Institute for the Behavioral and Social Sciences, Alexandria, VA, 2003.
[16] A. F. Healy and G. P. Sinclair, “The Long-Term Retention of Training and Instruction”, in E.L. Bjork and R.A. Bjork (eds), Memory: Handbook of Perception and Cognition, New York, Academic Press, pp. 525–564, 1996.
[17] W. Arthur, W. Bennett, P.L. Stanush, and T. L. McNelly, “Factors that Influence Skill Decay and Retention: A Quantitative Review and Analysis”, Human Performance, Vol. 11, pp. 57–101, 1998.
[18] C.E. Lance, A.G. Parisi, and W.R Bennett, “Moderators of Skill Retention Interval/Performance Decrement Relationships in Eight US Air Force Enlisted Specialities”, Human Performance, Vol. 11, pp. 103–123, 1998.
[19] R.A. Wisher, M.A. Sabol, H.K. Sukenik, and R.P. Kern, Individual Ready Reserve (IRR) Call-Up: Skill Decay, (Research Report 1595): US Army Research Institute for the Behavioral and Social Sciences, Alexandria, VA, 1991.
[20] J.D. Hagman and A. M Rose, “Retention of Military Tasks: A Review”, Human Factors, Vol. 25, pp. 199–213, 1983.
[21] J.D. Hagman, Effects of Presentation- and Test-trial Training on Motor Acquisition and Retention, (Technical Report 431), US Army Research Institute for the Behavioral and Social Sciences. Alexandria, VA, 1980.
[22] J.D. Hagman, Effects of Presentation- and Test-trial Training on Acquisition and Retention of Movement End Location, (Technical Report Draft), US Army Research Institute. Alexandria, VA, 1980.
[23] G.A. Allen, W.A. Mahler, and W.K. Estes, “Effects of Recall Tests on Long-term Retention of Paired Associates”, Journal of Verbal Learning and Verbal Behavior, Vol. 8, pp. 463–470, 1969.
[24] J.D. Karpicke and H.L. Roediger, “The Critical Importance of Retrieval for Learning”, Science, Vol. 319, pp. 966–968, 2008.
[25] H.L. Roediger and J.D Karpicke, “Test-enhanced Learning: Taking Memory Tests Improves Long-term Retention”, Psychological Science, Vol. 17, pp. 249–255, 2006.
[26] J.D. Karpicke and H.L. Roediger, “Repeated Retrieval During Learning is the Key to Long-term Retention”, Journal of Memory and Language, Vol. 57, pp 151–162, 2007.
[27] B.B. Schaab and J.D. Dressel, Training for Adaptability and Transfer on Digital Systems, US Army Research Institute for the Behavioral and Social Sciences. November 2001.
[28] J.M. Carroll, R.L. Mack, C.H. Lewis, N.L. Grischkowsky, and S.R. Robertson, “Exploring a Word Processor”, Human-Computer Interaction, Vol. 1, pp. 283–307, 1985.
[29] D. Charney, L. Reder, and G.W. Kusbit, “Goal Setting and Procedure Selection in Acquiring Computer Skills: A Comparison of Tutorials, Problem Solving, and Learner Exploration”, Cognition and Instruction, Vol. 7, No. 4, pp. 323–342, 1990.
[30] J. Michael, “Where’s the Evidence That Active Learning Works?” Advances in Physiological Education, Vol. 30, pp. 159–167, 2006.
[31] B.B. Schaab and F.L. Moses, Six Myths About Digital Skills Training, (Research Report 1774), US Army Research Institute for the Behavioral and Social Sciences, Alexandria, VA, 2001.
[32] R. Rosenthal and M. R., DiMatteo, “Meta-analysis: Recent Developments in Quantitative Methods and Literature Reviews”, Annual Review of Psychology, Vol. 52, pp. 59–82, 2001.
[33] J. Cohen, “Statistical Power Analysis for the Behavioural Sciences”, Hillsdale, NJ: Lawrence Erlbaum, 1988.
[34] R. Carmen, V. Wilson, and B.L. Morgan. “Understanding Power and Rules of Thumb for Determining Sample Sizes”, Tutorials in Quantitative Methods for Psychology, Vol. 3, pp. 43–50.
[35] E. Salas, L. Milham, and C. Bowers, “Training Evaluation in the Military: Misconceptions, Opportunities, and Challenges”, Military Psychology, Vol. 15, pp. 3–16, 2003.