Volume 11, Number 1, March 2008
A “Thin-Slicing” Approach To Understanding Cognitive Challenges In Real-Time Command And Control
- 1 The MITRE Corporation, 202 Burlington Rd., Bedford, MA 01730 USA.
Abstract
Before modernizing any information-intensive system, it is important to understand how people are performing their jobs using that system. This understanding is built not simply by observing what people are doing, but by digging into how people think about their jobs, what coordination they need to do, and the dependencies among subtasks that together dictate a workflow. Building a thorough understanding of complex tasks takes weeks or months rather than days, but we needed to get as much of an understanding as possible in three days of a real-time command and control centre for military unmanned vehicles. To help structure our investigation, we used a technique called Applied Cognitive Task Analysis (ACTA) [14]. To the best of our knowledge, ours is the first use of ACTA to study a military command and control centre in such a time-compressed fashion. We describe our application of ACTA and the types of recommendations we were able to generate from our analysis, and provide reflections on the study process. Another contribution of this paper is based on the fact that we were able to gain access to a facility that is not usually open to researchers; hence the ACTA results may be of interest to those who would benefit from knowing about the major cognitive challenges facing members of the Predator Unmanned Aircraft System community.
Introduction
Background
The Predator Unmanned Aircraft Systems (UASs) are becoming increasingly useful to US and coalition military operations. The number and complexity of Predator missions is rising as requests for Predators’ services pour in. To ease coordinating and supporting Predator missions, the Predator Operations Center (POC) was opened at Nellis Air Force Base (AFB).
The POC is rapidly evolving as new technologies and operational procedures are tried, evaluated, and kept or discarded. What is unusual in the world of military acquisition and command and control center design is just how rapidly the POC is changing. Instead of months or years, some POC changes are happening in days or weeks.
We are convinced that understanding and documenting POC operators’ cognitive demands is key to designing better technology for them: in particular, better visualization of information for decision-making and better means of facilitating collaboration. Understanding cognition is difficult, however, because the POC operations tempo leaves little time for operators to work with analysts. Also, analysis methods that attempt to document how operators do their jobs at a detailed level will likely result in studies that are obsolete before they are completed.
We concluded that we needed a type of cognitive task analysis (CTA) that could be performed extremely rapidly. The traditional task analysis technique often used as the starting point for CTA is Hierarchical Task Analysis, in which “tasks are represented in terms of a hierarchy of goals and subgoals, using the idea of plans to show when the subgoals need to be carried out” [19, p. 1]. CTA is an appropriate means of understanding cognitive challenges because such techniques were designed to be “the extension of traditional task analysis techniques to yield information about the knowledge, thought processes, and goal structures that underlie observable task performance” [1, p. 3].
We needed to gather all data, with the possible exception of later sending email messages with a few follow-up questions, during a three-day period. Because of the need to form almost instant—yet correct—impressions of operators’ cognitive challenges, we were reminded of the phrase “the power of thin-slicing” from psychology: “as human beings we are capable of making sense of situations based on the thinnest slice of experience” [6, pp. 43-44].
Choosing a cta technique
Unfortunately, CTA has not matured to the point where there is consensus regarding which CTA technique should be used in which situation. There are many CTA techniques: in fact, whole books full of them [17]. We examined some of the most frequently cited variants of CTA before determining the one that would be most appropriate for our analysis needs. We considered each technique based on whether it would allow us to accurately capture as many major cognitive challenges as possible within a three day period, be as non-disruptive as possible, not require recording and/or instrumentation, and take into account a dynamic, team-based environment. By “non-disruptive,” we mean that gathering the data could not adversely impact POC operations, which proceed for 24 hours per day, seven days per week.
Specifically, we considered six methods:
- Goal Directed Task Analysis (GDTA) [4].
- Team CTA [12].
- Task-Knowledge Structures (TKS) [10].
- Critical Decision Method (CDM) [11].
- Simplified Precursor, Action, Result, and Interpretation (Simplified PARI) [18].
- Applied Cognitive Task Analysis (ACTA) [13,14].
Gdta and team cta
GDTA [4] is an interview and observation-based technique that decomposes tasks at a fine-grained level and then examines the situation awareness needed for each part of the tasks. Team CTA [12] extends task analysis techniques designed for individuals to accommodate phenomena such as team members’ interdependence and cooperation. Note that both GDTA and Team CTA build on traditional task analysis techniques that normally require analysts to examine each work unit (the lowest-level standalone task) in great detail. Analysts document such information as the stimulus that triggers the work unit, its frequency of performance, relative importance, information required to perform the work, decision points, behaviour required on the part of the human, indication of the adequacy of the human or system response, and indication of errors.
Task-knowledge structures
TKS [10] is a theoretical approach to modelling tasks that assumes humans hold in memory a highly structured knowledge set that guides task performance. When using TKS, analysts model the status of individual “knowledge components” and look across them to make systematic predictions of an interface design’s usability. While TKS has more emphasis on individual human cognition than GDTA or Team CTA, it requires a similarly detailed understanding of such task characteristics as were listed above. Accordingly, we concluded that GDTA, Team CTA, and TKS would require much more contact with the POC personnel than the available time period would allow. We then turned to CDM, Simplified PARI, and ACTA.
Critical decision method
CDM [11] is derived from the critical incident technique [5] in that it calls for a retrospective interview with an expert decision-maker in which the interviewee is asked questions about a non-routine (“critical”) event. The advantage of this method is that critical events are usually memorable and the personnel involved usually reflect on what had happened. The disadvantage of this method is that human memory is inherently flawed and, since critical events tend to happen infrequently, interviewees may have great trouble remembering sequences of events with accuracy. Also, the limited number of interviews we would be able to conduct during the three day period would likely mean that we would not uncover many of the cognitive challenges associated with routine work processes in the course of probing the exceptional events; thus, we ruled out this method.
Simplified pari
The full PARI method consists of a nine-step process developed by Hall et al. [7]. It is not feasible to apply the PARI method to a complex system in three days, but Seamster et al. [18] developed a simplified version of PARI that has three steps: a subject matter expert (SME) describes a problem-solving scenario, a second SME solves the problem during a structured interview, and then both SMEs are interviewed together. While the method seemed otherwise promising, it was impractical and disruptive to try to schedule interviews with two SMEs simultaneously. We would only be able to interview POC personnel who could be pulled from the operations floor temporarily, which meant that we had to interview people individually.
Applied cognitive task analysis (acta)
ACTA is a streamlined form of CTA that relies on three sets of interview questions to identify the mental demands needed for tasks, and personnel can be interviewed individually. ACTA analyses have been previously performed on safety-critical tasks such as fire-rescue [14], command and control [2], air campaign planning [15], and military operations [16, 8]. What we have not seen previously, however, is the use of ACTA for capturing and describing extremely complex cognitive tasks within a contiguous three-day period.
Nevertheless, ACTA seemed the most appropriate of all the methods we considered. Because the ACTA method does not specifically examine team cognition, social dynamics, or coordination (necessary to characterize operators’ dynamic, team-based environment), we supplemented it with direct observation and additional questions about explicit coordination procedures and mechanisms.
Outline
Next we describe the Predator and its operations at a very high level to provide the necessary context to understand our results. Subsequently we describe ACTA in more detail in the course of describing how we applied the technique and the results we obtained. Finally, we discuss how the analysis impacted subsequent work and provide reflections on our experience applying ACTA.
Predator system background
The “Predator System” is defined as the aircraft itself, the Ground Control Station (GCS), the POC, and Predator Primary Satellite Link. Predators are medium-altitude, long-endurance aircraft that provide reconnaissance and also can be weaponized. Predators carry infrared, electro-optic, and synthetic aperture radar sensors, whose feeds are collectively referred to by operators as “video”. The GCS is located in an 8’ × 8’ × 20’ shelter that normally houses a pilot and sensor operator of one aircraft (there is a variant designed to house a pilot and four sensor operators).
In the early stages of Predator deployment, a third operator sat in each of the GCSs: a Mission Coordinator (MC). Today, the MCs are all collocated in the POC, along with the weather operator, Senior MC (SMC), and Mission Crew Commander (MCC). The MCs comprise the majority of the personnel in the POC and so we concentrated on this role.
MCs are responsible for overall mission planning, which includes downloading Air Tasking Orders (ATOs)/Airspace Control Orders (ACOs) and target decks, plotting target decks, and researching reference imagery/target background. They coordinate collection requirements and on-target times. MCs update pilots and other MCs regarding airspace deconfliction, targets, and threats. They also update the target folder file structure that contains historical target information.
The POC is housed in cinderblock building not far from many of the GCSs. Ten MC stations are arranged along the periphery of a room in a horseshoe shape. Each operator faces the wall, with three computer displays and Predator video feeds projected on the wall above the displays. A raised platform in the middle of the room provides space for a weather operator, Senior MC/intelligence officer, an intelligence support person known as the DMO/MSA, and the person responsible for all POC operations: the MCC. Printers are ranged along the back wall and servers are located in an adjacent, smaller room.
Predator tasking may be pre-planned and provided in a plan called the Scheme of Manouevres (SOM), or tasking may occur on a time-sensitive basis that may cause the original SOM to be abandoned or delayed.
Applied cognitive task analysis and observation analysis
We focused the ACTA on the POC personnel, with most emphasis on the MCs, but we also interviewed a SMC and MCC. We augmented the interviews with direct observations of POC personnel and GCS pilots and sensor operators. Because cognition alone is insufficient to understand the work environment, we were alert for social, behavioural, or cultural factors that impacted job performance.
The products of an ACTA are an overview task diagram, a knowledge audit, a simulation summary, and a cognitive demands analysis.
Overview task diagram
The overview task diagram shows how the task can be divided into a handful of pieces (three to six pieces is the recommended number of subdivisions). By asking interviewees which task portions require the most thought and judgment, it helps the interviewer to focus further questions on the most challenging portions.
At the beginning of a POC person’s shift, he or she sits with their outgoing counterpart to hear what they need to know regarding the mission in progress. The nature of missions varies but they will have one or more combinations of the following activities: looking for targets, surveilling a target once it has been found, and placing weapons on a target.
Crews may look for a target using an area search, meaning they are given coordinates that encompass a specified geographical area and are told to develop a search pattern to look at the entire area, or may be asked to search along specified roads (“lines of communication”). Note that “targets” is used generically—most often applied to people, buildings, vehicles, and improvised explosive devices—and is used either in the sense of the target of the surveillance effort or the target for the Predator’s weapons, as appropriate.
Once a target is located, the Predator operators are normally asked to keep that target in view for a specified period of time. Mobile targets, such as cars, can be difficult to keep in constant view, especially in urban situations.
Shooting a target involves many steps, including understanding that the Rules of Engagement have been met and the Collateral Damage Estimate is acceptable, passing along any instructions to the pilot who fires the weapon, and assessing damage from the weapon (called Battle Damage Assessment) once it has been fired. Because of the number of steps to be followed, the substantial coordination requirements, and high stakes, POC personnel identified this piece of their work as being very demanding.
Operators may perform these three types of activities any number of times during their shift. Throughout these activities, the mission coordinator must complete mission reports (MISREPs) that can be referenced by the customers or video exploiters. Finally, at the end of their shift they sit with their incoming counterpart and provide the pertinent information for them.
These five steps are depicted in figure 1.
Knowledge audit
According to Militello and Hutton [14], a knowledge audit “identifies ways in which expertise is used in a domain and provides examples based on actual experience.” (pp. 1621). It probes the nature of expertise, such as knowing when to use a trick of the trade or how to improvise. Questions fall into three categories, related to how the interviewee acquires the ‘big picture’, monitors their own performance, and applies ‘job smarts’. Table 1 contains the examples of how POC personnel attain the big picture, whereas Table 2 addresses self-monitoring and ways to apply job smarts.
Simulation summary
A series of interview questions ask the interviewee to step though a particularly challenging scenario, probing the interviewee’s cognition and decision-making processes. The questions include, “What actions, if any, would you take at this point?”; “What do you think is going on here?”; “What pieces of information led you to this situation assessment and these actions?”; and “What errors would an inexperienced person be likely to make in this situation?”.
We explored three scenarios with the interviewees: when troops are in contact with hostile forces, following a vehicle, and shooting a Hellfire missile. The results are in Table 3.
Cognitive demands analysis
Militello and hutton [14] recommend that an acta include a “cognitive demands table” as a way to analyse the data. the table lists difficult cognitive elements, reasons why they are difficult, what errors people commonly make, and the cues and strategies used to work through the cognitive difficulties. in particular, they note that such a table can help analysts to see common themes.
| Aspects of Expertise | Cues & Strategies | Why Difficult? |
|---|---|---|
| There are many different types of information to keep track of (e.g., chat, video feed, locations of Predator and other aircraft, MISREPs, getting target information from the ground reference grid (GRG), etc.). A big priority is being responsive to the customer on chat. | Stay on top of the customer’s chat window (at the expense of watching the video if need be – there is Tivo and other pairs of eyes on the video). | A novice could easily become overwhelmed and not know where to focus their attention. |
| Supporting the pilot and sensor operator. This involves prioritizing and filtering (e.g., need to filter information entered into chat by the supported unit when communicating with the crew). | One cue is to listen to the audio channel between the pilot and sensor operator to gain a sense of their activity levels. | The MC cannot hear the voice channel between the pilot and the JTAC (Joint Terminal Air Controller). Often the pilot is working through the mechanics of calling up or entering data and the MC just hears silence and doesn’t know they’re busy. |
| “The big picture we have is the ROE [rules of engagement], the unit, and the JTAC.” It is almost impossible to have the entire picture of a mission (e.g., often operators aren’t always given information to know why the target is important). | One strategy is to try to keep the same MC on the same mission for a week. Another strategy is the MC conducts his/her own searches on websites to learn more about the targets. A third is to ask the Senior MC for background. | The customer (supported unit) doesn’t supply the background information to the MC, even though it would help them be better able to anticipate the target’s next moves. |
| When it’s a [friendly and hostile] troops-in-contact (TIC) situation, the big picture is who is in contact, who they are being engaged by, what the terrain is like, etc. Need to get a mental image beforehand. | Immediately plot the location. Look briefly at the terrain and the friendly forces, and then focus on the enemy. Help the sensor operator because the sensor may be focused on getting a good picture during the stressful situation. | A novice may focus on the friendly forces in the video rather than looking for snipers. |
| Aspects of Expertise | Cues & Strategies | Why Difficult? |
|---|---|---|
| Self-monitoring: It is important to get a sense of the personalities of the people you are coordinating with over chat (e.g. supported unit; Intelligence, Surveillance, and Reconnaissance Cell (ISARC); Predator Liaison Officer, etc.) and be able to adapt what you say based on the personality and circumstances. | Can get an explicit reading of the personalities involved from the departing MC during a shift change. | Have no knowledge of the people on the other end of the chat and it is difficult to build relationships over time because of shift changes and the fact that people don’t often know which individuals they are chatting with (they are known by their roles, not names). |
| Job Smarts: When following a vehicle, need to anticipate where it might go next and communicate that to the crew. | Consider the context, such as if it is nearing prayer time to deduce if the vehicle is headed to a nearby mosque. | The customer (supported unit) doesn’t supply the background information to the MC, even though it would help them be better able to anticipate the target’s next moves. |
| Job Smarts: Ensuring the coordination between the pilot and JTAC in a situation with troops in contact (TIC) goes smoothly. If the situation arises, they immediately need to connect with the JTAC to ensure they’ll be able to coordinate over voice with the pilot. | Immediately when a TIC arises, begin coordination with the supported unit and the JTAC to ensure he will be able to communicate with the pilot. | A novice may wait until asked to ensure the pilot and JTAC will be able to coordinate. |
| Job Smarts: Need to be confident and speak up to let the pilot and/or sensor operator know if you feel they need to change something they are doing. | Refer to the crew as “MC”, “sensor”, and “pilot” rather than referring to each other by ranks. | Lower ranking MC sometimes will not feel comfortable suggesting a superior do something different. |
| Events | Actions | Assessment | Critical Cues | Potential Errors |
|---|---|---|---|---|
| Troops in Contact (TIC) | Read about TIC in Air Surveillance Operations Center (ASOC) chat room. If the TIC is occurring near another MC’s aircraft, warn that MC. Learn when troops will make contact, ask in advance for communication frequencies. | Need to make sure that the pilot and JTAC coordinate. Need to determine or establish who has sensor tasking authority (JTAC or supported unit). | What direction are they moving in? How many? | Not asking unit in advance for information, so always playing catch-up. Not warning an MC whose aircraft is in the vicinity of the TIC. |
| Following vehicle | Describe vehicle to crew (such as colour and type). Coordinate with SO so someone always has their eyes on the target. Use Tivo to help pinpoint a vehicle if the crew loses sight of it. Provide voice reports to the crew about terrain/view obstructions. | Deduce what direction vehicle may travel based on time of day. Use intelligence data to determine habitual travel routes. | If it’s prayer time, it may be heading to the mosque. | Not keeping “eyes on” the target, not setting up a surveillance orbit correctly. |
| Hellfire shot | When supported unit asks about preparing for HF shot, offer terrain assistance to SO. Observe crew manoeuvres to get in a position to fire. Report situation to JTAC and request permission to fire. | Need to look for conditions that would satisfy ROEs. Need to provide a sanity check for the crew. | Conditions that are specified in ROEs. | Not scanning around for hostiles; letting your eyes be drawn to friendly forces or to events that distract from looking for hostiles who are hiding. |
- We identified three cognitive challenges for the MC operator at a very high level, as can be seen in Table 4. There are significant cognitive demands involved in using many of the tools and in performing more detailed actions but we started with the three high-level tasks that were mentioned repeatedly by interviewees as being especially challenging. Note that the first challenge, finding and keeping track of dynamically changing information, corresponds to Level 1 and Level 2 Situation Awareness: perceiving and comprehending information within a volume of time and space [3]. The second challenge, anticipating what the target will do next, corresponds to Level 3 Situation Awareness (predicting how the situation will evolve). The third challenge, coordinating with the right person at the right time, pertains to how Situation Awareness is shared among team members.
Implications for design
Because the ACTA documents key cognitive challenges in a compact tabular form, it can provide an efficient means of presenting this information to multiple stakeholders. Such a presentation can be especially important to stakeholders who are either unable to conduct their own site visits, or cannot have as much time observing users as they desire. Some of these stakeholders could be systems engineers who are gathering functional, performance, or usability requirements.
In our case, the results of the ACTA influenced the design of a weather avoidance capability to improve the POC personnel’s tactical picture. A group of technologists was assigned the task of enhancing POC operators’ tactical displays but were unable to visit the POC to learn where they should focus their efforts. Thus, they were informed by our ACTA results. They decided to address the cognitive demand of anticipating how the situation may change, and they learned from the ACTA that getting weather updates was a strategy used by the POC personnel to anticipate the need for potential routing changes. Currently, there is only one weather forecaster on duty to assist all of the Predator crews. By directly viewing radar of weather parameters that can endanger the Predator aircraft on their own displays, the MCs are better prepared to advise the pilot to reroute the aircraft to maintain uninterrupted target surveillance. Without consulting the ACTA, the technologists may have developed capabilities that did not address any of the POC personnel’s significant cognitive challenges.
Also as a result of the ACTA, we made several recommendations, such as providing an easy way to see where the Predators have looked previously and improving their processes for filling out mission reports and leveraging target folders. Each of our short-term recommendations has been addressed with new tools and processes since our visit.
In the future, we plan to use the ACTA as a tool to compare how operators’ cognitive challenges change over time. Because it is a highly structured means of capturing cognitive task analysis data, the ACTA can facilitate picking out differences in tables compiled at one point in time versus another. Once analysts have pinpointed the changes, they can direct their attention to probing the specific circumstances surrounding the changes. Of particular interest would be any changes in the ACTA that can be traced back to a change in interaction design or work process modification.
We can envision a new evaluation technique based on a longitudinal study of repeated ACTA analyses. A potential difficulty in using ACTA for evaluating interaction design, however, may result when the contents of the ACTA tables are unchanged over time but the severity of the cognitive challenges increases or decreases. It is possible that ACTA could be combined with a cognitive workload measurement technique, such as the cognition-related components of the NASA Task Load Index [9], to overcome this difficulty.
| Difficult Cognitive Element | Why difficult | Common Errors | Cues and Strategies used |
|---|---|---|---|
| Finding and keeping track of dynamically changing information | Impossible to look at multiple windows simultaneously. | Manual updates of info can be error-prone (includes handwritten updates to the Scheme of Manoeuvres—the tasking); can miss a critical update of information. | Scan a subset of chat more frequently. Use the Tivo to back up and see something that happened recently. Say, “head down” when looking at something other than the video, so others can keep their “eyes on”. |
| Anticipating what the target will do next or how the situation may change. | There are many different ways the situation could change and there is often insufficient context to be able to predict. | Being reactive instead of successfully being prepared for future changes. | Knowing what a similar target did yesterday, understanding the environmental factors such as weather or time of day or terrain or proximity of mosques or other significant entities. |
| Coordinating with the right person at the right time | Lines of coordination are ambiguous; lack of contextual info when being tasked; multiple-levels of security hamper information flow; sometimes need to resolve conflicting requests when multiple customers feel their tasking is urgent. | Transmitting incorrect information due to passing tasking through many people; interrupting the pilot when he/she is talking to someone else. | Learn the information presentation styles of remote collaborators (e.g., person 1 has a curt manner, person 2 has a relaxed manner). |
Reflections on the acta analysis
We validated the ACTA findings by having a representative group of POC personnel review the results and note what was incorrect. Nearly all of the changes the reviewers recommended pertained to our description of their coordination, which is not part of the ACTA. Because of this independent review, we were confident that the ACTA accurately captured their major cognitive challenges.
While not a part of the ACTA method, we arranged to spend time observing operators in the POC. We watched operators spending a significant fragment of their time talking on the telephone, reading chat, and typing in their own chat postings. The interview questions posed by the ACTA method did not lead us to understand who coordinated with whom for what purpose, via what means, and how frequently. We wanted to know this information because of the potential benefits of selectively automating the transmission of critical information between collaborators that is currently entered by hand or passed by voice. Also, collaboration technologies research could potentially be applied to make it easier for frequent collaborators to have better awareness of each others’ activities. We supplemented the ACTA by asking the POC personnel to describe who they coordinate with, by what means, how frequently, and what information they shared. Each person we interviewed described a slightly different coordination flow and we were unable to obtain a consensus during the very brief visit, which means that the coordination procedures vary too significantly to be described in a few “typical” procedures or our questions were not sufficient. Regardless, we found it very difficult to understand coordination procedures in addition to the major cognitive challenges within the three-day period.
Another type of information that ACTA did not specifically uncover pertained to non-cognitive, social behaviour patterns. For example, we learned that POC personnel have little external motivation to record comprehensive Mission Report information because they do not know how the information is used by the end consumers—or even who the end consumers are for this data, other than the “intelligence community.”
An advantage of the ACTA methodology is that it provided us with a structured way to gather and organize a great deal of information during a short time period. Creating the task diagram, knowledge audit table, and simulation summary table helped us to synthesize the content of each into the major challenges. Because the cognitive demands table presented the major findings in a concise way, it was a useful communication tool both to the POC personnel to validate our findings and to technologists to develop solutions to address the challenges.
- The ACTA was useful for quickly understanding the major cognitive challenges and can be applied to other time-critical and safety-critical tasks. Our application of ACTA was unique because we were conducting it within an extremely short time period in an actual operational military setting as opposed to a training or simulation event.
Acknowledgments
This work was supported by the United States Air Force Electronic Systems Center and performed under MITRE Mission Oriented Investigation and Experimentation (MOIE) Projects 03067531, 03077543, 03067539, and 03077544 of contract 19628-94-C0001. All product names, trademarks, and registered trademarks are the property of their respective holders.
References
S.F. Chipman, J.M. Schraagen, and V.L. Shalin, “Introduction to Cognitive Task Analysis”, in Cognitive Task Analysis, Schraagen, Chipman, and Shalin (eds.), NJ; Lawrence Erlbaum, 2000, pp. 3–24.
M.F. Eddy, H.D. Kribs, and M.B. Cowen, Cognitive and Behavioral Task Implications for Three-Dimensional Displays Used in Combat Information/Direction Centers, Technical Report 1792, San Diego: Navy SPAWAR Systems Center, 1999.
M.R. Endsley, “Design and Evaluation for Situation Awareness Enhancement”, in Proceedings of the Human Factors Society 32nd Annual Meeting, Santa Monica, CA, Human Factors Society, 1988.
M.R. Endsley, B. Bolte, and D.G. Jones, Designing for Situation Awareness: An Approach to Human-Centered Design, London: Taylor and Francis, 2003.
J.C. Flanagan, “The Critical Incident Technique”, Psychological Bulletin, Vol. 51 (4), 1954, pp. 327–358.
M. Gladwell, Blink: The Power of Thinking Without Thinking, Boston: Little, Brown and Company, 2005.
E.M. Hall, S.P. Gott, and R.A. Pokorny, A Procedural Guide to Cognitive Task Analysis: The PARI Methodology, AL/HR-TR-1995-0108. Brooks AFB, TX: Air Force Materiel Command, 1995.
N. Hardinge, “Cognitive Task Analysis of Decision Strategies of Submarine Sonar and Target Motion Analysis Operators: Phase I”, Final Technical Report, Contract No. N00014-99-1044, Washington, D.C.: Office of Naval Research, 2000.
S.G. Hart, and L.E. Staveland, “Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research”, in Human Mental Workload, P.A. Hancock and N. Meshkati (eds.).Amsterdam: North Holland Press, 1988, pp. 239–250.
H. Johnson, and P. Johnson, “Task Knowledge Structures: Psychological Basis and Integration into System Design”, Acta Psychologica Vol.78, 1991, pp. 3–6.
G.A. Klein, R. Calderwood, and D. MacGregor, “Critical Decision Method for Eliciting Knowledge”, IEEE Transactions on SMC, Vol. 19(3), 1989, pp. 462–472.
G. Klein, “Cognitive Task Analysis of Teams”, in Cognitive Task Analysis, Schraagen, Chipman, and Shalin (eds.), NJ; Lawrence Erlbaum, 2000, pp. 417–430.
L.G. Militello, R.J.B. Hutton, R.M. Pliske, B.J. Knight, G. Klein, and J. Randel, “Applied Cognitive Task Analysis (ACTA) Methodology”, NPRDCTN-98-4.San Diego: Navy Personnel Research and Development Center, 1997.
L.G. Militello and R.J.B. Hutton, “Applied Cognitive Task Analysis (ACTA): A Practitioner’s Toolkit for Understanding Cognitive Task Demands”, Ergonomics, Vol. 41(11), 1998, pp. 1618–1641.
T.E. Miller, R.R. Copeland, J.K. Phillips, and M.J. McCloskey, “A Cognitive Approach to Developing Planning Tools to Support Air Campaign Planners”, Rome, NY: Air Force Research Laboratory, 1999.
J. Phillips, P.L. McDermott, M. Thordsen, M. McCloskey, and G. Klein, Cognitive Requirements for Small Unit Leaders in Military Operations in Urban Terrain, Research Report 1728.Alexandria, VA: US Army Research Institute for Behavioral and Social Sciences, 1998.
J.M. Schraagen, S.F. Chipman, and V.L Shalin, Cognitive Task Analysis, NJ: Lawrence Erlbaum, 2000.
T.L. Seamster, R.E. Redding, and G.L. Kaempf, Applied Cognitive Task Analysis in Aviation, London: Ashgate, 1997.
A. Shepherd, Hierarchical Task Analysis, London: Taylor and Francis, 2001.