Human Factors Analysis Of Accidents In System Of Systems

Introduction

Modern systems are increasingly complex and can produce dangerous emergent phenomena. Systems comprise complex interrelations between people (such as pilots and air traffic controllers in command structures) operating technology (such as aircraft, communication systems, command systems and weapons) to perform mission and tasks that create effects. The System of Systems (SOS) approach appreciated that the system cannot, and indeed should not, be broken down into its constituent parts. Rather, understanding and explaining the emergent phenomena of the whole system comprising people and technology working together needs to be examined. In this paper, a case study of the Black Hawk Friendly Fire Shoot Down (BHD) Incident [1] is used to illustrate this complexity and emergent behaviour. To evaluate the usefulness of the four Human Factors (HF) methods for SOS analysis in the military domain, each method was applied to the BHD incident. A SOS as defined here as the combination of independent systems into a larger interdependent system that delivers capabilities that is beyond the individual systems (that is, the SOS is greater than the sum of the individual systems that it comprises). For instance, in the BHD system, the enforcement and management of a No-Fly Zone (NFZ) which required co-ordination and communication between Airborne Warning and Control System (AWACS) controllers, communication technology, patrolling aircraft, pilots and Identification Friend or Foe (IFF) technology.

In the aftermath of the Gulf War, a United States-led coalition mission, code-named Operation Provide Comfort (OPC), was set up in order to provide protection for the Kurdish refugees within Northern Iraq. A Tactical Area of Responsibility (TAOR) was defined that included a NFZ in order to monitor the movements across Northern Iraq and intercept any hostile activity. On 14 April 1994, two United States Air Force (USAF) Black Hawk helicopters were shot down by two F-15 fighter jets, in a friendly fire incident in the TAOR, having been misidentified through a series of errors as enemy aircraft Iraqi “Hind” helicopters. All 26 armed forces and civilian personnel aboard the two helicopters were killed.

A systems-related accident is an unexpected, emergent phenomenon that occurs as a result of unanticipated interactions between components of a complex system [5]. The outlined BHD incident was a SOS accident as there were multiple, interconnecting systems, all separately distributed but working together on the day of the shoot down—that is, the Black Hawk helicopters, the F-15 jets, the AWACS and the TAOR. Errors occurred at multiple levels of a system and there was no singular mistake that accounted for the shoot down. According to Leveson [6], over 130 separate mistakes by multiple operators interlinked to lead to the BHD incident.

Previous analysis into the BHD incident [1,3,4,6,7] have found that the key causal factors were:

Poor Intra-SOS Communications

Lack of transfer of information from the AWACS to the F-15 pilots regarding the Black Hawk helicopters.

The Black Hawk helicopters were unaware of the incorrect Identification Friend or Foe (IFF) responder code being used and were not made aware whilst flying in the TAOR.

The helicopters were not fitted with sufficient contemporary radio equipment in order to communicate directly with the F-15 pilots.

Poor Emergent Behaviour

Insufficient visual identification training was given with regard to recognition of friendly and enemy aircraft, leading to the misidentification of the Black Hawks as Iraqi “Hind” helicopters.

Poor Managerial and Operational Independence

Insufficient Rules of Engagement (ROE) training leading to miscommunications and misunderstanding about ROE.

Helicopter missions were not viewed as part of the air remit of the Combined Task Force and subsequently were not adequately integrated into mission schedules nor monitored within the TOAR appropriately.

Problems Related to Distributed Ownership

There was unclear guidance as to role of regulations throughout the coalition forces: this was particularly true with regard to helicopter missions—that is, it was unclear who was in charge of regulating helicopter mission information.

The HF approach recognises that a technological failure or individual human error is not the sole cause of an accident [8,9]; rather, accidents are seen as emergent phenomena [10] which occur as a result of smaller, latent and immediate interlinking failures across multiple levels and by multiple culprits—that is, human, machine and organisation errors [9,11,12]. This a key concept in SOS, as risk increases as a consequence of the size and complexity of a SOS. Higher risk is also associated with the distributed ownership inherent to a SOS. It is therefore important that HF methods for risk analysis can deal with this scale of interaction, so that problems can not only be investigated, but also anticipated before accidents occur (as accidents arise from emergent behaviour in routine and everyday activities).

In the following sections, HF risk analysis techniques are reviewed in relation to the above criteria. The potential for error analysis methods to be applied within a military SOS context is also explored via a case study of the Black Hawk Down incident. There are many HF methods for identifying and analysing risk within systems; however, these have traditionally focused on sequential or “chain event” causes of error, categorisations of error and individual causal events [13]. By contrast, SOS risk results from the compound of events—one event interacting with another to create risk—in a layered network of systems.

System of system analysis methods

There are, however, some methods which have the potential to support useful analyses of the interactions between the multiple layers involved in a SOS, as they are founded in systems theory and apply a holistic multi-leveled approach [1,14]. These methods include AcciMaps, [15,16] the Human Factors Analysis Classification System (HFACS) [17], the Systems Theory Accident Model and Process (STAMP) [6,7], and the Event Analysis of Systemic Teamwork (EAST) [18]. These methods were developed to address the short comings of the traditional individual-focused approaches and as a consequence of the recognition of a need for system based analyses for complex problems [7,14,16–19].

ACCIMAP

The AcciMap method is an accident analysis technique that is used to investigate causal relationships between decision and actions [15,16]. It combines cause-consequence events mapping with pictorial representations of the various individual systems which make up a SOS, in a graphic format [16]. The output provides a multi-layered chart that displays the causal interrelationships within and between the systems and illustrates any known failures within a system that ultimately lead to an accident. It is arranged in multiple layers, which cover the actions of: government policy, (such as budgets, official and unofficial regulations), regulatory bodies and associations, (such as culture, training, guidance, regulatory policy), company management, (such as departmental policy and training), technical and operational management, (such as communications and procedures), physical processes and actor activities, (such as physical actions that occur within the system such as radio communications), equipment and surroundings (such as function and availability of impact of the weather and wider environment). The resultant AcciMap diagram represents each of these categories as horizontal layers, with the largest system (government policy) at the top and the smallest system (equipment and surroundings) at the bottom level. According to the AcciMap analysis for the Black Hawk shoot down incident, the pilots of both the Black Hawk helicopters and the F-15 jets had the highest proportion of errors; however, although the pilots of both aircraft had the highest proportion of errors, they had no initiating factors associated with their actions—that is, the initiation for their actions came from elsewhere in the system. Initiating factors are outlined using a box with an arrow on the diagram and are cumulated in Figure 1. The analysis suggests that the pilots of both aircraft were not wholly to blame and their errors were as a result of previous system errors—such as problems at a government and regulatory bodies levels as well as errors occurring with the equipment (inadequate radio use and incorrect frequency).

HFACS

The Human Factors Analysis and Classification System (HFACS) was developed by Weigmann and Shappell [17] as a method to study military aviation accidents; however, it can also be applied to a range of diverse areas—such as civil and military aviation, [17,19–21], helicopter transport [22] and road transport [23]. It consists of a series of error taxonomies, each set at a different level in Reason’s Swiss Cheese Model [8,12]. There are four taxonomy levels each complete with a sub-taxonomy. The errors at each level are a mixture of latent conditions (such as poor design) and active failures (human errors [8,17]). The latent conditions are inadequate conditions or failings within a system and can be determined by higher level decisions, whereas unsafe acts tend to be at a lower operational level (the “sharp end”) and can have a more immediate impact on systems safety [8,9,11]. HFACS analysis for the BHD incident described that higher levels within the system—that is, unsafe supervision and organisational issues—had the highest proportion of errors that led to the incident. This corresponds with previous government reports that the Black Hawk and F-15 aircraft pilots were not the main cause of the incident. The initial categorisations in HFACS analysis were then further broken down to reveal specific type of error within each level (see Figure 2). From the HFACS breakdown into the most prevalent causes (organisational issues and unsafe supervision) the analysis shows that within the level of organisational issues, the main type of errors were resource and acquisition management errors and within the level of unsafe operations, there was a higher proportion of planned inappropriate operations. HFACS was thus able to identify key causal errors that led to an accident as well as errors that existed at the lower levels [1].

STAMP

Systems Theoretic Accident Model and Process (STAMP) is an accident analysis technique that explores accident causation with systems and systems safety in mind [7,24]. STAMP is founded on the premise that a system is a series of interrelated components, interconnected and maintained by information and control feedback loops and that systems safety is dependent on the control of the systems interaction and views [7]. STAMP is based upon three basic concepts: constraints, hierarchical levels of control and process models. The model suggests that accidents occur where there is inadequate control of safety constraints on the system, which in turn can cause external disturbances. It also shows that component or interaction failures cause an undesirable reaction [7]. Control can be implemented at various levels, including managerial, organisational, physical and manufacturing. The impact of control at these levels can vary from system to system [25]. It is the lack of control that leads to violation of safety restraints and results in unplanned emergence found within complex systems [25]. STAMP has a control flaw classification system [7] that is applied to each level and control loop in the system’s control structure in order to identify all control failures (see Figure 3).

Leveson [7] proposed a control flaw classification scheme as a taxonomy to identify control failures and causal factors at each systemic level. Using this taxonomy, the errors at each level were identified and were classified into core causal factors.

The system level analysis outlines that there were a number of errors committed across multiple levels, particularly at the AWACS level (see Figure 4). When grouped by taxonomic types of error, the most frequent error types were caused by inadequate action on the part of the actuator and from an actor’s process models being inconsistent, incomplete or incorrect. This analysis suggests the errors leading to the incident primarily involved a series of inadequate control hazard operations and the actors involved had a flawed or inadequate mental representation of their role and the situation. This led to erroneous actions proceeding to develop within the system, control is therefore compromised and the incident occurred. These results suggest that in line with previous research [6,7] and government enquiry [2,3], the F-15 pilots were not wholly to blame for the shoot down and the accident occurred as a result of multiple interacting failures.

EAST

EAST [18] offers a way of systems as a network of networks and provides a framework of methods that allows collaborative performance to be comprehensively described and evaluated. Since its conception, the framework has been applied in many domains, including land and naval warfare [26], aviation [27], air traffic control [28], railway maintenance [29] and the emergency services [30]. EAST is underpinned by the notion that complex collaborative systems can be meaningfully understood through a network of networks approach. Specifically, three networks are considered: task, social, and information networks. Task networks describe the goals and subsequent tasks being performed within the system. Social networks analyse the communications taking place between the actors working in the team. Information networks describe the awareness that the different have during task performance, as shown in figure 5, coded by the agents involved. The EAST framework lends itself to in-depth evaluations of complex sociotechnical system performance, examination of specific constructs within complex sociotechnical systems (such as SA, decision making, teamwork), and also system, training, procedure, and technology design. The analyses produced are often highly useful in identifying specific issues limiting performance or generating system redesign recommendations. EAST is an integrated framework formed from a combination of a number of HF methods and is composed of three levels: data collection methods, data analysis methods and representational methods [18].

The information diagrams revealed that the F-15 pilots held no knowledge of the Black Hawk helicopter mission and that they held contrary information—that is, that no flight should enter the TAOR whilst they were performing a sweep. Standard protocol within a NFZ also dictated that all flights should be identifiable and tracked by the AWACS. It was found that in the actual scenario there were no individuals who held enough appropriate schemata for all required aspects of the mission; whereas in the ideal scenario there were a number of appropriately informed key agents who held enough schemata in order for the mission to proceed effectively.

Method x Criteria

Based on a review of the literature, and analysis of how effectively each method was applied to a well-established SOSA accident, a table was constructed to summarise the results from the application of the four proponent methods against the HF and SOSA criteria (see Table 1). The SOSA criteria [1] are listed from 1–5 and the general HF methods criteria [31] are 6–11. From Table 1, it can be seen that the EAST method is rated the highest with regard to the SOSA criteria as it completely fulfils criteria 2-5 and partially fulfils criteria 1 (explore all elements of SOSA). STAMP and AcciMaps are rated equally, completely fulfilling criteria 2-4, partially fulfilling two criteria and not fulfilling one criterion each. AcciMaps partially fulfils criterion 4 (provide clear guidance), whereas STAMP is a complex method and there is little clear guidance on how to conduct the method and thus does not fulfill criterion 5. AcciMaps is not able to fulfill criterion 6 as it cannot account for local rationality and reasoning behind the actions of actors within the system, whereas STAMP can begin to account for local rationality. Both AcciMaps and STAMP partially fulfill criteria 1 (explore all elements of SOSA). HFACS completely fulfils two out of six SOSA criteria (identify key decision makers involved and provide clear guidance), however it is unable to fulfill criteria 2 (explain causal relationships) and 6 (local rationality). HFACS is able to partially fulfill criterion 1 (explore all elements of SOSA). When examining the general methods criteria, HFACS was rated the most favourable, when accounting for completely fulfilled criteria and subsequently partially fulfilled criteria. HFACS is able to completely fulfill three out of six criteria, partially fulfill two criteria (resources and reliability) and was unable to fulfill criterion 8 (validity). AcciMaps is rated higher when considering general methods criteria that are completely accounted for, whereas EAST is rated higher when considering the total number of criteria at least partially fulfilled. AcciMaps was able to completely fulfill three of the general methods criteria (resources, related methods and underpinning theory), partially fulfill criterion 11 (application time) and was unable to fulfill criteria 7 (reliability) and 8 (validity). EAST was able to completely fulfill two criteria (related methods and underpinning theory) and was able to partially fulfill all other criteria. STAMP was rated the least favourable in general methods criteria, completely fulfilling two criteria (related methods and underpinning theory), partially fulfilling one criterion (resources) and not fulfilling three criteria (reliability, validity and reaction time.

Explore all the Elements of SOSA (Criterion 1)

Each of the four methods identified were able to adequately explore the majority of the components within a SOS, were able to explore the distributed ownership within a SOS and were able to investigate each of the erroneous actions of the BHD incident that were outlined by the US government and several previous analyses. AcciMaps and EAST were able to provide the most explicit detail of key factors, such as highlighting the initiating factors of the BHD incident (AcciMaps) and giving comprehensive detail into each action (EAST), however all the reviewed methods analysis, became complex with the scale and complexity of the BHD incident. When effectively employed, the AcciMap method fulfils these criteria through the use of multiple hierarchical levels and uses the actor map investigations to explore all components. HFACS is able to identify all components using taxonomic levels of action, operation, supervisors and organisation; however, its analysis is not as robust as the other methods, providing only a brief and general description of the interactions between systems. This means that HFACS does not provide as much depth as the other three analyses. STAMP analysis is able to determine each system component through the strict use of the hierarchical control structure. The control structure and the control map allow for all decision makers to be identified. EAST employs HTA to explore all tasks and actions, as well as using a SNA which explores all the “nodes” within a system and defines their interactions.

Explanation of Causal Relationships (Criterion 2)

An AcciMap provides a graphical representation of all the interrelated factors and how they are linked; however, in a scenario like the BHD incident, there are numerous causal factors, which makes the analysis complex. HFACS was unable to explore all the causal links between levels because there is nothing in the various HFACS taxonomies to account for causal relationship explanation; in other words, errors can be identified, but the interlinking relationships are difficult to explore. The STAMP taxonomy explores erroneous actions at differing stages of emergence—that is, a physical chain of system events, conditions surrounding the events and overall weaknesses. According to Leveson [7], this method allows for views of the multiple stages of an incident, and thus relationships between the various levels can begin to be analysed. EAST uses network analysis and complementary coding models and according to Rafferty et al., [1] and Walker et al., [28] this network of networks illustrates a system’s decision space and is therefore able to map all the interlinking relationships in a SOS.

Identify All Decision Makers Involved (Criterion 3)

All four methods were able to identify the decision makers involved. AcciMap is able to do this through the use of the actor map and then the overall AcciMap as a graphical representation of a SOS. HFACS identifies all decision makers through the use of the HFACS taxonomy, which identifies problems at each level. Through the use of its control structure and hierarchical control loops, STAMP is able to identify all decision makers involved in a system. EAST is able to identify all decision makers, through the use of HTA and SNA.

Provide Clear Guidance (Criterion 4)

AcciMaps did not provide a taxonomy of identifying errors and the analysis involved exploring all strings of error; however, the organisation and structure of an AcciMap is useful, adaptable and clear [1]. There are also guides to conducting AcciMaps in several HF methods books [31,32,33] and a number of studies have applied the method successfully [34–36]. The HFACS taxonomy and sub-taxonomy are easy to use and act as a guiding method for analysis [1]. The STAMP method is designed to help the analyst to explore and identify error causality; however, previous authors have criticised the method for providing little guidance to analysts [10,34]. EAST is highly complex and uses multiple methods. It is well documented in recent HF books [18,32].

Local Rationality (Criterion 5)

Svendung and Rasmussen [16] suggested that local rationality cannot be removed from a decision making situation analysis, however AcciMaps only provides a graphical representation of a relationship and there is no guidance on how to explore local rationality [34]. As a taxonomic method, HFACS is unable to adequately explore local rationality. It does provide a description of the errors that occur; however, the taxonomy cannot account for the local reasoning behind the error. STAMP provides an illustration of the context in which the decisions were made, allowing for some local rationality to be considered and understood. STAMP attempts to explore the expectations, goals and motives of the human components of a system [7]; however, as the method has a taxonomic nature and is a guide for the analyst, it has been argued that the method fails to provide enough guidance for exploring local rationality [1]. EAST explores local rationality through HTA analysis and information networks. The HTA analysis outlines system and individual goals and the information networks investigate system and individual distribution of information—that is, activated knowledge and SA.

Resources (Criterion 6)

AcciMaps requires little in the way of specialist tools (such as Microsoft Visio or Adobe Illustrator) significant resources might need to be invested in collecting data to support the analysis. It is also recommended that SMEs are consulted in the end stages of refining the overall diagram [32, 33]. Simple HFACS analysis can be conducted using pen and paper; however, a drawing package is required to create a usable data output and a statistical data processing package is required in order calculate reliability and analyse the different failure associations between levels. Access to the relevant taxonomies is also required in order to complete a HFACS analysis. HFACS is easier to employ if the analyst has some experience with HF or HF concepts such as Reason’s Swiss cheese model of defence however, the HFACS method does not normally require access to SMEs in order to conduct the analysis [32]. STAMP can be employed using pen, paper, whiteboards and flipcharts, followed by a visual data software package to construct the final data outputs. SMEs are required during STAMP analysis to ensure validity of the process [32]. The method is complex and therefore the analyst would require some HF methods training or access to more experienced analysts / SMEs in order to complete the task [32]. EAST is a resource-heavy method as it requires access to a number of different tools, such as drawing and statistical packages, due to its methodological nature as well as access to SMEs for validation. A new user would require a high level of training and access to SMEs in order to complete the analysis. The method is also supported by a software tool (Workload, Error, Situational Awareness, Time and Teamwork: WESTT) which reduces application times [18,32].

Reliability (Criterion 7)

AcciMaps have been successfully used to map the processes leading to other major events, such as the Longford Esso Plant explosion in Australia [35], as well as systems accidents, such as the mistaken Stockwell Shooting [35] however some researchers have argued that as AcciMaps lacks a taxonomy and specific guidance, inter-rater analysis is likely to be subjective and reliability is likely to be low [14]. HFACS has previously been applied within several safety critical domains including civil and military aviation [17,19, 20,21], helicopter maintenance [22] and road transport [23]. Salmon et al [32] report on studies that show acceptable levels of reliability for HFACS. They suggest that the taxonomic approach is likely to improve consistency but reduces the flexibility in error classification, particularly when applied outside of aviation accidents [14]. Some research however has suggested that inter-rater reliability results for the HFACS method are low [37]. The STAMP taxonomy is more general and therefore less restrictive than that used by other methods, such as HFACS [14]; however, this can lead to reduced inter-rater reliability. STAMP has previously been applied in the military domain (such as friendly fire, [4]) and aviation [38]. It can be argued that due to the limited guidance available for STAMP that reliability is also likely to be low due to discrepancies between different analysts’ descriptions [1,32]. Due to the number of methods used in EAST there is likely to be a low level of reliability among analysts; however, this multi-method approach does increase the comprehensiveness of the analysis and enables results to be verified by more than one method, in a process of data triangulation [32].

Validity (Criterion 8)

There is little data available on the validity of AcciMaps, HFACS and STAMP [14,32]. Validity is difficult to determine within EAST but higher validity is assumed as a result of the involvement of an SME in the analysis [18]. The various methods used in the EAST methodology are similar in theoretical background and this allows for viable and effective method integration. This has been argued to be an example of congruent validity [31]. Previous work has suggested that EAST has notable construct and face validity, as the component features of interest are able to be observed within the intended environment and the methods are easily relatable to constructs within the military domain [18,31].

Related Methods (Criterion 9)

Collecting data for the initial AcciMap, HFACS, STAMP and EAST analysis often requires the use of multiple data collection methods, including interviews, questionnaires, observational studies and documentation review [32]. There have been some attempts to extend the HFACS framework for specific domain analysis—such as the Human Factors Analysis and Classification System – Maintenance Extension (HFACS-ME) to analyse failures in helicopter maintenance [22]—which suggests that the method is adaptable and can be integrated with other techniques to enhance the analysis. EAST is a methodology that is comprised of various different HF methods, including HTA [39], CDA [40], SNA [41], the CDM [42], OSDs [31], CUD [43], and propositional networks [44].

Underpinning Theory (Criterion 10)

The AcciMaps method is founded in the systems approach and is based on Rasmussen’s risk management framework [15]. In order to collect a suitable amount of data for the analysis, multiple data collection methods are used, such as interview, observations and questionnaires [32]. STAMP has a sound underpinning theory [7]; however, as this is routed in control theory and systems engineering, the language used can make STAMP difficult to interpret and apply to complex human decision based failures [14]. HFACS is based on Reason’s Swiss Cheese Model of organisational management [8,12], and acknowledged as a systems approach theory [10]. This theory suggests that a system has multiple levels and each level has boundaries. Any potential risks or flaws in a system are analogous to the holes in slices of Swiss cheese. When all the errors directly interlink without meeting a barrier then an accident occurs. If a system is built with directly interlinking risks or “holes”, the system is fundamentally flawed. The HFACS taxonomies are structured on the levels in Reason’s model—that is, organisational influences, unsafe supervision, preconditions for unsafe acts and unsafe acts [32]. The EAST method comprises a number of frequently used HF methods, such as HTA [39]. EAST is based on the concept that systems are underpinned by networks, such that a SOS will be supported by a network of networks. It is based on a theory of distributed cognition that interleaves social, informational and task networks.

Application Time (Criterion 11)

AcciMaps are easy to learn; however, depending on the size of the system, gathering the initial data and construction of the map can be time-consuming activities [32]. HFACS requires little training, providing the analyst has some previous knowledge of HF [32]. STAMP is based on a complex model [4,6,7], and thus the method is complex and has a long training time The data collection and the analysis stages are likely to be time consuming [32]. The concept of EAST is simple; however, due to the comprehensiveness of the method it takes a long time to train and ideally requires some prior experience in HF. The application time may be reduced if the WESTT software package is used [18,32].

Discussion

The aim of this paper was to review HF methods with a view to assessing their suitability for analysing the SOS (comprising interactions between military personnel and battlefield technologies). Four methods (AcciMaps, HFACS, STAMP and EAST) were applied to an established SOS accident—the BHD incident in the aftermath of the Gulf War—and the suitability of these methods for assessing human related risk within a SOS were evaluated.

AcciMaps analysis provides a graphical representation of a system and from this, flaws in a SOS may be identified. AcciMaps can be used to analyse all the elements within a SOS, explore all errors and identify all key decision makers. It also provides an illustration of interlinking causal factors across multiple layers of a SOS. Although the use of AcciMaps is well documented in recent HF literature, there is little in the way of guidance for the technique and there is no way of accounting for local rationality in the diagram. As the method moves away from taxonomic labels, the analysis can vary between analysts, negatively impacting reliability. The analysis requires few resources to complete and can be employed using a paper and pen, and requires little HF knowledge. The initial data analysis may require an array of different methods and can be time consuming, and if the system is complex then the analysis can become complicated.

HFACS is a taxonomic method that can be used to explore all the elements and decision makers within a SOS; however, the analysis is not comprehensive and there is no provision within the taxonomies to account for local rationality or explain the relationships found. The advantages of HFACS include it has comparatively favourable reviews of reliability, clear guidance from the taxonomies which reduces application time, low resource demands and a sound theoretical grounding. HFACS may be best utilised as an initial guide to exploring a SOS, to be used in collaboration with other more in-depth methodologies [1].

STAMP analysis is a generic method that can be applied to a SOS. It fulfils all of the SOS criteria with the exception of being able to provide clear guidance. The STAMP methodology has been criticised for its subjectivity, as the analysis is determined by the analyst’s interpretation of the taxonomy [1,34]. As the underpinning theory and the method itself are both relatively complex it can also be difficult for new researchers to learn and the method can become time consuming, particularly if the SOS is large or complex. STAMP’s major advantage over AcciMaps and HFACS is that it supports investigations into local rationality. This has been argued by some researchers to be the most important consideration during risk and accident analysis [9, 45]. Leveson [7] also argues that STAMP can be used as a method for prospective analysis. The idea of constraints within systems provides a framework for analysts to design a system where safety constraints would be enforced, preventing a SOS accident [7].

In this review, EAST was rated highest across all the criteria, as the method either complete or partially fulfils all the criteria for SOS and criteria for HF methods (see Table 1). The method can be used to explore all the elements of a SOS in depth due to the use of multiple methods, including HTA. The methodology is well structured and uses a variety of well-established methods. The use of SME in the analysis improves the validity; however, the use of multiple methods, each with varying levels of reliability and validity make the overall reliability and validity of the EAST methodology difficult to determine [32]. EAST can be time consuming, particularly if the SOS is large and complex and requires HF knowledge and access to experts. The application time can be reduced with the use of the WESTT software tool [30]; however, this will increase resource requirements. It is conceivable for the EAST method to be used for prospective as well as retrospective analysis.

Conclusions

All four of the HF risk analysis methods reviewed in this report can be applied within a SOSA, however the depth of analysis required will influence which method is most applicable. AcciMaps and HFACS are able to retrospectively identify flaws within a SOS and are relatively easy to employ. STAMP provides more detail into local rationality across multiple layers of a system; however, the method requires more HF knowledge than the other methods and is more difficult to learn. EAST provides the most in-depth analysis; however, it also requires a higher level of analyst expertise. The type of analysis required—that is, prospective or retrospective analysis will also influence the suitability of the four risk analysis techniques. All of the methods have shown that they can be used for retrospective analysis (that is, after an accident has occurred). AcciMaps provides a graphical representation of failures that have occurred after an accident, and HFACS provides a taxonomy of errors that are usually applied to multiple accidents which have already occurred. It is conceivable that the constraint network component of STAMP could be used to provide prospective analysis in order to determine risk mitigation strategies when designing a new SOS. The EAST methodology has the potential to be used prospectively as well as retrospectively: using the task, social and information networks, EAST would be able to identify potential flaws within a SOS and would be pertinent to evaluating a SOS with regard to distributed cognition and SA. Whether or not any of the four methods explored here can actually be used prospectively is an important goal for future research.

Acknowledgements

This work from the Human Factors Integration Defence Technology Centre was part-funded by the Human Sciences Domain of the UK Ministry of Defence Scientific Research Programme.

References

Authors

Professor Neville Stanton, PhD, holds a chair in Human Factors in the Faculty of Engineering and the Environment at the University of Southampton, UK.

Dr Rafferty is a research fellow in the Faculty of Engineering and the Environment at the University of Southampton, UK.

Ms Blane is a research assistant in the Faculty of Engineering and the Environment at the University of Southampton, UK.