Improvisation In Complex Sociotechnical Systems—a Systems Phenomenon?

Introduction

Improvisation is the spontaneous, real-time conception and execution of a novel response to a situation that is beyond the boundaries for which a system has prepared [1]. It provides one means by which systems can adapt in order to maintain stability when faced with situations for which no procedures exist or where circumstances prevent known procedures from being deployed as specified. Such situations are likely to arise in environments characterised by high levels of risk, uncertainty and dynamism, such as military conflict zones, the wilderness, or natural/manmade disaster zones. Evidence suggests that improvisation can enhance operational effectiveness and safety within complex sociotechnical systems that operate in these environments (for example, [2,3,4]), and that failure to improvise can impair performance and safety [5]. However, other incidents in highly dynamic domains, such as the Mangatepopo Gorge tragedy [6] demonstrate that improvisation also has the potential to negatively impact on performance and safety. Despite this range of outcomes, little is known regarding how positive outcomes can be achieved through improvisation. In short, the concept remains ambiguous [1].

The concept of resilience—the ability of systems to anticipate, respond to, and recover from unexpected, unprepared for disturbances—to which Grøtan and colleagues conclude improvisation contributes [3], has been the focus of significant research attention over the last decade [7] [8]; however, less attention has been paid to the concept of improvisation. Moreover, why positive outcomes were achieved in situations where improvisation has mitigated systems failure is not commonly investigated. As a result our understanding of improvisation and how it can best be supported to impact positively on safety and operational effectiveness in complex sociotechnical systems is limited. This represents a significant gap in the literature; without a comprehensive understanding of improvisation and how it is influenced by factors across all levels of complex sociotechnical systems, any interventions aimed at enhancing positive improvisation are unlikely to succeed.

Within the military context the concept of improvisation remains unexplored; however, it has the potential to impact positively on safety and operational effectiveness in certain circumstances. These include situations during operational missions in which communication mechanisms are rendered inoperative as a result of hostile action, preventing defence force personnel from receiving orders or communicating intentions; situations in which necessary equipment is not available and personnel have to construct solutions from what is available to them in the environment; and situations where forces are faced with a previously unknown form of weapon or threat.

For the potential usefulness of improvisation as a concept used to improve military system performance, there are various pressing lines of inquiry. This paper describes the findings from the first three stages of a research program that aims to enhance our understanding of the concept of improvisation in complex sociotechnical systems, with particular reference to improvisation in safety critical situations

Contemporary thinking from within the Human Factors discipline would suggest that improvisation is a system phenomenon. That is, it is an emergent property of complex sociotechnical systems that is influenced by interacting factors from across the overall system; however, there is little research that has examined this assumption [1].

This paper aims to, first, clarify whether improvisation is indeed a systems phenomenon, and, second, present a prototype model detailing the factors that influence improvisation. Specifically, the first part of this paper examines the extent to which improvisation has been investigated from a systems perspective. The second part aims to address shortfalls in our knowledge in this area by the examining two improvisation incidents using a systems-based framework. Finally, the third part aims to explore the improvisation concept in more depth within a specific context. This is achieved using a survey methodology to determine the relevance of the factors identified from the literature to a particular complex sociotechnical system, the led outdoor activity (LOA) domain.

Stage 1: literature review

Method

Peer review articles were identified from the online databases ‘ScienceDirect’, ‘Web of Knowledge’ and ‘PsycINFO’ using the keywords “improvisation” and “improvise” .The contents of Human Factors journals known not to be contained in these databases were searched separately online using the same terms. Other relevant articles, chapters and books were identified by following the references of these initial articles. A subset of these articles relevant to improvisation in safety critical situations (as opposed to in relation to product development, business outcomes, drama or music) was identified and these articles were categorised according to the factors that they identified as influencing the conception and execution of improvised responses.

Results

The review identified two decision-making models: Mendonça and Wallace’s cognitive model [9], Klein’s Recognition Primed Decision model (RPD) [10], and two performance models: Rasmussen’s Skill-Rule Knowledge framework (SRK) [11] and Klein’s model of flexecution [12] [13], that went some way toward explaining the processes involved in improvisation; however, with the exception of [9], none were developed specifically for improvisation, thus do not give a complete description of factors that influence the appropriateness and effectiveness of improvisations. In fact, no coherent model of the concept, detailing influencing factors and their interactions, was identified [1]

The review identified nine factors described within the literature as factors previously found to influence improvisation in safety critical situations: experience, skill and training; education and systemic knowledge; situation awareness and sensemaking; team work quality; information flows, feedback and communication; role system; organisational memory; organisational culture, values and commitment; and organisational structure. These factors were mapped onto Rasmussen’s systemic risk management framework [14], shown in Figure 1, which includes six organisational levels: government; regulatory bodies and associations; company management; technical and operational management; staff; and work. According to this framework, it is the interaction (vertical integration) between these levels that controls systems’ safety and performance. If improvisation is an emergent property of systems, factors at these systems levels will ‘release’ or restrict improvisation. In the mapping process factors such as organisational culture, organisational structure, organisational memory and role system mapped on to the Company level of the framework, while expertise and experience mapped on to the Staff level, skill and teamwork mapped on to the Work level, and situation awareness mapped to both the Staff and Work levels. Communication and information flows served as a means of integrating information up and down the system levels, while education and training served to integrate information from the company level down to the staff and work levels [1].

From the mapping process it is concluded that the factors identified in the review reside at different levels of complex sociotechnical systems, but the small number of interactions and integrating factors indicated that, to date, the factors have been studied in isolation rather than as interacting components of a system as is appropriate within a contemporary Human Factors approach. The review also revealed that no factors beyond the organisational level (such as at a regulatory of government level) have been identified, despite such factors being vital part of a comprehensive systems-based model [15,14]. For example, in a military context, this means that the effects of changes in defence force funding or international treaties governing the laws of war, such as the Geneva Convention or the Hague Convention, remain unknown. Despite appearing to be an emergent property of complex sociotechnical systems, the review revealed that improvisation has yet to be examined from a systems perspective and hence no systems-based model of the concept exists.

Stage 2: case study analyses

The literature review indicated that factors influencing improvisation reside across different levels of complex systems, supporting the hypothesis that improvisation is a systems phenomenon [1]. To gather further evidence to support this hypothesis the factors that shaped behaviour during real-world improvisation incidents were examined. Two high profile incidents involving improvisation were examined using a systems analysis framework. One involved improvisation that led to a negative safety outcome, and one involved improvisation that led to a positive safety outcome. The first case, the Mangatepopo Gorge tragedy, ended with in the deaths of seven people as they attempted to follow an outdoor activity instructor’s improvised plan to escape a flooded gorge [6]. The second case, the response to the Apollo 13 Lunar Module (LM) consumables issue, involved improvised procedures being developed to allow the crew to survive in the LM with sufficient electricity, water and breathable air to permit the successful re-entry of the spacecraft [16], is a well known example of appropriate, effective improvisation.

Construction of accimaps

The Accimap accident analysis framework was used to describe the factors influencing improvisation during both incidents. Rasmussen developed Accimap as a means of graphically representing factors from different system levels that contribute to accidents [14]. Specifically, Accimap describes the decisions and actions involved across the following six system levels: Government policy and budgeting, Regulatory bodies and associations, Local area government planning and budgeting and company management, Technical and operational management, Physical processes and actor activities, and Equipment and surroundings. Data to support both analyses was derived from documentation review (incident reports [6,17,18], accounts from those involved [19], media reports [20]). One human factors analyst performed the analyses and produced the initial Accimap descriptions. These were then reviewed by a second analyst upon completion. Any discrepancies or disagreements’ between the two analysts were resolved through discussion until consensus was reached.

Results

The Accimap descriptions of both incidents are presented in Figures 2 and 3. Factors at all levels have been presented in summary form with the exception of factors at the ‘Physical processes and actor activities’ level. This level has been detailed in full to provide an understanding the events involved in each incident. The detailed Accimap descriptions are presented in [21]. The Accimap description of the Mangatepopo Gorge incident (Figure 2) indicates that factors at every level of the led outdoor activity system influenced the improvisation. What’s more, these factors interact within and across the systems levels. The impact of higher level factors is propagated throughout the system, for example, high level factors, such as the lack legislation covering occupational safety and health at adventure tourism operators at the ‘Government policy and budgeting’ level and lack of an outdoor industry regulator at the ‘Regulatory body and associations’ level, impacted on lower level factors such as the induction and training provided to instructors at the organisation at which the incident occurred. This in turn resulted in inexperienced and under qualified instructors leading outdoor programs which occurred at the ‘Technical and operational management’ level. This ultimately contributed to the instructor involved in the incident improvising in a way that was inappropriate for the environmental conditions (students left a ledge and entered the river to follow their instructor downstream at the peak of the flood) and ability level of the students (many of the students were not confident swimmers). The improvisation was also ineffective – many students were unable to catch the throwbag the instructor threw to them as they floated past, and even those who did were at an increased risk of drowning as they were carried over a spillway and held under by the force of the water.

In contrast to the Accimap description of the Mangatepopo incident, the Accimap description of the Apollo 13 consumables issue (Figure 3) indicates that factors across all levels of the system can positively influence improvisation. The analysis indicates that factors at high levels of the system, such as legislative and financial support for the Space program at the ‘Government policy and budgeting’ level, impacted on lower level factors such as the provision of state-of-the-art simulators and laboratories at the ‘Equipment and surroundings’ level, which, for example, allowed the rapid development and testing of improvised power up procedures and the Carbon Dioxide filtration system at the physical processes and actor activities level.

Summary

The Accimap outputs for both case studies demonstrate that factors exist across all levels of both systems and interacted with one another to influence improvisation. Combined with the findings from the literature review this provides confirmatory evidence that improvisation is indeed a systems phenomenon that can be influenced by factors residing across various levels of complex sociotechnical systems. Through the Accimap descriptions it is possible to understand how the policies, decisions, and actions at higher levels in the system contribute to the sharp end factors, allowing efforts to enhance effective improvisation to be targeted at these levels. Interventions focusing on the system as a whole are likely to prove more effective than simply making changes to lower level factors while the conditions that led to these sharp end factors remain unchanged within the system.

Having established that improvisation is a systems phenomenon, the next requirement of this research program was to investigate the phenomenon in more depth within a specific context in order to develop a comprehensive model of the factors and their interactions influencing improvisation in this context. The context chosen was the led outdoor activity (LOA) domain. LOA is defined as facilitated or instructed activities within outdoor education and recreation that have a learning goal associated with them (such as school camps, hiking, and abseiling). This domain was chosen because the dynamic and unpredictable environments in which LOA organisations operate was predicated to mean improvisation would be comparatively common, and, as the Mangatepopo tragedy demonstrates, inappropriate improvisation within this context has the potential to result in catastrophic outcomes

Core components of activities undertaken by LOA leaders in this context, such as planning hiking activities in harsh terrain, operating in geographically isolated areas, communicating within a command structure in which higher level decision makers are geographically removed from the situation, could be seen to broadly resemble some generic components of activities undertaken by defence personnel in certain military contexts. While the activities themselves differ between the two contexts, as does the level of skill of those involved in the activities (LOA leaders being responsible for untrained participants), in both contexts leader are required to make decisions impacting the safety of those under their ‘command’. The findings of this stage of the research program therefore have some applicability to the military context.

Stage 3: survey

During the third stage of the research program, a survey was conducted to identify the relevance of the factors identified in the literature review to the LOA domain and to confirm that improvisation was a systems phenomenon within the domain.

Participants and recruitment

Participants were 157 employees of the Australian LOA industry. All participants were over 18 year of age and currently employed in LOA either as instructors, supervisors or managers. Participation was voluntary and the survey study was approved by the Monash University Human Research Ethics Committee.

Ninety hardcopies of the survey were distributed to Victorian Outdoor Activity organisations who had agreed to participate in the research and 35 were returned (response rate: 39%). In order to increase participation, a link to an online version of the survey hosted by Survey Monkey™, was distributed through the two Australian outdoor activity providers’ networks. In total, 122 people completed usable surveys online. Data from the 122 online participants was combined with that of the 35 hardcopy participants.

Survey tool and analysis

The survey consisted of 37 questions. Questions 1 to 13 collected demographic, employment and qualifications data, questions 14 to 21 collected information on the occurrence and nature of safety related improvisation in led outdoor activities, and questions 23 onward focused on the specific factors identified in the literature review as potentially influencing improvisation in safety critical situations [1]. Participants’ responses to the factor specific questions are discussed here. Analyses of the complete set of survey questions will be included in a future publication [22].

The survey was estimated to take about 30 minutes to complete. Prior to distribution, the survey was reviewed and piloted by six Human Factors experts to ensure the questions were unambiguous and easy to complete.

Participants’ responses were collated in Microsoft Excel and summarised using descriptive statistics where appropriate. Responses to open ended questions were imported into NVivo 9™, a qualitative data analysis software package, and coded for themes. One human factors analyst performed the initial coding and the identified themes were then reviewed by a second analyst.

Results

In questions 23 to 37 participants were asked to indicate whether they believed that, for each factor identified in the literature as potentially influencing improvisation in safety critical situations, this factor supported their ability to improvise at their current employer. Responses of ‘Strongly agree’, ‘Agree’, ‘Disagree’ or ‘Strongly disagree’ were counted as an indication that this factor influenced a participant’s improvisation (as opposed to responses of ‘Neither agree nor disagree’, ‘I don’t know’ or ‘This factor does not influence improvisation’). Where appropriate participants were then asked to complete open-ended questions for each factor outlining what it was about this factor that was particularly supportive (for ‘Strongly agree’ and ‘Agree’ responses) or unsupportive (for ‘Strongly disagree’ and ‘Disagree’ responses) of their improvisation within their organisations.

Table 1 presents the proportion of respondents who felt that the factors influenced their ability to improvisation either positively or negatively. In general, a greater proportion of participants felt individual level factors influenced their ability to improvise, particularly situation awareness (96%) and experience (95%). The exception to this was the factor systemic knowledge which had the third lowest percentage overall. A lower proportion of participants felt organisational level factors influenced their ability to improvise. In particular, organisational memory and information flows within the organisation were the factors that the lowest proportion of participants felt influenced their ability to improvise; however, even these factors were seen as influencing factors by at least half of participants (51% and 66% respectively).

Participants were also given the opportunity to identify any other factors that they felt influenced their ability to improvise. A qualitative analysis of their responses revealed nine additional factors. These factors are listed in the far right column of Table 1 and include resource availability and mobilisation at the organisational level, group dynamics at the team level, and confidence and positivity at the individual level. A factor beyond that organisational level—External standards and consideration—was also identified. This consisted of references by participants to specific pieces of federal, state or local government legislation or policy, or restrictions imposed by statutory authorities such as Parks Victoria, who are responsible for managing national, state and marine parks in Victoria.

Discussion

The aim of this paper is to describe the findings to date from a program of research currently being undertaken to identify how appropriate, effective improvisation can be supported in complex sociotechnical systems. The findings derived from the three stages of research described in this paper confirmed that improvisation is a systems phenomenon and that the factors which influence improvisation can reside at all levels of complex sociotechnical systems. As yet no systems-based model of improvisation exists, however, the case studies presented here demonstrated the suitability of Rasmussen’s Risk Management Framework and the accompanying Accimap methodology for a systems-based examination of improvisation.

The results of the first three stages of the research program go some way toward the population of an improvisation model based on Rasmussen’s framework [14]. Figure 4 maps the factors identified in the literature review and survey on to the framework in a prototype model. Although this prototype model is based on data from the LOA domain, factors in the literature were drawn from a variety of contexts and it is anticipated that the model will generalise to other sociotechnical systems operating in other highly dynamic contexts including in military conflict zones.

In a military context the ‘Government’ level of the framework would include the Ministry/Department of Defence headed by the Defence Minister or Secretary of Defence, the Prime Minister or President of a country, and other government Ministries. The ‘Regulatory bodies and associations’ level would include international organisations concerned with international peace and security such as the United Nations, and intergovernmental military alliances such as the North Atlantic Treaty Organisation (NATO) and the Australia, New Zealand, United States Security Treaty (ANZUS). The ‘Company’ level represents particular elements of a country’s Defence Force such as the Air force, Army or Navy. The ‘Management level’ represents the upper levels of the command hierarchy of a particular element of a Defence Force, for example in the Australian Army Chief of Defence Force, Chief of Army, Vice Chief of the Defence Force, Chief of Joint Operations, Brigadier and Colonel ranks, and the management structure itself. The ‘Staff’ level reflects all defence personnel but particularly those involved in the improvisation, so those in the field during operational missions. Finally, the ‘Work’ level represents the operational missions themselves.

There is some evidence of the relevance of the model to military operations in the similarity between some of the factors identified here and cognitive requirements identified by Militello and colleagues [23] as key for pararescue jumpers (responsible for the rescue of personnel and equipment from hostile or denied areas). The factors ‘Organisational Structure’ and ‘Role System’ are reflected in Militello’s ‘Fluid Command’ requirement; ‘Team work quality’ is reflected in the ‘Team Coordination’ requirement’; and ‘Situation Awareness’ is reflected in the ‘Big Picture Situation awareness’ requirement.

A systems model of improvisation will allow Defence Forces (and other organisations in complex sociotechnical systems) to understand the interactions between these factors both within and across different systems levels. An important question to ask in the military context is whether current military systems adequately support defence force personnel when they are required to improvise. This is important not only for the safety of personnel, but also for mission success where improvisation could be the difference between success and failure. Understanding the relationships between factors allows any interventions designed to enhance appropriate, effective improvisation and/or prevent inappropriate, ineffective improvisation to reflect these. This is in line with contemporary Human Factors approaches that emphasise treatment of system-wide factors as opposed to concentrating intervention efforts solely on factors at the ‘sharp’ end of the system (for example, [15,24,25]). Interventions focusing on the system as a whole are likely to prove more effective than simply making changes to lower level factors while the conditions that led to these sharp end factors remain unchanged within the system.

The model depicted in Figure 4 is a prototype and many top level factors and interactions between factors have yet to be identified; however, the model still provides some direction in terms of how organisations can support appropriate effective improvisation. For example, effective improvisation can be enhanced by having a flexible organisational structure that allows decision-making authority to be mitigated to those with the greatest expertise in the situation (rather than relying on hierarchy), and allows supervisors the discretion to provide subordinates with the freedom to develop and deploy novel solutions in the face of unexpected problems, including solutions that may violate standard operating procedures. This is supported by findings in the firefighting domain where it is referred to as ‘minimal structure’ [2], and in the military context it reflects Militello and colleagues’ ‘Fluid Command’ requirement, in which the teams adapt their leadership structure in response to the situation and expertise, and the pararescuer with the most experienced or best situation awareness takes the lead regardless of the initial configuration of the team [23].

This is not to say that a hierarchical system structure necessarily curtails improvisation. In fact clear lines of authority and role accountabilities were identified in the Apollo 13 Accimap as factors that positively influenced improvisation. In this incident each mission controller having a clear picture of who was responsible for what aspect of the mission, whom to defer to, whom to delegate to, and whom to speak to in order to gain particular information enhanced the effectiveness of the team and allowed information to be obtained and communicated rapidly. An important aspect in this incident however, was that although it took place within a highly hierarchical structure, permission to act outside this structure and of mission rules when necessary for the safety of the crew and the flight was legitimised in the Lead Flight Director’s job description [20]. It may prove more problematic to encourage effective improvisation in systems where scope for this kind of flexibility does not exist.

The model also indicates that the design of training and education programs interacts with factors in the staff and work levels of the system. Establishing a rapid and effective feedback system between employees and those designing the training programs ensures that the program can adapt to incorporate what knowledge and experience people found effective when improvising successfully, and that realistic examples of improvisation situations can be incorporated into training scenarios to give trainees practice at improvising. Ensuring that after action reviews examine and actions that have involved effective improvisation and have ended successfully may contribute effectively to this feedback loop in the military context.

Practice at improvising successfully is important not because it teaches employees how to respond in the particular situations used in the scenarios (improvisation by definition occurs in response to situations where there are no procedures or where the improviser has no experience of the specific situation), but because optimism that solutions do exist, and confidence and that you will be able to improvise a successful solution were identified by participants as important to their ability to improvise appropriately and effectively. This optimism and confidence can only be developed through opportunities to improvise successfully and these opportunities can be provided in low risk forms via training scenarios or simulations.

These interactions (training program design—feedback—optimism—confidence) are only a few of the complex web of interactions that influence improvisation in sociotechnical systems. A comprehensive systems-based model will include many more. A limitation of the literature review and survey methodologies employed in this paper is that while that have allowed identification of the identification of factors, they could not be used to determine the relationships and interactions between factors necessary to populate a systems model of improvisation based Rasmussen’s framework. While the Accimap methodology employed in the case studies was able to identify interactions between factors, the small number of cases (two) from different domains did not allow the identification of interactions that could be included in a generalised model. Stage four of this research program will examine a more comprehensive improvisation incident dataset obtained from the LOA domain using Critical decision method interviews [26] in order to obtain the data of sufficient depth and quality to allow for the identification of systems level factors and interactions. These factors and interactions will then be used to population a model based on Rasumassen’s framework.

This survey described in this paper is also limited by its sample size (157). This is reasonably small as a representation of the Australian LOA industry; however, given the difficulties in accessing respondents due to the remote locations in which many operate, a total of over 100 responses was considered a substantial achievement. The self-selective nature of survey is also a limitation as there is the potential that only those who felt particularly strongly about improvisation will have responded and that this sample may not be representative of employees of the Australian LOA industry as a whole. As this study was exploratory and aimed to identify factors rather than determine conclusively any ranking or order of factors, this was not felt to be a concern. It was felt that it was more important to use a method that would allow data collection from the largest number of respondents, including those in remote areas.

Conclusion

Improvisation has the potential to enhance the ability of complex sociotechnical systems’ to maintain safety and operational effectiveness when faced with situations for which no procedures exist, or more commonly, where circumstances prevent known procedures from being enacted as specified. Such situations commonly arise in fast-paced, high-risk, uncertain, dynamic environments such as hostile military areas. The concept of improvisation, however, is not currently well understood [1]. The analysis presented here establishes that improvisation is indeed a systems phenomenon and that there are multiple interacting influencing factors that reside at all levels of complex sociotechnical systems. Therefore, a systems-based model of improvisation is imperative if organisations are to take advantage of the potential safety and performance benefits afforded by appropriate, effective improvisation while simultaneously preventing inappropriate, ineffective improvisation. This analysis indicates that a model, based on Rasmussen’s framework and accompanying Accimap methodology will allow the conditions across the system that influence improvisation to be identified. Understanding these factors will increase the likelihood that systems reforms generated to support appropriate, effective improvisation will succeed, which in turn will enhance systems’ ability to maintain operational effectiveness in safety critical situations.

Acknowledgements

Ms. Trotter’s contribution to this research is supported by an Australian Postgraduate Award Postgraduate Research Scholarship. Dr. Salmon’s contribution to this research was funded through the Australian National Health and Medical Research Council’s Post doctoral training fellowship scheme.

References

Authors

Margaret J. Trotter is a PhD candidate and research assistant within the Human Factors team at the Accident Research Centre at the Monash University Injury Research Institute in Melbourne, Australia. She holds an honours degree in Psychology from the University of Otago, New Zealand. Margaret’s research focuses include organisational resilience and culture, accident analysis and modelling, and transportation human factors. Corresponding author. Email: margaret.trotter@monash.edu Phone: +61 3 9905 4142

Paul M. Salmon is an Associate Professor in Human Factors and leader of the USCAR (University of the Sunshine Coast Accident Research) team at the University of the Sunshine Coast. Paul holds an Australian National Health and Medical Research Council (NHMRC) post doctoral training fellowship in the area of Public Health and has over 12 years experience in applied Human Factors research in a number of domains, including the military, aviation, and road and rail transport.

Michael G. Lenné was awarded a PhD in experimental psychology/human factors in 1998 from Monash University. He is currently an Associate Professor at the Monash University Injury Institute where he leads the Human Factors team at the Institute’s Accident Research Centre. His research centres on the application of psychological and systems approaches primarily to transport injury but also to workplace settings.