Volume 3, Number 1, March 2000
Command And Automation - A Risky Dualism?
Abstract
Much attention is paid in the management literature as to why software-intensive command and control projects fail during the acquisition process. But many do enter service successfully only to fail during operational use, sometimes catastrophically. The author suggests that quite fundamental differences between the natural characteristics of command and the virtual rule-bound properties of software are a major cause of these catastrophic incidents. Some well-publicised examples are used to illustrate the extent of the breakdown between command and automation that can occur in real operations. Since in the future, it is likely that military operations will be more complex and unpredictable than those of the past, the author suggests that Military Users should be increasingly on guard to prevent technological hubris from constraining the operational flexibility of future systems. It is suggested that Military Users should take an active leadership role alongside Industry in the procurement process to ensure the ‘Risky Dualism’ between command and automation is addressed during system design, and not when it is too late.
Introduction
A surgeon, astronomer and programmer sat down for tea and a fierce argument arose over whose was the oldest profession. The programmer claimed that Ada Lovelace was the first programmer. Then the astronomer traced astronomical observation right back to the ancient Babylonians. The surgeon claimed his was the oldest profession since removing Adam’s rib to make Eve was surgery. Challenged like this, the astronomer claimed that her profession was the oldest, since before that, God had made the universe out of chaos. The programmer said ‘And who do you think created the chaos?
Of course computers and their software have brought many benefits to our society. Government departments, businesses, banks and offices could not operate without them. But not all projects have been successful. Some have entered service over-budget with serious performance shortfalls. British examples include the National Insurance Records System for paying social security benefits and the Home Office passport issuing system.
But there is another category of project that, although entering service successfully, then fails during operational use, sometimes catastrophically. It is on these ‘low probability of occurrence but high consequence of outcome’ incidents that this paper concentrates. Working from first principles and from a small number of examples, it is suggested that a breakdown in the operational relationship between the command and automation (dubbed the ‘risky dualism’) is a root cause of the problems.
But first a few explanations.
Automation and Control. Automation refers to automatic control where the algorithms and rules designed into the software define absolutely the performance of the automated part of the system. Clearly the performance of the automation will only be as good as the correctness of its requirements specification. Many ‘physical and causal’ systems requirements can be expressed in a formal mathematical language enabling software, when constructed with much care, to achieve predictably reliable control. But for ‘non-causal’ solutions such as those found in command support systems, requirements specifications are much more difficult to define. They may they be error-prone because some functions can only be expressed in natural and not formal language. They may also be deficient since it is not possible to capture all future intentions simply because they cannot be known in advance. Hence the software automation may perform with errors and omissions stemming from ambiguities and shortfalls in the initial requirements. It could also function incorrectly, even dangerously, when put in scenarios for which it was not designed. The major risks posed by delegating authority to software in command systems were highlighted when the dependency of the Star Wars missile defence system on software was analysed in the 80’s by Fetzer [1] and Parnas [2] et al. Today similar issues underpin the current round of strategic and theatre missile defence deliberations. In sum, by automation refers to control without human intervention. It is premised on deterministic certainty, predictive models and verifiable theory, clearly the world of computer programming and not of command.
Command. In contrast, command is about coping with uncertainty and not the certain world of software. It is a quintessentially human function and is analogue not digital. Humans function in contact with the world around them and, using their skill, experience, judgement and intuition, adapt to changing circumstances. The commander’s role expressed by Janowitz [3] of being part heroic leader, military manager, politician, public relations official, father figure and psychotherapist further distance the commander from the artificial rule-bound world of software and the computer. Clearly the mental activities which enable a commander to make decisions in the face of uncertainty are quite unlike the calculative reasoning performed by computer programs.
Yet, even though the characteristics of command and automation are as different as chalk from cheese, both concepts have essential contributions to make in real operations. They must operate together and support each other seamlessly if an increase in unexpected outcomes, typified by the following examples, is to be kept to a minimum in the future.
Some examples
The antares
The collision between a Navy submarine and the fishing boat Antares off the Clyde on 23 Nov 90 was widely reported in the Press [4]. In the collision the submarine snagged the trawl nets of the Antares, pulled it under and four fishermen lost their lives. In the fatal accident inquiry it was reported, inter alia, that the submarine had no clear picture of what was happening on the surface. The report said the chance to feed in data into the submarine’s computer when at periscope depth was not taken. In the subsequent court martial, the submarine commander is reported to have blamed the accident on a computer error. He said he trusted the computer when it told him he was at least three miles and 40 minutes away from a possible collision. Of significance was his reported statement to the Court that his young generation of submariners (he was 31 at the time) preferred to rely on the computer rather than on a manual plotting system [5]. Yet a senior submarine captain had previously told the Court that in such busy waters, a manual plotting system would have been a more trustworthy method to use. The young commander was convicted on three counts of negligence and given a severe reprimand.
Shooting-down of the iranian air bus
In 1988 the USS Vincennes equipped with the very latest AEGIS automated command and control systems was patrolling the Straits of Hormuz when it shot down a civilian Iranian A320 Air Bus with great loss of life. An analysis of the ship’s computer records following the accident showed that the technology worked as designed although the system as a whole had clearly not. What transpired was that a combination of preceding events had resulted in cumulative stress building up in the operations room team and a false mental model of the outside world being formed by the command which resulted in the fatal firing of the SM-2 surface to air missile.
In his analysis of the incident, Gene Rochlin [6] said the AEGIS command and control system had scenarios programmed into it for the air defence of a carrier battle group in a total war scenario. The system had not been designed for operation in the gunboat-infested waters of the Gulf in which there was considerable civilian air and sea movements. It was not therefore surprising that a crew so extensively trained and practiced on simulators which played out the games for which the system was originally designed, should mistakenly fall back into one of these pre-programmed behavioural patterns when put under severe operational stress and time pressure.
Missile attack warning systems
The North American Air Defence (NORAD) system has been a rich source of examples illustrating the potentially dangerous interactions which occurred between automation and command during the time the US was faced with the early Soviet ballistic missile threat. The problems were well illustrated starting in 1961 when alarms were triggered by the Ballistic Missile Early Warning System (BMEWS) in Greenland to say that the Soviets had fired missiles. In fact they had not. What had happened was that the radar data processing software had not been programmed to recognise that the rising moon had created radar glints. These in turn were interpreted by the automation as the launch of Soviet ballistic missiles. Needless to say the software was hastily modified to take account of these lunar movements which had not been included in the original BMEWS specifications [7].
Even 20 years later, worrying problems were still being reported with the complex NORAD system. On 3 June 80 the display system at Strategic Air Command (SAC) HQ indicated two submarine launched ballistic missiles were heading towards the US. But the displays at the NORAD HQ were clear. Then the displays at the National Command Centre in the Pentagon indicated the missiles heading towards the US. A threat assessment conference was quickly put on-line and B-52 manned bombers were got ready. However after three minutes of discussion, a number of different factors suggested no attack was actually underway. There had been no political warning, neither did the displays follow a logical pattern and command posts were receiving different information. Some three minutes and twelve seconds later, the alert was cancelled. Interestingly NORAD then decided to leave the system in the same configuration hoping the error would repeat itself. And it did three days later. The fault was traced in this case not to a software error but to a faulty integrated circuit chip in the general purpose Data General computer then widely used throughout the NORAD system.
The story of three generals
The final example is based on Norman Dixon’s fictitious story about the three generals [8]. With just slight alteration it runs like this. The Defence Department had put in place a major command and control system for a mobile battle HQ. The distributed system had been constructed to the highest standards of dependability; no expense had been spared. When deployed in a high state of tension and unrest overseas, the automation had handled, fused and successfully delivered via the telecommunications infrastructure a key message to the general which said ‘enemy prepared to attack’ and gave details of strength, disposition, time and sector. The message was a masterpiece of clarity, correctness and timeliness. Yet when received by the recipient, the outcome varied markedly from the first to the third general.
The first general had anticipated such an attack, so the message contained very little information. It merely confirmed a hypothesis he already held. In fact since he had already made extensive preparations for such a counter attack, the message was largely redundant. In the case of the second general, the message when received was quite unexpected. So little had he anticipated an attack, that the message was charged with information. It reduced his ignorance by a huge margin and gave him and his staff much to occupy their minds in the short time available to ward off the attack. For the third general, the message was so totally unexpected that he chose to ignore it, with disastrous results. It clashed with his beliefs and emanated so he thought from an unreliable source. Since his mind was closed to its reception, he found plenty of reasons for refusing to believe it.
Discussion
So what observations might be drawn from these four examples? Several thoughts are presented briefly in the following paragraphs – the reader will no doubt identify others.
Stress. Well it seems danger can arise from both stress and from routine. MacKenzie [9] indicates that under stress and pressed for time, the command cannot always be expected to enquire whether the situation they face is actually the one for which the automation was designed to meet. Indeed even if they enquired of the system, the information would probably not be there to answer the question nor if it was, in the time available. In the Vincennes incident the command acted out a scenario which was more compatible with the deep blue water threat the system was originally designed to meet rather than the low intensity but harassing scenario they were actually experiencing in the Straits of Hormuz. In effect, the automation took over control of the scenario thereby effectively reducing the command function to one of a slavish following of the rules built in to the AEGIS software by the original designers.
Trust. Nearly the converse of the above is that, after long hours of successful operation of automated technology, we should not be surprised when commanders and their staff begin to trust the automation too much. They can lose contact with the outside world and fail to check using other inputs whether changes are taking place in it. This is likely to occur during long and quiet periods of little apparent activity as probably occurred in the case of the naval submarine. Indeed if young commanders are trained to operate the system solely via the automation, one wonders if some could ever develop the skills and confidence to cope with the truly unexpected and be able to override the rules programmed into the technology should that ever be necessary. Even if it were technically possible (which might not always be the case), the nature of military culture might inhibit some of them from taking such an initiative for fear of being court-martialled should they get it wrong.
Time. The time available for consultation and command decision-making is clearly a key factor in determining the outcome. Fortunately the NORAD early warning system was designed to give the US High Command time to assess the veracity of the information they were receiving and for the various commanders to consult with each other to conclude the warnings were false. Conversely the build up of stress in the operations room of the USS Vincennes and the command team’s shared conviction of being under attack from an Iranian missile left little time for contemplative reflection of the situation. Similarly the second general was so overwhelmed with information that he had little time to plan a counterattack. Conversely the long time-frames and routine in which the submarine commander was operating, might have caused complacency to set in.
Situation Awareness. The submarine commander had effectively lost contact with the external situation in which his vessel was steaming. It is not clear from the reports whether or not his sonar operators had received contact information from the noise of the fishing boats engines. Nor did the opportunity seemed to have been taken to input data when the periscope was up. Indeed an experienced commander said that it might have been preferable to use the manual plotting system given the amount of surface traffic around at the time. Clearly by trusting his automation, the submarine commander formed a mental model of the outside world which was very different from the reality. So too with the Vincennes. The Tactical Operations Officer engendered a group-think mentality in the command team into believing that the target fast approaching the Vincennes was an Iranian F-14 armed with an air-to-surface missile, a belief which from the computer records turned out to be at total variance with the correct information which the AEGIS automation was actually presenting. Interestingly the command team on the picket ship USS Sides, a vessel equipped with much simpler command automation than the Vincennes, had already correctly identified the target as the Iranian AirBus 320 still in the ascent phase of its scheduled flight! In the case of Norman Dixon’s story, the mind sets of the three commanders were also very different which explains why, even though they received the very same message, three very different outcomes occurred. All in all, the examples show how the pre-existing mind sets (sometimes called the cognitive maps) of commanders and their team can influence not only their understanding of the real world around them but, even more importantly, can shape the consequential outcomes.
Looking ahead
Of course many other military and civil incidents have been reported. Lessons from the Gulf War are only now beginning to emerge and it is certain that new ones are being experienced in the Balkans theatre. In the commercial world there is the ongoing ‘glass cockpit’ debate as to whether computers or pilots should be in command of civil aircraft. Yet many of these incidents, although significant, are of a bounded and self-contained nature.
But a current trend is for previously separate systems to be meshed together into a tightly coupled network. This is very apparent in defence where future capability is becoming increasingly dependent on systems inter-working successfully with other systems, many of which will be designed, constructed and then subsequently modified by different contractors in different countries. And as systems become more interdependent, so overall system complexity will increase exponentially and the possible modes of unexpected behaviour will rise accordingly. Increasingly it will become almost impossible for people to understand the possible states and modes in which the overall network of systems can exist, particularly when faced with situational novelty and/or failure of one or more systems [10].
This is a significant issue because the military must expect to operate in unexpected situations in the future. So, not only is it likely that automation will be deployed in scenarios for which it was not designed (as with the USS Vincennes and some would say Patriot in the Gulf) but the command may also face new ‘adversaries’ who, whilst being technically unsophisticated, might in all probability be very intelligent and cunning, similar to the Viet Cong the US encountered in Vietnam. Future situations will therefore be full of ‘surprises’. And history shows that ‘surprise’ (in all its forms) is a tactic which can defeat an opposing military force whose doctrine is constrained by inflexible, fragile and dumb automation. So perhaps the designers of future battlefield automation should bear in mind the words of that famous Prussian General, Moltke, who said ‘the enemy always seems to have three alternatives open to him and he usually chooses the fourth’.
In his rather gloomy recent book ‘Trapped in the Net’, Rochlin [11] suggests that in extreme situations the mounting complexity of networked automation could straight-jacket the command into a control mode of operation in which the rules of the software determine the outcome rather than the Command. He says (with slight alteration from me) ‘The new image of the military of the 21st Century is towards a totally integrated, electronically linked and fully computerised fighting force where training with simulators and games is being taken as the equivalent to actual combat. Even the Army, the most traditional and conservative of the Armed Forces, is moving in this direction as fast as budgets will allow. But such tight control goes against the lesson that confusion and disorder the famous fog of war (and increasingly of peace - my words) will remain the natural state of combat, however advanced, sophisticated and precise the new systems might appear when tested in peacetime. Abandoning the lessons of independence for a more integrated doctrine that could prove tragically fragile in the face of an enemy capable of interfering with the elaborate command networks, runs the great risk of trading off overall effectiveness for the sake of technical efficiency.’
Conclusion
The above passage neatly sums up the ‘risky dualism of command and automation’. Clearly command and automation must be planned in the context of each other to ensure overall capability is not jeopardised as unexpected situations arise in the future. An overall balance is needed not only at the project level but at the overall network level as projects are increasingly meshed together. But the systems thinking required to ensure this goal is achieved must not be left solely to the technocrats. It should led by the military in both the requirements setting AND in the balancing of the human aspects of command with the appropriate level of automation during the design process.
It has been said by Dreyfus and Dreyfus [12]: ‘No unprecedented and potential reliance on automated systems should be adopted without the widespread and informed involvement of the people affected’. From this we should conclude that Military Users should take positive steps during the procurement process with Industry to ensure technological hubris does not jeopardise command capability and put at risk the successful operation of defence capability in the unexpected and high consequence situations that are surely set to increase in frequency in the digital battlespace of the future.
References
[1] J. Fetzer, “Program Verification: The Very Idea”, Communications of the ACM, Vol. 31, No. 9, pp. 1048-1063,Sep 1988.
[2] D. Parnas, “Software Aspects of Strategic Defense Systems”, American Scientist Nov 1985.
[3] M. Janowitz, The Professional Soldier, The Free Press, 1960.
[4] “Collision submarine made errors”, Report of fatal accident inquiry in the Independent newspaper, 23 Oct 1991.
[5] “Computer blamed for sea collision”, Report of evidence to court martial in the Independent newspaper, 5 Jun 1992.
[6] G. Rochlin, “Iran Air Flight 655 and the USS Vincennes”, in Social Responses to Large Scale Systems La Porte (ed), Kluwer Academic Publishers, Netherlands, pp. 99-125, 1991.
[7] A. Borning, “Computer System Reliability and Nuclear War”, Communications of the ACM, pp. 112-131, Vol. 30, No. 2, Feb 1987.
[8] N. Dixon, On The Psychology of Military Incompetence, Jonathan Cape, 1976.
[9] D. MacKenzie, “Computer-related accidental death: an empirical exploration”, Science and Public Policy, 1994.
[10] Grabowski and Roberts, “Human and Organisational Error in Large Scale Systems”, IEEE Systems, Man and Cybernetics Part A, Vol. 26, No. 1, pp. 2-16, Jan 1996.
[11] G. Rochlin, Trapped in the Net, Princeton University Press, 1997.
[12] H. Dreyfus and S. Dreyfus, Mind Over Machine, B.Blackwell, 1986.
This paper is based on a presentation given to the 4th European Electronic Battlespace (Land) Symposium, Cranfield University, RMCS Shrivenham 7-9 September, 1999.
The views expressed in this paper are those of the author and not those of his employer.