Volume 7, Number 1, March 2004
An Immersive Approach to Command and Control
- 1 Iowa State University, Virtual Reality Applications Center, 2274 Howe Hall, Room 1620, Ames, IA 50011-2274, USA.
Abstract
More than ever before, success in battle depends on effective command and control—but the increasing complexity and speed of modern engagements makes it ever more difficult to develop the comprehensive situational awareness upon which effective command and control depends. In the face of this increase in pace and complexity, developing systems to expose cost-effectively battle managers and weapons directors to the full range and scope of potential conflict situations is an ever-increasing challenge. This paper presents a distributed immersive command and control visualization system. Networked participants visualize and interact with the Virtual Battlespace based on a JSAF simulation from the first-person, tactical or strategic viewpoints using one of several different immersive devices. This paper also describes how the Virtual Battlespace visualization system can be extended for use in a variety of control tasks such as battle manager training, real-time command and control, and UAV swarm management. Battle managers can interact with both simulated and “manned” entities using a mixed mode interface that includes wireless palmtop interaction. The system has been developed under the guidance of the Iowa National Guard’s 133rd Air Control Squadron, which has also cooperated in the evaluation of the system.
Introduction
The exhaustive review of prior campaigns, engagements and plans is a staple of military command training. Consider the staff ride, pioneered by Maj. Eben Swift at the turn of the last century [1]. After extensive study of the battle’s history and context, instructors and students would physically ride out to a battlefield site to examine the terrain of the field first hand, taking the vantage points of friend and foe, to see for themselves the interplay between ground, objectives and available force that constrain military strategy.
Staff rides and related activities, such as tactical exercises without troops, are time-honoured military training aids. Instructors set exercises and then students present their solutions for comment and discussion by staff and other students. These techniques help to teach the vital connection between battlefield conditions and tactics.
Modern engagements are no less dependent on a thorough knowledge of the field. However, unlike battles of the Civil War era, where the majority of a battlefield could be envisioned from the highest hill in the county, today’s battles are fought over thousands of square miles. The battle landscape is now defined not only by natural features, such as mountains and rivers, but by “invisible” features such as friendly and enemy sensors, the threat zones of long-range weapons, and the forest of targets that must be struck precisely to minimize loss of life. Creating consistent and complete mental pictures of this complex environment is one of the tasks of training, whether as part of pre- and post-mission briefing, or as integral part of command and control of distributed mission training exercises.
The complexity of modern warfare increases as the number of battle assets grows. With this escalating complexity, commanders are handed an increasingly difficult task of maintaining a clear mental picture of the engagement. This fact heralds the need for an improved method of command and control. Many of the same issues faced by modern training tools run parallel to those now faced by command and control in the field. These issues include the visualization of the visible and invisible features of the battle landscape, as well as the coordination of manned and unmanned resources. An example of this complexity can be found in the use of unmanned aerial vehicles (UAV). While the introduction of the UAV provided the armed forces with a powerful new tool, it quickly became apparent that it required a more effective human interface. The desire for one person to manage a swarm of semi-autonomous UAVs demands a new UAV control paradigm. A primary challenge with current UAV control stations is that it is difficult for one person to maintain situational awareness of both the UAVs and manned craft—once again, a visualization problem related to battle resources and their interactions.
We believe that immersive virtual reality (VR) technologies based on recent work at Iowa State University’s Virtual Reality Application’s Center (VRAC) can be extremely valuable in all three command and control contexts. Such technologies can allow battle managers and war fighters to traverse and analyse the complex information landscape that is the modern battlefield as it unfolds; they can allow trainers to develop strategies and tactics prior to an exercise or training engagement; and they can provide the basis of the control station for UAV swarm management.
Immersive battlespace visualization can fuse information about tracks, targets, sensors and threats into a comprehensive picture that can be interpreted more readily than other forms of data presentation. It is this quality that makes immersive battlespace visualization ideal for these command and control contexts.
Working with the Air Force Research Lab’s Human Effectiveness Directorate and the Iowa National Guard’s 133rd Air Control Squadron, a research team at VRAC has developed an immersive VR system for distributed mission training called the Virtual Battlespace. Virtual Battlespace was demonstrated at I/ITSEC 2002 as a part of the AFRL booth. It was also exhibited at the 133rd Air Control Squadron at the unveiling of their BCC-X system [2]. It was well received at both events.
Virtual Battlespace is a prototype system intended to explore the advantages of immersion in command and control. While it is not as fully featured and proven as current training systems, such as PM CATT [3], it can provide a platform for the validation of future directions for such training simulators. Where systems such as PM CATT have classified information on exactly how military equipment operates, Virtual Battlespace used an unclassified simulation to approximate the complexity of the problem and to provide an experience that can be used to evaluate the effectiveness of the immersive technologies.
Virtual Battlespace is evolving into a useful exercise planning, pre-briefing, debriefing, and live engagement management tool. This paper describes the basic design and implementation of Virtual Battlespace, some of its applications to-date, and the future directions it could be taken.
System description
Virtual Battlespace uses virtual reality immersion display technology along with the fusion of multiple data streams to provide a user with a clear representation of the information needed to understand and control a battle. The Virtual Battlespace system connects users to information streams using a display system and a role-based user interface.
The Virtual Battlespace visualization system is flexible, to allow it to support multiple end users. Common to most of these end users however is the need to see the entire battlefield or scenario. The goal for Virtual Battlespace is to provide a comprehensive view of the overall field that can provide additional detail as a user narrows their visual focus to a portion of the space. The Virtual Battlespace architecture can also accommodate system nodes that generate data streams associated with individual units, such as pilots in a flight simulator. Virtual Battlespace incorporates these users into a common system, allowing them to interact with one another in a distributed way.
There are many different streams of information that provide support for battlefield decision making. Some of these include radar and other sensor feeds, satellite imagery, communication links, and weapons information. Virtual Battlespace is designed to fuse multiple information streams and make them centrally available to command and control personnel. The goal of this comprehensive presentation is to improve a user’s ability to make effective and intelligent decisions [4,5].
A general architecture of the system is shown in Figure 1. In Virtual Battlespace, data streams are separated into two main categories: entity-based data and battle-level information. Entity-based information streams deal with the location, attitude, path, weapons, and sensors for a particular weapons system or entity in the battlespace. This information is needed to give the commander an indication of the assets and threats that are present and to paint a global picture of the overall field. Battle-level streams include: satellite imagery, video feeds of sectors and munitions, and communication networks among units. In Virtual Battlespace, these streams are presented graphically to reduce the amount of textual information presented to the commander allowing them to focus more time on critical decisions. Several research groups have explored using virtual “sand tables” [6–8] to display and interact with such data using large screens projected from beneath.
The information streams are made available to the user of Virtual Battlespace through the immersive display system. To make the Virtual Battlespace useful in the widest possible context, the display system is designed to support the complete range of delivery platforms, from permanent, high-
end multi-walled immersive projection theatres to lower-cost, deployable systems. With such a design, units with deployable systems in or near the field could be connected with a permanent installation at a central command centre to provide a common operating picture.
The user controls information display in Virtual Battlespace with a distributed, cooperative user interface. To avoid information overload and allow the user to tailor information display to meet individual needs, Virtual Battlespace allows users to interact easily with the system and focus solely on the information that they need. By decoupling Virtual Battlespace’s user interface from the underlying application, individual users can simultaneously interact with a common application through interfaces specifically tailored to their roles. Using these decoupled interface tools, users can choose the scale and presentation level of information on a common display to highlight particular aspects of the overall engagement. In this way, Virtual Battlespace facilitates not only a user’s ability to view and understand the battle but also provides a means to control it.
System architecture
This section discusses some of the design goals and decisions made in developing Virtual Battlespace. Figure 2 presents a subsystem-level diagram of the system architecture showing the relationships between its major components. In this diagram, the flow of data is from bottom to top. Data streams originate either from a simulator or a mission participant, flow through the data stream managers to the proxies, and are then displayed to the user.
An individual data stream is a connection to a data source that produces a time-series of data packets. This time series is processed by a stream manager to create time-stamped entries in the proxy database. The data proxies encapsulate common interfaces for data types that are displayable within the Virtual Battlespace system. New streams of data are incorporated by specializing one of Virtual Battlespace’s defined data proxy interfaces, allowing for stream specific manipulation of entity or battle level data while facilitating its display within the common interaction environment. The proxy interface provides the rest of the system with a common set of object interfaces that insulate the system at large from specific data stream encodings. This approach allows the system to incorporate disparately defined data streams more easily.
Information stream management
Central to Virtual Battlespace is the ability to fuse diverse data streams into an integrated display. This requires a system that allows incorporation of undefined data formats, while simultaneously creating a set of information display tools that can be used to display information from a variety of sources in a common way. Virtual Battlespace could easily be made HLA-compliant through the addition of a component that would subscribe to a HLA-based federation [9]. However, Virtual Battlespace had as a further design goal that the addition of non-HLA streams is easy and straightforward. This goal is achieved through the implementation of an application-level stream manager responsible for integrating multiple data streams and for providing a common set of internal interfaces for data interaction. This critical component, the Multistream manager, manages the process of conversion of raw stream data into stream object data.
A stream of data can be generated by several diverse sources such as a simulated force generator like Joint Semi-Autonomous Forces (JSAF), or a live sensor such as a radar feed, or a multimedia signal such as audio or video. The streams need not have a common format. The Multistream manager is responsible for fusing these disparate, dynamic streams into a coordinated set of data objects, which can be interfaced in a common way by the rest of the application. In the current implementation of the system, a video stream, a force stream, using the Distributed Interactive Simulation (DIS) communication protocol [9], and a proprietary vehicle simulation stream (VehSim) [10] are fused by the Multistream manager into a coordinated data structure.
The VehSim stream is the output of a human-in-the-loop vehicle simulation containing a time series of vehicle data including position, acceleration, and orientation. The simulator takes the inputs from the human and uses a dynamics engine to generate time-stamped vehicle data. This data is then sent via TCP/IP as the VehSim stream. The VehSim protocol supports a small number of simultaneous vehicles updated at a high frequency. The opposite of this stream in behaviour is the force stream. This stream sends DIS packets across a UDP connection and is capable of handling a large number of individual entities, each updated at a low frequency. In Virtual Battlespace, the DIS stream is generated using a JSAF scenario builder and is used to generate the bulk of the battle participants. The final stream implemented is a simple video feed. The Multistream manager allows a video stream to be integrated into the overall time stream, coordinating when and for how long each frame is played, and where it is to appear. The video stream can be either a live video feed, or a series of stored clips.
Proxy database
The graphical elements used to display the data streams are a major component of the system. They not only portray the physical attributes of entities in Virtual Battlespace, such as relative position, orientation, status and speed, they also portray derived attributes such as prior and future paths or sensor and threat ranges. To maintain the system’s flexibility with respect to the format of the input streams, the display of the data streams are separated from the management of the streams and from the base application.
Entity proxies provide the application with a uniform interface to individual entities, independent of the data stream the proxies were generated and updated from. This means that a proxy generated from a flight simulator stream can be displayed with the same graphical components as an entity generated from a DIS stream. This approach simplifies the interface not only between the application and the proxies, but also between the user and the entities. The user has no direct knowledge of the number of different information streams that are driving the system. All graphical functionality is expressed in terms of a common interface which all entity objects support. This allows entities represented by disparate data streams to be treated uniformly by the remainder of the application.
An example of this approach is in the implementation of the VehSim Proxy and DIS Proxy. The VehSim and DIS streams represent similar information, but at widely differing update rates, referencing distinct coordinate systems. The proxy implementations for each stream encapsulate the transformation of this information into a common representation and common coordinate system. The base implementation of proxy provides methods to support graphical entity display based on the rest of the proxy interface. However, derived proxies can override these basic definitions to define type specific behaviours if need be.
Another important aspect of the proxy database is that it supports the display of aggregate representations of groups of entities. These aggregate objects suppress the individual entity representations to reduce information overload. This allows, for example, flights of aircraft to be displayed as composite entities to simplify a commander’s view of a battle. The recursive nature of the proxy model allows aggregation at arbitrary levels by supporting aggregates of aggregates.
User interaction
The typical approach to user interface in immersive applications is a combination of gestural or positional interaction, combined with graphical display cues such as three dimensional menus and selection rays [11]. These interfaces support illusion of immersion by allowing users to interact directly with virtual objects. However, as the complexity of the application increases, the virtual metaphor must be augmented.
For Virtual Battlespace to be effective, users must be able to interact with the simulation to accomplish a wide variety of tasks such as navigation, view scale, aggregation, and selective information display. While some of these tasks are compatible with the usual immersive interface methods, many others are not. The Virtual Battlespace user needs a wide variety of interaction mechanisms that are intuitive yet provide access to a large number of configurations options. Furthermore, while much of the useful information in a battlespace can be conveyed graphically or iconically, sometimes there is simply no substitute for text. In these cases, immersive displays are handicapped because their display resolution is typically not sufficient to display graphics and text simultaneously.
The Virtual Battlespace system uses a combination of two modes of user input. In addition to the gestural navigation and graphical selection interfaces typical of immersive environments, Virtual Battlespace allows participants to interact wirelessly with the simulation via personal interface devices (PDAs, tablet computers, or other Java-capable devices). This is accomplished via an extension to VRJuggler (see below) known as Tweek. Based on CORBA as a remote procedure call mechanism, Tweek allows Java interfaces running on personal interface devices to communicate with the Virtual Battlespace. The Virtual Battlespace registers an interface that allows two-way communication between these devices and the application. Using this interface Java applications can give remote commands to drive the Virtual Battlespace application or issue queries to obtain status information. Because the interface is decoupled from the application, it is straightforward to provide custom simultaneous interfaces for multiple participants. Figure 3 shows a picture of a Virtual Battlespace’s Java interface implemented on a tablet PC via Tweek.
With this interface, users can navigate through space, select entities via the interactive radar screen, and perform actions on those entities such as toggling graphical features. Some of these graphical features include height sticks, sensor sweeps, threat zones and heads up displays. This interface not only provides the user the ability to interact with the application but it also provides information to the commander about the Virtual Battlespace. The Java-based interface complements the typical immersive interface well. The display devices are portable and non-intrusive yet provide crisp display of detailed information. The Java-based interface can support much greater complexity and yet remain very intuitive to a user because it uses familiar paradigms displayed on a device familiar to the user.
System implementation
Virtual Battlespace is a VRJuggler application [12]. VRJuggler is a platform for the development of virtual reality applications that provides developers with the ability to use a single source code base to support a broad range of VR devices, from desktops and head-mounted displays to Powerwalls and Caves. VRJuggler abstracts I/O devise to allow the applications developer to focus the application and not the VR device configuration. VRJuggler is offered under an open source license.
Since it is built on VRJuggler, the Virtual Battlespace supports all of the immersive display devices found at the Virtual Reality Applications Center (VRAC) at Iowa State University (http://www.vrac.iastate.edu). In addition to desktop and head-mounted displays, VRAC has several large scale immersive environments that have been used as test beds for the Virtual Battlespace.
VRAC most immersive device is the C6 (Figure 4), a 10’×10’×10’ room on which stereo images can be projected on all four walls, and the floor and ceiling. The result is a totally immersive 360° field of view. The C6 is driven by a SGI InfiniteReality2 system and achieves a frame rate of approximately 40 Hz. Users inside of the C6 are tracked by a wireless Ascension Flock-of-Birds tracking system. The wireless tracking system leaves the user free to move about untethered.
In addition to the image generation resources required by the Virtual Battlespace are the networked computing resources that generate the various streams of incoming data. For example, VehSim streams representing individual ground vehicles and aircraft are generated by Windows-based vehicle dynamics engines, while the JSAF forces may be simulated on either Linux or Irix resources. The Virtual Battlespace supports a wide range of input devices including, for example:
- a Microsoft Sidewinder Steering wheel and pedals for ground vehicles,
- a Microsoft Joystick for air vehicles,
- a variety of physical bucks for ground or air vehicles, and
- several wireless-enabled personal interface devices (PDAs and Tablet PCs).
Features
Consider a scenario involving an engagement between Red team and Blue team. Blue team is tasked with destroying Red team’s headquarters located in Nellis Air Force range in Nevada. Two SAM sites and five squadrons of fighter aircraft defend the Red team headquarters. Blue force consists of seven groups of aircraft. When the engagement is viewed strategically, these groups of aircraft are shown as aggregate entities and are scaled greatly to be visible from a long distance. The aircraft aggregates appear as symbolic entities but are placed in the space at the correct position height.
When the application starts, the user is presented with a view that encompasses the entire engagement. In addition to the terrain and the units engaged, the user is also presented with an information “billboard”—so called because it appears across the top of the display no matter where the user navigates (see Figure 5).
The billboard allows for the presentation of multiple simultaneous information channels. These may include symbolic views of the battlespace, such as synthetic “radar” screens, maps indicating additional features of the battlespace not contained in the main terrain display, orientation aides, and graphical keys.
The individual entities use a variety of graphical methods to display information about their status. For example, in addition to position, orientation, and velocity, entities in the space can also leave a coloured trail indicating where they have been or where they may be targeted to go. Through the decoupled Java-based interface, one or more users control the configuration of these additional display mechanisms. Using this interface, the user is able to navigate through the battle and focus on areas of interest. The interface can also be used to select entities by position, call sign, or type, and reconfigured to display additional attributes. For example, as shown in Figure 5, the Blue team lead sensor sweep reveals which Red team units are within the range of the Blue team’s “vision”.
Virtual Battlespace incorporates a variety of points of view to allow users to gain useful perspectives on simulated engagements. Figure 5 depicts the battle from a long-range (or strategic) point of view. Units are displayed symbolically at a size consistent with the unit’s importance, rather than its physical distance.
Figure 6 shows an alternative view that combines a realistic first-person entity perspective with symbolic, but physically accurate representations of ranges of threat. This allows a user to adopt a tactical perspective combining the participant’s first-person view with battle-level sensor information or other abstractions.
Future work
We plan to continue development of Virtual Battlespace along several lines: to increase deployability, broaden its applicability, and enhance display quality.
While the technologies underlying Virtual Battlespace have been carefully chosen to facilitate the application’s portability, the computational complexity of the display and the time demands of the system make deploying the original system on commodity level hardware a challenge. To address this challenge, we developed a Linux-based implementation of the system that uses a cluster of commodity PC’s as image generators and simulation engines. This work is based on recent extensions to the VR Juggler platform that simplifies some of the complexities of synchronizing multiple image generators with simultaneous, time critical input sources [14]. This new Linux cluster implementation will need to be evaluated by potential users. To provide this valuable feedback, we will enlist the aid of the 133rd ACS since they have familiarity with our first implementation. Additionally, we look forward to integrating a more deployable version with their MCS control module to allow immersive visualization of a combination of live and simulated sensor feeds.
To broaden the system’s applicability, we will incorporate a wider array of data input streams, including additional sensor streams as well as more sophisticated voice and video streams. Part of this work will be devoted to integrating these streams with the Multi-stream manager, but the larger effort will be the extension of the display and user interface systems to more effectively integrate these information streams into the overall system. A key consideration is to add to the user experience without cluttering the display and overwhelming the user. Among the data streams we are considering are additional sensor streams and weather information such as temperature and wind conditions.
We also plan to continue to enhance the visual quality of the display, by experimenting with new ways to represent units, terrain, sensors and threats at various levels of detail. We are continuing to enhance the graphics to add realism to first-person views and explore other visual enhancements that improve upon the immersive character of the application.
We also plan to extend Virtual Battlespace to address the management of a UAV swarm. This work will integrate a virtual reality aided vehicle teleoperation system developed in two other VRAC projects [15,16]. The results of these theses projects strongly suggest that the difficulty of vehicle teleoperation can be reduced through the use of a VR visualization tool in combination with a high fidelity vehicle simulation. Virtual Battlespace offers an ideal platform to integrate this combination. It already provides an immersive visualization tool and the system architecture is amenable to including different types of vehicle simulations. When the teleoperation system is integrated into the Virtual Battlespace, the operator will be able to see all the UAVs, their positions and situations (as well as that of all entities in the engagement), and be able to drill down into a first-person view of an individual UAV, and if needed, briefly take control of it. We feel that this type of control station, with variable levels of detail, scope and control, will be essential for the management of multiple semi-autonomous unmanned vehicles.
We are also interested in experimenting with Virtual Battlespace as an interface for real-time command and control of live units. The vision is to empower a user to communicate with any entity on the battlefield and manage the engagement, whether it is in a computer generated training scenario, or an actual battlefield with semi-autonomous aircraft and human controlled units.
Acknowledgements
This research is supported by a grant from the Air Force Research Laboratory, Rome, NY. The authors also wish to thank Mr. Terry Steadman, Dr. Rebecca Brooks, and Lt Col. Breitbach, and the men and women of the 133rd Air Control Squadron for their invaluable assistance and system evaluation. Finally, the authors gratefully acknowledge the contribution of the VRJuggler development team.
References
[1] W. Robertson, The Staff Ride, Prepared for the U.S. Army Center of Military History, Washington, D.C., CMH Pub 70-21, Supt. of Docs. no.: D114.2:R43, 1987.
[2] R. Breitbach, Tactical Ground C2 Transformation, Iowa National Guard Website, http://www.iowaguard.com/C2.htm.
[3] US Army PEO STRI Website, http://www.peostri.army.mil.
[4] D. Alberts, Information Age Transformation: Getting to a 21st Century Military, Washington, DC: CCRP Publication Series, June 2002.
[5] J. Posdamer, J. Dantone, N. Gershon, T. Hamburger and W. Page, “Battlespace Visualization: A Grand Challenge”, Proceedings of the IEEE Symposium on Information Visualization 2001 (INFOVIS’01), San Diego, CA, October 2001.
[6] J. Durbin, J. Swan II, B. Colbert, J. Crowe, R. King, T. King, C. Scannell, Z. Wartell and T. Welsh. “Battlefield Visualization on the Responsive Workbench”, Proceedings IEEE Visualization ‘98, IEEE Computer Society Press, pp. 463–466, 1998.
[7] D. Hix, J. Swan II, J. Gabbard, M. McGee, J. Durbin and T. King, “User-Centered Design and Evaluation of a Real-Time Battlefield Visualization Virtual Environment”, Proceedings IEEE Virtual Reality ’99, IEEE Computer Society Press, pp. 96–103, 1999.
[8] B. Brehmer and C. Sundin, “Command and Control in Network-Centric Warfare”, Proceedings of Conference on Network-Centric Warfare, London, June 23-24, 1999.
[9] G. Jense, N. Kuijpers and A. Dumay, “DIS and HLA: Connecting People, Simulations and Simulators in the Military, Space and Civil Domains”, Simulations and Simulators in the Military, Space and Civil Domains, 48th International Astronautical Congress, Turin, Italy, October 6-10, 1997.
[10] O. Balling, M. Knight, B. Walter and A. Sannier, Collaborative Driving Simulation, SAE Paper No. 2002-01-1222, February 2002.
[11] M. Mine, F. Brooks Jr., and C. Sequin, “Moving Objects in Space: Exploiting Proprioception in Virtual-Environment Interaction”, Proceedings of SIGGRAPH 97, Los Angeles, CA
[12] Homepage of VRJuggler.org, http://www.vrjuggler.org.
[13] Fakespace Systems, http://www.fakespacesystems.com
[14] E. Olson, Cluster Juggler—PC Cluster Virtual Reality, Masters Thesis, Iowa State University, 2002.
[15] B. Walter, Virtual Reality Aided Teleoperation, Thesis, Iowa State University, Ames, IA, August 2003.
[16] J. Knutzon, Tracking and Control Design for a Virtual Reality Teleoperation System, Thesis, Iowa State University, Ames, IA, 21 August, 2003.