The Management of Distributed Databases in Military Command Support Systems

Introduction

Information systems used to store and process information relating to military operations are normally referred to as Command Support Systems, or CSSs. CSSs utilise database technology to store mission-critical information on locations, tasks and readiness states. The nature of modern military operations, with the emphasis on wide geographic dispersion, rapid movement and frequent organisational regrouping, creates significant problems in ensuring the availability and consistency (synchronisation) of databases. The fact that tactical-level communications systems are likely to have intermittent connectivity and limited bandwidth further complicates the problem.

The following sections review the management of distributed databases in the military CSS environment. The likely nature of military operations is outlined, as are the effects and constraints that these types of operations are likely to place on database management. The paper also describes how modern distributed technology is able to provide solutions for the management of distributed databases in tactical CSS. Relational database technology is mainly considered, as this is the prevalent technology used in CSSs at this time. However, the issues and techniques are applicable to any database technology.

The nature of modern military operations

Several characteristics of modern military operations will impact on the management of distributed databases:

Frequent and Rapid Changes in Task Organisations. Most modern conflicts tend to be of relatively short duration (months rather than years) and forces are assembled from a variety of Services (Navy, Army, Air, Marines) and a variety of different nations. Task organisations are created and changed frequently, often with little lead time. This puts a strain on CSSs, which are usually heterogeneous, and often not designed to interoperate with other systems.

Mobility. Modern military operations involve rapid mobility of forces, whether by sea, land or air. Coupled with the rapid reorganisation noted above, this creates a significant problem for the management of distributed databases – these must be structured in such a fashion and flexible enough to be able to be kept consistent in spite of rapid changes to system connectivity and layout.

Survivability/Availability. Databases in a CSS contain data that is critical to the conduct of military operations. There is a high probability of losing access to data due to normal communications or hardware failures, or due to enemy action; such unavailability is not acceptable. This points to a need for database survivability, which almost certainly includes some form of replication. This, in turn, raises issues of consistency: ensuring that multiple versions of the same data elements are synchronised.

Communications. The factors of mobility and frequent organisational regrouping also affect communications connectivity. Many high capacity military communications systems, while deployable, cannot be operated when actually moving. Other potential communications problems include:

• certain organisations may operate in conditions of electronic silence (that is, not permitted to emit electromagnetic radiation) for considerable periods of time; and
• communications bandwidth may be limited, especially to highly mobile elements such as reconnaissance and surveillance units.

Key requirements for distributed databases in command support systems

Consideration of the above factors results in the following key requirements impacting on the design and management of distributed databases in CSSs:

• Flexibility. The system must be able to accommodate a variety of national and single-service sub-systems and, probably, different database management systems (DBMS) (that is, the database environment will be a heterogeneous one), and frequent changes in organisational structure, geographic location and communications connectivity.
• Survivability/Availability. The system must be resilient enough to withstand loss of communications connectivity or of discrete nodes, either through normal hardware or software failure or physical destruction due to enemy action.
• Integrity. The system must be designed and managed in such a way as to ensure that data is consistent and current, or, at the very least, that potentially erroneous or out-of-date data is flagged as such to the user.
• Performance. The system must ensure retrieval of data in acceptable timeframes. This includes some real-time and near-real-time applications.

Distributed database management - general considerations

One option for managing disparate, geographically dispersed databases is to utilise a formal distributed database management system (DDBMS). Such an approach is well described by the following:

"Distributed database technology is concerned with making a number of distinct database systems appear like a single database system to a user. The need for this technology arises frequently, as a consequence of the need to distribute data geographically, or as a result of the need to integrate different software development efforts [that is, a heterogeneous DBMS environment]." [1]

Chris Date [2] defined a number of criteria which must be met for a system of geographically separated databases to be considered as "distributed". Date's fundamental rule (“Rule zero”) is that "To the user, a distributed system should look exactly like a non-distributed system" [2]. He then defined twelve subsidiary rules, including:

1. Local autonomy

2. No reliance on a central site …

4. Location independence …

7. Distributed query processing

8. Distributed transaction management …

11. Network independence

12. DBMS independence.

The general architecture of a DDBMS is shown schematically at Figure 1. [3]

In this architecture, Date's rules, particularly that of location independence, are satisfied by all users accessing both local and non-local data via a DDBMS and Global Data Dictionary (GDD). The GDD is, effectively, a "master index" of where all data elements are stored in the overall distributed system. The GDD, in itself, may also be distributed, and/or replicated to ensure availability.

The key feature of a DDBMS is the presence of some sort of automated, system-wide management system. Commercial databases (such as Ingres/Star and Oracle) provide such facilities for both homogeneous and heterogeneous environments.

The other main option for managing distributed databases is to use manual or semi-automated techniques. Databases at different sites are regarded as autonomous or semi-autonomous. Formatted messages, such as in EDI or military character- or bit-oriented message systems, would be used to exchange records which were required to be shared or replicated between sites. In such a system, there would be no GDD. This is the database management strategy for many existing military CSSs.

There are major difficulties in such an approach. Frequent loss of communications connectivity and organisational regrouping will make this difficult to manage. For example, if Unit 1 commences an operation under command of Headquarters A, there will almost certainly be a need for data on locations and tasks of Unit 1's sub-units to be replicated at Headquarters A. If at some later stage, command of Unit 1 passes to Headquarters B, the relevant data on Unit 1 and its subunits must be transferred to the database at Headquarters B. This would require manual intervention, and, if not carried out properly, would lead to incomplete and/or inconsistent data in the system.

There will also be significant data administration issues: who would be the “custodian” of a particular data element – for example, the location of Unit 1? Presumably Unit 1 would enter the location data initially (although this may be accomplished in automated form, via GPS). However, once Unit 1’s location is passed to other databases by means of formatted messages, Unit 1 effectively loses control of the content and accuracy of that data element. At other locations, it is possible to alter that data element (by design or accident) and Unit 1 would not be aware this has occurred. The data element would remain in error until Unit 1 sends its next update (possibly after a considerable time gap, due to communications constraints).

In most cases, the more formal, holistic approach of an automated DDBMS, with a GDD, is a far more reliable method for managing distributed databases in military CSSs.

Distributed table structure and replication

A fundamental principle of DDBMS design is to ensure that as high a proportion of operations (queries and updates) as possible is performed on local data. This is to minimise the amount of communications traffic and, consequently, reduce communications costs and improve performance.

In relational databases, data is stored in tables, which are related to one another according to the mathematical rules of relational calculus. Tables consist of rows (sometimes referred to as records, or tuples) and columns. The basic options for distribution of tables are as follows: centralised, partitioned (both vertically and/or horizontally), replicated and hybrid (a combination of two or more of the above) [3].

The centralised option means that a table exists complete, and in only one location. Partitioned tables are "split" between more than one location: horizontal partitioning means that different rows of the table are stored in different locations whereas vertical partitioning means that different columns of the table are stored in different locations.

Replication means that more than one version of a table, or elements of a table, is held in different locations, to ensure availability/survivability. However, it should be noted that replication carries a significant overhead in storage, management (to ensure consistency) and communications traffic.

In military CSSs, the most likely, and probably most appropriate, table distribution option is a mixture of horizontally partitioned and replicated; that is, hybrid. Replication of key data will be necessary to ensure survivability. Horizontal partitioning will be required to allow non-key data to be held locally, but in a standardised table structure to allow remote queries when required.

If replication is to be used, it is essential that this is properly managed with an automated Replication Manager, otherwise inconsistencies will result. For example, a local copy of data is updated, but the replicated copy(ies) is not, due to human error or because a communications link is not available. An example of a fully automated replication manager is found in the Oracle 7 DDBMS:

"Asynchronous replication allows multiple copies of data to be maintained at different sites in a distributed environment. It provides immediate, local access to data and allows systems to function autonomously even when other systems in the distributed environment or networks fail....

Changes to local data are stored locally and then propagated and applied to remote copies in a store-and-forward manner within separate transactions. If a remote system is unavailable, the changes remain in the local system and are propagated when the remote system becomes available." [4]

Options available for replication management include primary site, where one site is designated as the principal "owner" or custodian of the data, dynamic ownership, where the right to update replicated data moves from site to site but that only one site has the update rights at any given point in time, and shared ownership, where multiple sites are permitted to update data. (The latter alternative can give rise to update conflicts, which can lead to inconsistency between multiple copies of data. This is addressed in the next section.)

For the majority of military CSS applications, the primary site technique is probably the most appropriate, because it supports the concept of "ownership" of data. That is, the organisation to whom the data relates has sole responsibility for updating it (which, in turn, reduces the probability of erroneous data being entered) but that records are available to other organisations who need to share the data, on a read-only basis. However, for certain types of data (for example, enemy sightings, where more than one friendly unit may be in contact) dynamic ownership may be appropriate.

Another aspect which needs to be addressed in implementing a DDBMS architecture in military CSSs is that of table structure and data element definitions – that is, a common data dictionary. Distributed tables must have common structures and data elements must have common definitions to allow for distributed queries to be performed in any sort of meaningful way. A common data dictionary needs to be agreed on an inter-Service and international basis. The Combined Communications-Electronics Board (CCEB) commenced an action item around 1997 to produce an Allied data dictionary; however, there appears to have been little substantive progress.

Any data dictionary produced for Allied database interoperability must take into account, and be harmonised with, the various data element dictionaries associated with the formatted message systems in use.

Distributed transaction management

A transaction is regarded as a single logical unit of work in a database. It may be the running of an entire program, a portion of a program or the read or update of a single data element. Distributed transaction management is the management of transactions which are executed across more than one site. Concurrency control is "the ability to manage simultaneous processes without having them interfere with one another". Recovery is "the process of restoring the database to a correct state in the event of a failure." [3]. Both concurrency control and recovery will be critical to satisfy the survivability and consistency requirements of military CSSs. Another aspect of distributed transaction management considered in this section is that of commit protocols, which have a major impact on performance and consistency in a distributed environment.

Concurrency control

Although distributed databases should be designed so that the majority of operations will be performed on local data, there will be instances when multiple users will be attempting to concurrently execute operations (read and/or write) on data elements that are distributed across multiple tables at multiple sites. Also, the replication of data will involve multiple update transactions across more than one site. In this situation, concurrency control, to ensure consistency of data, is essential.

There are two basic options for concurrency control: the so-called pessimistic and optimistic techniques. [5] In outline, pessimistic techniques expect that conflicts will occur (for example, two users will attempt to perform a transaction which writes to a data element simultaneously) and apply measures to ensure serialisability; that is, each transaction is performed in sequence and in a consistent fashion. Optimistic techniques are based on the assumption that conflicts will not occur very often, and allow transactions to proceed as if there were no concurrency problems. Just before a transaction commits, a check is performed to determine whether a conflict has occurred and, if so, the transaction is rolled back to a point where it can be restarted and executed without conflict.

Pessimistic techniques include locking, in which a transaction "locks" data to write-access by any other transaction until the original transaction is completed. Transactions attempting to write to data elements that are locked are forced to wait until the lock is released. The level at which locks are applied (ie, whole database, table or record) is known as the granularity of the locks. The principal advantage of locking is that it is the safest technique in terms of ensuring consistency of data. However, it has significant management overheads and tends to degrade performance. There is also a potential problem of deadlocking, where two or more transactions are effectively stalled while waiting for the release of a lock held by the other. If this is likely to be a serious problem (in terms of performance), a Deadlock Detector system will need to be used.

Another pessimistic technique that eliminates the deadlocking problem is timestamping. In this technique, each transaction is allocated a timestamp, which is a unique identifier of the start time of the transaction. Transactions are then completed in the serial order of which they commence, with rules to govern situations where serialisability may be compromised. This technique is less subject to performance problems than locking, but requires close synchronisation of local clocks to ensure viability. In military CSSs, with frequent losses of communications connectivity, this could represent a major problem.

Optimistic techniques are based on the premise that conflicts will seldom occur, so no precautions against conflicts are taken until just before the commit, when a check is done. If a conflict is detected, the whole transaction is rolled back, and restarted. Such transactions normally involve read, validation and write phases. If transaction levels are such that conflicts do not often occur, then this technique will provide much better performance levels than any of the pessimistic techniques. However, in a system where there is likely to be a need to frequently roll back, performance will be adversely affected.

In military CSSs, consistency of data is essential, performance must be able to be guaranteed at times of both high and low activity, and strict synchronisation of distributed local clocks will not be possible. In these circumstances, locking is considered to be the most appropriate technique.

Recovery

Replication and recovery techniques are essential to ensure survivability and availability of data in military CSSs. Recovery techniques relate to partial failures at a local node in the context of a local query, or the total or partial loss of a whole node in the context of a distributed query. In each case, the recovery techniques are essentially similar, and depend on the orderly reconstruction and completion of a query by utilising non-volatile storage, or the cancellation of the query with a re-attempt at a later time.

Transactions are required to be atomic; that is, all portions of the transaction must succeed or the whole transaction must be aborted. This is particularly critical for a distributed transaction, where the failure of one element of a transaction could leave a local database in an inconsistent state. Failure recovery is normally controlled by an automated Recovery Manager.

There are two main techniques used to manage recovery: logging and shadow versions [6]. In the logging technique, a recovery file is kept in non-volatile storage which records details of all transactions currently in progress (ie, that have not yet been committed). If some element of the system fails, the log is used to reconstruct the transaction(s) which are not complete, or if not possible to complete, are aborted. The shadow version technique uses a second "provisional" version of data and transaction parameters in volatile storage, which is then copied to non-volatile storage when the transaction commits.

The shadow version method provides faster recovery than logging, and is therefore more suitable for the CSS environment where failures are commonplace, and the need for rapid recovery is critical.

Commit protocols

An important element for maintaining consistency and performance in distributed databases is the method used to commit the results of a transaction once the various steps have been performed. There are essentially two broad classes of commit protocol; single-phase and two-phase commit [6]. In a single-phase protocol, a node is assigned the status of transaction coordinator. On completion of the transaction, this node will rebroadcast either a commit or an abort command to all other involved nodes until an acknowledgment is received. In the two-phase protocol, each node has the ability to "vote"; that is, advise the transaction coordinator of its ability to carry out the operation requested of it. Once the transaction is ready to commit, the coordinator will request a final vote from the other nodes before it issues the command to commit (or abort).

The two-phase protocol is more suitable for CSSs, because of the frequency of which nodes will lose communications and the resulting repetitive rebroadcasting of commit and/or abort messages that would result. That is, interactive, rather than neutral, input from remote nodes is preferable.

Query optimisation

The standard data manipulation language (DML) for relational databases is SQL (Structured Query Language). SQL is used to both read and update (write) tables. The basic DML operations are select (select rows from a table according to given criteria), project (select columns from a table according to given criteria) and join (join tables, or elements from tables, to form a new logical table). In a DDBMS, these operations will frequently be performed on multiple tables in different locations.

SQL is a non-procedural language (that is, the order of operations cannot be specified). A query may therefore carry out a join operation before a select or a project, which may result in whole tables being transmitted across the network, when possibly only a few data elements were required to satisfy the query. This points to the need for some form of query optimisation manager to ensure query efficiency.

Bell and Grimson [7] analyse the importance of query optimisation and give examples where non-optimised queries result in quantities of data being transmitted across the network that are orders of magnitude larger than for similar queries that have been optimised. The basic order of operations should be select/project then join. However, this is situation-dependent, and the existence of other query operations such as semi-join and outer-join mean that each query needs to be analysed and optimised on a case-by-case basis.

Given the likely communications constraints in military CSSs, in terms of bandwidth and availability, query optimisation is essential.

Model-Based Battle Command

Chamberlain [8,9] has taken the debate over distributed database management in military CSSs to a new level, by suggesting a new paradigm in command and control systems – model-based battle command (MBBC). He argues that:

“In spite of the incredible advances in computing, the automation of battle command remains relatively antiquated because of the common pitfall of simply automating manual techniques. Computer processing power is advancing at a rapid pace … in spite of this power, battle command automation remains focussed on communications-intensive, message-based paradigms.” [9]

MBBC is based on an architecture of distributed databases known as distributed fact bases (DFB). DFBs are updated and managed by a protocol that is cognisant of the potential limitations of the wide area communications environment (low bandwidth, intermittent connectivity) known as fact exchange protocol (FEP). Users interact with their local DFB via a local command and control application, and the workings of the DFBs and FEP are totally transparent to them. The difference between the two paradigms is illustrated by Figure 2.

The FEP should be capable of operation in an austere communications environment, and will automatically manage services such as replication and synchronisation. In the model:

“… synchronisation criteria can be defined as a function of the derived communications performance, and as available bandwidth varies, so does the synchronisation between databases. … when communications fail completely, the application programs can provide predictions until communications are re-established.” [9]

A set of dynamic, business-oriented rules, termed information distribution commands (IDCs) is developed to regulate the exchange of data between DFBs. For example, “when information about four or more armoured vehicles arrives (criteria), then send that information to my parent and adjacent nodes (action)” [8]. IDCs which cause an automated change of state to an application are termed triggers (a term also used in the modern commercial database world). A set of fundamental rules (such as: never send a message back to its source) ensure that infinite loops are avoided. Use is also made of opportunistic receipt of information, via overheard messages, to update DFBs.

The complete set of IDCs allows what Chamberlain proposes is a new information distribution policy: “when passing information up the hierarchical organisation tree, also include any sibling and adjacent units that require the information.” [8] This allows units who most require critical information (such as enemy contacts or presence of friendly units for fratricide avoidance purposes) to obtain it the earliest. This differs from the traditional hierarchical tree-based information distribution policy, and would allow, for example, the passage of information from a sub-unit to another sub-unit on its flank, even if it belonged to another formation.

Chamberlain also proposes a new form of replication (which he terms promiscuous) which is based on “ … ‘best effort’ rather than a discontinuous guaranteed result.” [9]

MBBC fits well with the earlier theme of this paper – that CSS databases need to be managed in a holistic, distributed fashion, independent of user intervention – and is a logical extension of the DDBMS philosophy.

There are recent research areas and developments in commercial database technology, which offer at least some of the required features of MBBC. These include:

• Temporal and Spatial Databases. [11,12] These offer advanced features for handling temporal and spatial data, which is a key requirement of CSSs, particularly in the area of situational awareness.
• Deductive Databases. [13] These offer the capability to define rules, which can deduce or infer additional information from the facts that are stored in a database. The rules can also be used to manipulate or distribute the data, although this area is more the province of active databases, described below. Deductive databases can be though of as a combination of traditional database data storage and manipulation capabilities and those of artificial intelligence systems.
• Active Databases. [14] These provide the functionality of a traditional database and, additionally, are capable of reacting to state changes, either internal or external, without user intervention. This includes the functionality associated with triggers.

Recent versions of SQL, the principal relational Data Definition/Manipulation Language, support spatial and temporal queries, and triggers. Commercial DDBMS technology also provides a degree of network awareness; for example, automated replication management takes account of whether communications links are available or not, and if not, will keep the distributed database in as consistent a state as possible. However, this is unlikely to provide the degree of network awareness required by MBBC.

Conclusions

Military CSSs extensively use relational database technology to store and process critical operational data. The nature of the modern battlefield means that flexibility, survivability/availability, integrity and performance are key requirements when designing and managing databases that may be geographically dispersed and, frequently, utilise disparate DBMSs.

There are essentially two broad options for managing databases in this environment. A holistic, automated approach utilising a DDBMS, GDD and a Distributed Transaction Manager, is one option. The other is an architecture based on stand-alone databases, with manual or semi-automated synchronisation using formatted messages such as EDI or military formatted message systems. In view of the nature of the environment, with mobility, frequent organisational regrouping and unreliable communications, the use of a DDBMS would appear to be the better option.

Replication of data helps to ensure survivability/availability. To ensure that replicated data remains consistent, there will be a need for an automated replication manager. The most likely option for replicated data management is primary site, whereby data is effectively "owned" by the site at which it is entered, although other users can have read-only access.

There is also a requirement for a distributed transaction management system, including concurrency control, recovery and commit protocols. Given the unreliability and limited capacity of communications in a CSS environment, automated query optimisation is essential.

Chamberlain’s MBBC paradigm builds on the DDBMS philosophy to provide improved command and control processes and systems; the updating and management of which should be largely transparent to the user. In this model, it is acknowledged that strict synchronisation of distributed databases will not be possible, and will be on a “best efforts” basis, with predictions being made when local data is known to be superseded.

In summary, modern distributed systems technology provides techniques and tools to allow for the effective management of distributed databases in a military CSS environment, including at least some of the functionality required by MBBC.

References

Author

David Wilton has been employed by the Australian Defence Science and Technology Organisation, in the field of distributed systems, since February 1999. He has a BSc degree in physics from the Royal Military College of Australia, a postgraduate diploma in computer science from Victoria University of Wellington, NZ, and is currently completing an MSc in Information Technology at the Australian Defence Force Academy.