Software Engineering or Evolution? The Battle for an Army Command Support System

Introduction

The Australian Army has pursued the development of a computerised command support system since the mid-1970s. The concept for computerised support to command and control originated during the Vietnam War, after a Defence Science and Technology Organisation (DSTO) computer was deployed in support of intelligence operations. The Army developed a capability proposal for the Australian Army Tactical Command Support System (AUSTACSS), the first phase of which was approved by Government in 1983. At around the same time, the information technology revolution was beginning to accelerate. However, system development methodologies for evolving systems that could cater for rapid technological change had not yet been developed. Consequently, the evolution and development of AUSTACSS was bound to have the difficulties, both within the Defence Department and industry, that almost led to its failure.

AUSTACSS was initially developed under a classic “waterfall” development methodology. The development plan involved three sequential phases; a feasibility study phase, a requirements analysis phase and a development phase. When it commenced it was considered that a fully bespoke software development would be required. When the project reached the development phase in 1992, a firm price contract and fixed specification was used to engage the prime systems integrator and developer. By 1996, the project had delivered a mix of Commercial-Off-The-Shelf (COTS) products and more than 600000 lines of bespoke code. While the product met the original specification and had some very successful components, it no longer met overall user needs or expectations. In 1998, the project was revamped to be come a purely COTS integration and high-level applications development project, changing its name to the Battlefield Command Support System (BCSS). At the same time, in response to user requirements and the emerging Defence Common Operating Environment (COE), the system moved from Unix to Windows NT. The BCSS project successfully delivered COTS based systems to users in 1999.

This paper examines the development of the AUSTACSS and BCSS projects and describes the difficulties that arose and the solutions that were adopted to overcome them. Consideration includes the software development, project management and contractual issues involved in managing a complex software and hardware project.

AUSTACSS feasibility study and requirement analysis

In 1984 and 1985 the feasibility study looked at the fundamental concepts of a command and control system and concluded that a single, integrated command and control system was required. The study also recommended the adoption of a common architecture for all of Army’s battlefield command support systems [1]. The feasibility study established a testbed that was collocated with the headquarters of the then primary user, Headquarters 1^st Division. The testbed was utilised to establish the user requirements for the military operations and intelligence functionality of the overall system. In so far as a generic system and functional requirements could be determined with the level of technology available to the testbed, this phase was successful.

At that time the user community in Headquarters 1^st Division had virtually no computing systems, so the user had no framework for comparison with other systems. The testbed utilised systems that were at the leading edge of available computer technology. Even so, this technology was initially based around centralised computing systems using terminals with limited graphics or windows type capability. The limited human-machine interface and relative unsophistication of software meant that user requirements development focussed on capturing the existing military processes rather than considering how computing systems might enable new ways of doing business. The users’ unfamiliarity with the potential business changes offered by computing systems and the desire for manual reversion as a redundancy measure led the Army to insist that AUSTACSS “...must be entirely compatible with the manual system” [1]; a goal which was later to constrain development flexibility.

The second phase was the requirement analysis phase and utilised the same contractors. Utilising the testbed to engage the users at Headquarters 1^st Division, a detailed AUSTACSS Requirement Specification (ARS) was developed over the period 1986-87. Software was developed on the testbed to confirm and evolve user requirements. The ARS produced was an eleven-volume document with a further four volumes of supporting documents. It provided a highly detailed and complete description of the requirements at that time. They covered the operational environment, operational requirements, functional requirements, performance requirements, information exchange, and management plans. In many respects the ARS was visionary, including many functionality areas which had still not yet been considered in the commercial software environment (particularly the then unknown concept of a “windows” style interface). The ARS included a number of key lessons arising from the requirement analysis, some of which remained valid throughout the project life:

• “Close user involvement is essential in all aspects of the development and acquisition of a system of this nature.
• The Human Machine Interface must provide the user with an environment characterised by ease of use, minimal training requirement [...].
• The use of battlemaps is fundamental to command and control, and thus employment of automated battlemap graphics is essential to AUSTACSS.
• The vertical flow of automated information within an AUSTACSS user community must be far reaching so that information can be “captured”, disseminated and utilised.
• The system must be sufficiently flexible to allow the configuration and re-configuration of the AUSTACSS facilities.
• The system must be based on distributed, rather than replicated, storage of information. AUSTACSS must be able to continue operating, within a local area (for example, within HQ Div main), without external communications being available.
• AUSTACSS must be capable of operating with degraded communications.”[1]

The project spent a further three years, from 1988 to 1990, converting the ARS into a Commonwealth Request for Tender (RFT) and a detailed design specification (designated the AUSTACSS DD(X)). The project office and a Melbourne-based Army agency responsible for technical engineering developed the DD(X). It took an additional two years, 1991 and 1992, to conduct the tender process and award the contract. Over these five years the user organisation, Headquarters 1^st Division, reduced its support and commitment to the project. There were approximately twelve 1^st Division staff internally assigned to the project when it started but a range of pressures led to their eventual reduction to one. Headquarters 1^st Division still had no basic computer support capabilities for use in barracks or in the field and their need for such support had become urgent. The headquarters began to install commercial PC and LAN technology to meet their in-barracks and field requirements. Unfortunately this was generally done in isolation from developments within the AUSTACSS project. Furthermore, the role of the headquarters changed from a military divisional headquarters to a deployable joint force headquarters, increasing their requirement for seamless applications interoperability with the Joint Command Support System, an area that was not a driving consideration within the original AUSTACSS DD(X).

The level of detail to which the product was specified was even more problematic. The DD(X) included specifications of screen display functionality and detailed process descriptions. It included capability specifications that were not yet available in commercial information technology systems and were unlikely to become available. Significantly, it included the requirement for Multi-Level Security (MLS). Simply speaking, MLS provided the ability for an authorised user to access multiple levels of classified data from the one terminal. This requirement forced an architectural and engineering approach which ultimately increased the complexity and cost of the project, reduced functionality and was unlikely to ever be successfully achieved. Even if it became technically feasible, it was highly unlikely that the security accreditation process would ever provide workable approvals. Other standards such as GOSIP and X.400 were enforced in the specification, however these standards had not yet been adequately defined in the commercial environment. The DD(X) specified the software development standard as 2167A (Tailored)[2], however the tailoring was never adequately defined or completed. While the DD(X) was a comprehensive document, which reflected the perceived needs of the user in 1987, it was rigid, technically unrealistic and suited for a bespoke software development. A further step of technological risk assessment and tailoring was required, perhaps during contract negotiation, however this never occurred. All the above risks were never properly considered due to the fact that different stakeholders were driving different requirements without any project management or board approach in place which could resolve the competing requirements and risks.

AUSTACSS development phase

The November 1992 contract for the first development phase, Phase 3.1, was a firm-price, three-year contract to produce an Initial Basic System (IBS). This phase would define the system hardware and software architectures. Networking and communications interfaces would be built and a number of software components would be developed and integrated. This would include the development of a so-called ‘Command Support Platform (CSP)’ which would provide consistent applications programming interfaces to the AUSTACSS applications and request and provides services from the operating system and other COTS packages. The common “Tactical Electronic Office” tools would be fully developed for delivery in the IBS. Interestingly, the contractor signed up to the DD(X) without significant amendment. The IBS would be delivered to all the high priority units within the Army by the end of 1995.

The contractor solution was constrained to a large extent by the MLS requirement and the requirement for extremely high levels of redundancy. It forced the adoption of Unix workstations for both system servers and user clients. This meant the DOS/Windows, Novell and Lotus client applications that were exploding onto the commercial and PC marketplace could not be considered for use in AUSTACSS. The AUSTACSS solution was COTS-based, but these Unix compatible COTS applications generally had less functionality. Furthermore, the requirement for Unix client workstations meant that the cost of software and hardware was far higher than originally estimated. These problems of lack of user acceptance and the high cost of software and hardware eventually forced the abandonment of the design. However, because the project had no effective risk management systems in place, these fundamental business case risks were neither properly identified nor considered at the time. The project accepted the proposed design solution and the contractor commenced software development.

As software development proceeded the problems with CSP/COTS interfacing, the DD(X) and the 2167A standard became more apparent. As the CSP was being developed it had to handle continuing version upgrades in the COTS packages. It became increasingly difficult for the contractor to develop a stable CSP and to identify whether any fault was a COTS problem or a CSP problem. When the domain that caused the problem could be identified, the Contractor and Commonwealth argued over whether the fix was within the contract price or not. This inability to share risk and mitigate costs led to an increasingly defensive development approach. The development of CSP services were also complicated by continuing delays in the standardisation and specification of GOSIP, X.400 and a range of other military interface protocols. The AUSTACSS project was required to interface with other Defence communications and computer projects at a range of levels in the OSI stack, however these other projects continued to adjust their implementation of standards and protocols. At the user applications end of the work, the DD(X) created additional problems. The DD(X) was, in many areas, so specific and stringently applied that the contractor was obliged to alter or adjust COTS functionality. The MLS requirements often required the removal or constraining of COTS services and the direct connection of applications to the operating system, bypassing the CSP and making future COTS upgrades more difficult to integrate.

The 2167A approach for software testing meant that the faults arising from these problems were often not detected until very late in the test program. Furthermore the 2167A regime was defective. The Commonwealth testing would frequently only determine if the function was present, not necessarily whether the function worked or whether it was “fit-for-purpose” for a user. This was not assisted by the fact that the testing agency had been located at the Army Engineering Agency in Melbourne, isolated from the users and contractor in Brisbane. Software modules that seemed to work independently failed after integration at system testing. The extensive 2167A documentation requirements were entirely unsuited to the changing COTS environment. Consequently, project management on both sides became increasingly concerned with the management of the 2167A process and its related paperwork.

While all these technical problems were occurring, the lack of an adequate Commonwealth project management methodology was exacerbating the situation. As a matter of course within Defence, once the project sponsor had obtained the project approvals, the project was effectively handed off to the acquisition organisation. The acquisition organisation bureaucratic culture tended to be defensive and secretive about the problems with any project. Generally, projects were pursued by the acquisition organisation with little reference to the original sponsor or the end user. There was no mechanism in place for regular collaborative review of the project business case, plan or risks. As the technical problems within AUSTACSS increased, the pressures on the project director increased but there was no board of management or review to whom he could turn. The response behaviour was to internalise problems and continue to advise that, apart from some cost escalation or schedule slippage, the project remained on track. This approach continued throughout 1995 while AUSTACSS software Builds 1 and 2/3 were successively released and rejected by the project office. It was eventually the contractor who raised the alarm in early 1996, just after the project sponsor had separately succeeded in obtaining Defence agreement to the follow-on phase.

While there was a joint responsibility for the failure to address risks, the contractor clearly bore some responsibility for not adequately initially assessing the scope and complexity of the project. However, once the contract was signed there was no contractual or commercial incentive for the contractor to adjust or improve the outcome. The contract was firm price with a fixed DD(X). As time went on the contractor had less and less flexibility to adjust to user requirements or to changes in technology. Every suggestion for change represented an additional cost to the contractor and if it could not be recovered from the Commonwealth then it could not be done. Once the additional technical problems became evident, the contractor was forced to focus on fixing those problems rather than search for innovative or alternative solutions. The project office wanted changes to accommodate requirements expressed during user workshops and often felt that these changes were within the intent of the DD(X). Requirements change control was not managed well and informal change requests arose from users, sponsors and other interested parties. Consequently, interpretation of the specific requirements in the DD(X) became central to the project office - contractor relationship, underscoring the already over-specified and inadequate nature of the document. The wrangling over the DD(X), combined with disagreement over the 2167A process and documentation overheads, led to an increasingly difficult relationship between the contractor and the project office.

It was hardly surprising then, when Build 2/3 Modified was accepted by the project office in March 1996, the end-user community did want to use the AUSTACSS IBS. In the period 1992 to 1996, the 1^st Division Headquarters had completed roll-out of a commercial PC LAN environment and was about to adopt the alternative JCSS solution to provide joint interconnectivity. Attempts had been made to introduce the Unix operating system and associated AUSTACSS office applications onto the client platforms, however the users would not accept the reduced functionality offered by the Unix-compatible office applications. The project had effectively lost the interest and support of its primary user. The adoption of a ‘big bang’ approach with the associated failure of the project to provide effective computing support in an incremental manner had forced the primary user to look elsewhere. For the remainder of 1996 the project and contract was under review. It was eventually decided to amend the contract to remove some of the technically infeasible requirements such as MLS. The 1^st Brigade in Darwin was adopted as the new user, with an aim of trialing the IBS with this user in mid 1998. This would be followed by Phase 3.2, which would further develop the software and fully equip the 1^st Brigade with the system.

The BCSS approach

Phase 3.2 provided an opportunity to correct the range of problems that had been identified in the project thus far. During 1997 an acquisition strategy was developed and approved that would allow further development to adopt an evolutionary acquisition approach. The Defence Department had gradually come to accept that an evolutionary acquisition approach was essential for command support system software projects. However the management and contractual mechanisms for such an approach had not yet been established. It was decided to adopt the “PRINCE2” project management methodology [3] for overall management of the project. This methodology provided the project with an oversight and review Board that consisted of the sponsor, the user and the acquirer. Independent assurance appointees who had no direct involvement with the project supported the Board. The project director reported to the Board on a regular basis through Board meetings, highlight reports, stage reports and exception reports. The project required a range of management products that were regularly reviewed and updated for the Board. These included products such as a business case, project plan and stage plans developed and approved prior to each stage of the project. The Board would control the tolerances for reporting and agree the major decisions of the project. The PRINCE methodology included a process for visibility and management of risks and issues that would arise during the project. This methodology brought two significant changes to the project. Firstly, the sponsor and user had greater visibility and control over the project. Secondly, the sponsor and user could assist the acquirer and project director in managing the issues, risks and subsequent decisions arising from the project. Significantly, the Defence Acquisition Organisation (DAO) had allowed appropriate delegation to the Board executive so that decisions about schedule and functionality could be made quickly and efficiently. The project was consequently able to keep to schedule and trade off functionality as a risk mitigator.

Early in 1998 the Board approved the project plan, which included a new set of system development principles: [4]

• End-user focus. The project would be focussed on the end-user and the use for BCSS Phase 3.2 would be the 1st Brigade in Darwin. In Phase 3.2 the project would focus on satisfying the requirements and needs of the 1st Brigade rather than become distracted by trying to satisfy the requirements of the wider Army or the ADF. While this enabled the delivery of BCSS to 1st Brigade on schedule, there were some disadvantages. There has since been identified the need to rework some existing functionality as procedures mandated within 1st Brigade were shown by Army doctrine centres to be deficient.
• Incremental Capability Improvement - On Time. A significant advantage of an Evolutionary Acquisition approach was the ability to provide the user with an operational capability early and to continue to evolve the capability. Every delivery would provide an improvement in capability. Delivery would occur when the user requires them but at intervals not exceeding six to seven months. Delivery would occur on schedule and capability would be the contingency. There were several decisions to drop functionality that were contested by both the users and project staff, however delays in including the functionality would have resulted in failure to deliver on time and loss of confidence by the user. The key point is that functionality was sacrificed when schedule was threatened.
• Re-use of AUSTACSS Initial Basic System (IBS) and other Command Support Systems. BCSS Phase 3.2 would utilise features from the AUSTACSS IBS. The IBS features that might be required for re-use in BCSS would be identified during the IBS trials in mid-1998. It would also re-use software applications and products from other command support system projects. For example BCSS would utilise as many of the applications developed in the Joint Command Support System (JCSS) project that were relevant and could be easily ported. Software re-use would not necessarily require direct porting of code and may involve only transfer of design documentation for implementation in the new software product suite.
• Maximise use of COTS - minimise new or bespoke development. The use of COTS products would be maximised to ensure timely delivery and best use of commercially available technology. New or bespoke software development would be minimised to avoid associated delays, risks and higher costs of ongoing maintenance. A COTS product that satisfied 80% of the user requirements early was considered better than delivery of a late product, a product that is not fit-for-purpose or one that would quickly become obsolescent. Generally, if a COTS product cannot satisfy a requirement then the requirement would be deferred until an appropriate COTS product became commercially available.
• Utilise “market-leader” COTS products. The selection of COTS products would generally be based on five factors:
• relative cost for similar capabilities,
• life cycle support costs,
• user acceptance and ease of use,
• the development and support environment associated with the product, and
• the likelihood that the product will be upgraded and supported by the original developer.
• Minimise dependencies between COTS products and component software. BCSS Phase 3.2 development would minimise dependencies between components and COTS products so that they could be easily replaced and upgraded. Development of functionality requiring use of multiple COTS products would interact at the highest level via widely used high level languages, interfaces and protocols. Newly developed software applications would be discrete components that could be independently installed. There were also obvious risks in this approach. Many of the component systems were not as good at being plug-in and pull-out components as intended, having continuing impact on interoperability with other components, JCSS and the Defence Common Operating Environment.
• Build to allow for continuous product improvement and support. Newly developed applications would be designed, built and documented to allow for continuous product improvement as well as ongoing maintenance support. It was anticipated that COTS products would be upgraded in accordance with their commercial release cycles, thereby having relatively short life cycles with new versions appearing every one to two years. BCSS Phase 3.2 software would be data independent and would consist only of additional modules rather than complete system re-installations. This principle is at least partially at odds with the one above. In the final analysis, it is seen as more cost effective to throw out a COTS application which is not transferable or supportable and buy a newly developed product, rather than adhere assiduously to this principle at great expense.

Allied to these principles was the use of a software development approach that included only high-level definition of the user requirement. Once functionality component areas had been determined through high-level user requirement analysis, the project office would conduct market surveys, technical feasibility and business case studies to determine whether the functionality would be provided off-the-shelf, developed or deferred. The Board would agree the priority, scope, funding, schedule and reporting tolerances for the procurement and/or development of each component area. If development became the preferred option, detailed software design and development would occur through the use of rapid application development workshops. These workshops would involve the user and the developer, generally removing the project office from the development process. The workshops would utilise prototyping and concept demonstrator software tools to refine the product with close and continuous user input within an eight-week development cycle. After one or two development cycles the product would move to integration into the next six monthly BCSS release. If products were not fully tested and stable they would not be included in the release. Software maintenance was not considered to be a separate activity. Instead, debugging and upgrade would be included in subsequent development cycles for inclusion in the next release.

Commonwealth – contractor relationships

A new contract approach was also required which could ensure a product was delivered while providing adequate reward and incentive for the contractor to perform at the standard of world’s best practice. A firm price, fixed term and whole-of -project approach was unsuitable for an evolutionary acquisition project. Instead, the contractor was engaged on an annual basis at an agreed level-of-effort for design and prime system integration services. This contract was renewable each year and the project office was under no obligation to renew or to justify non-renewal. Fundamentally, if the contractor was not performing at the level expected, the project could move to a new contractor. Equally, while the contractor delivered, there was no requirement to switch. This mechanism was deliberately intended to build confidence into the Commonwealth - Contractor relationship over the long term, promising a genuinely innovative and collaborative development environment. The Commonwealth owned the Intellectual Property associated with work done under the contract and the contractor was obliged to assist in transfer of knowledge to any new contractor. While the contractor was on a short lead, the contract provided additional incentives through granting certain marketing and licensing rights outside Australia for the term of the contract.

Software development work could be contracted under additional work orders, providing flexibility to adjust priorities and respond to user requirements. Software development could be sub-contracted and this also fostered alternative options to that presented by the prime. Generally it was in the interests of both the contractor and the project office to ensure a reasonably constant level of effort was planned. Commercially competitive and indexed rates for the design and prime system integration services and the work packages were established. Significantly, the overall risk for development of the project was shared between the contractor and the project office. In order to conform to Defence corporate information technology standards, the contractor was required to use specific software and hardware products. The concept of transferring all risk to the software development contractor was no longer workable, but with the above approach the risk to the project office was limited by relatively small increments and packages.

The outcome of BCSS Phase 3.2 has so far been very successful. The user was delivered an initial Microsoft NT network environment in-barracks less than eight months after the commencement of the project. Field hardware and specialist military operational software tools were delivered within a further six months. The specialist military operational software was a COTS product identified through market survey. The product appears to cover about 80% of the original functionality needs of AUSTACSS. Other specialist software components were successfully developed through the rapid application development process. Most importantly, the system works, the user has happily accepted the product and demand for its expansion throughout the Army is high. Additional work will continue to be required in refining the software suite, expanding functionality and refining the approach for military platform installation. This is to be expected in an evolutionary project of this type. The PRINCE2 project management methodology has provided appropriate level of oversight. The contractual arrangements appear to be working well. The project office and the contractor have a good working relationship and a common goal of satisfying user requirements. The major risks for the future relate to configuration control and component interoperability as the system expands to met the needs of additional users across the Army. One approach here might be to maintain a lead user organisation, allowing only minimal tailoring or separate components for other user requirement sets.

Conclusion

The AUSTACSS/BCSS project is a prime example of the dichotomy between traditional software engineering approaches and the more evolutionary development approaches required in times of rapidly changing information technology. The traditional 2167A design and development approaches were appropriate for bespoke software development projects where the user requirements remained relatively static and where delivery schedules could be managed in much shorter time-frames. They may still be necessary for components of command and control systems that require very high reliability and have little human-machine interaction (such as sensor and weapon systems). However, where COTS products need to be used and where the human-machine interface is significant an evolutionary approach is essential. The evolutionary software engineering approach needs to be matched by an appropriate project management and oversight mechanism that supports appropriate collaboration between the user, sponsor and acquirer. The traditional Defence habit of ‘handing off’ projects to the acquisition organisation for procurement is not sustainable for information system projects. Finally, the nature of the relationship between the project office and the contractor needs to be fundamentally cooperative. Risks will be shared and the contract must allow for work package flexibility and incentives for high levels of performance. With all these approaches in place it is be possible to create the conditions for success.

Disclaimer

The views expressed in this paper are personal and do not represent the views of the Australian Department of Defence.

References

Author

Colonel Peter Lambert graduated from the Royal Military College, Duntroon, in 1979 into the Royal Australian Signals Corps. He subsequently served in a range of postings in Australian Army communications and signals intelligence units. He served for a year with a signals intelligence unit in the British Army On the Rhine (BAOR) in 1985. He completed Australian Command & Staff College in 1990. In 1993/94 he served as the Commander of the Australian Service Contingent to the United Nations MINURSO mission in the Western Sahara, Africa. In 1996/97 he served as Commanding Officer of the 7^th Signal Regiment (Electronic Warfare). He has completed staff appointments in strategic command division (then Joint Military Ops and Plans), the capability development staff and the Defence Acquisition Organisation (as Project Director Battlefield Command Support System). His current appointment is Director Information Operations Policy and Capabilities (DIOPC) in the C4ISREW Capability Staff at Australian Defence Headquarters (ADHQ). Colonel Lambert has a Bachelor of Science degree with first class honours in Computing Science. He has Masters degrees in Defence Studies and Information Technology. He is currently studying towards his Doctorate in Information Technology.