Meeting The Challenges Of Delivering Software Development Projects

Introduction

This article is the first in a two-part series which examines software development projects and how we might improve their management. Modern communication and information systems (CIS) and weapons systems cannot be separated from the software that enables their operation. This software takes a wide variety of forms. Software might be a computer application designed to generate codes for communications security, it might control the flight of a missile. Software might be embedded in a microprocessor integrated circuit (IC) chips or might be designed as an application or operating system for a microcomputer.

The need to develop highly functional software for military applications is accompanied by an imperative that the software is highly reliable. Unlike commercial software applications for a desktop computer used in an office environment, where temporary failure might be tolerated albeit with considerable frustration, it would be unacceptable for a combat net radio to fail, for example, as a consequence of contention between modules of code being “called” simultaneously. The ramifications of such failure occurring cannot be tolerated in critical activities such as when an air strike is being coordinated during the heat of battle, or when attempting to attract attention of friendly forces, who have just begun to fire on your own position.

Despite the demands for high levels of reliability, given enough time and resources, software can be developed to meet the most demanding functional requirements. However, a critical challenge for those who define capability requirements develop military systems, and those who have responsibility for acquiring them is that they meet user requirements and need to be delivered in a cost-effective and timely manner.

Regardless of whether the software is bespoke, and thereby having to be developed ab initio, or is the product of integration of existing commercial off-the-shelf (COTS) or military-off-the-shelf (MOTS) software, the threats to achieving the required performance on time and within budget can be substantial. Indeed, software development and integration projects are costly and frequently by the time they are ultimately delivered they can exceed the original schedule and budget estimates by factors often between two and three.

Recognizing this reality, a pragmatic project manager might make estimates of time and cost for such a project using traditional methods based on amount of work to be done and rate of working and then scale those estimates up by a similar factor, without being able to rigorously explain how these inflated these estimates will be appropriate. History tells us that conventionally based estimates need to be inflated to this extent reflect real costs and schedules. Evidence of the difficulty managers have in providing plausible explanations for cost and time overruns in developmental projects, particularly for those military CIS projects involving software development, or software integration can be found in ANAO reports, Joint Parliamentary Accounts Committee reports and questions posed in Senate estimates hearings.

This article does not seek to apportion blame to anybody in particular for cost and schedule overruns incurred in the achievement of necessary system performance. Rather, it addresses the systemic influences which underpin cost and schedule overruns and examines how we can make robust estimates of time and cost for software projects, and indeed for all developmental projects. It also examines how we might develop effective strategies for managing resources allocated to such projects, particularly for the management of quality.

What makes developmental projects different?

Mankind has delivered complex projects for thousands of years. If the primary focus is on delivery of some highly desired capability, or impressive end products, then countless successes have been delivered. Around 4500 years ago the ancient Egyptians engaged in building the first of the mighty pyramids. Over 2000 years ago the Romans built cities with water reticulation and sewerage systems and cities of the Roman Empire were connected by over 90,000 kilometres of paved roads. Some roads, bridges and aqua ducts built by the Romans are still in use. Early last century, the Pacific and Atlantic oceans were joined by the building of the Panama Canal. In the 1960s the United States of America fulfilled the dream of President John F. Kennedy of taking man to the Moon and bringing him back alive. In the 1990s, England and France were joined by the Channel Tunnel.

These projects are held as exemplars of mankind’s achievements and when measured in terms of what has been ultimately delivered, doubtless they have been outstanding successes. However, if achievement of the original cost and schedule estimates were applied as key measures of success, then it is likely that we would be seriously challenged to defend any of these impressive projects against being classified as failures.

What these project share is that they were unique undertakings. At the time they were undertaken they involved new technology, which often had to be developed before the project could be finished. They involved innovation, experimentation and development of new ways of doing things.

For example, building the Panama Canal was not simply a matter of excavating a long canal wide enough to enable the passage of tankers and container ships. To build such a canal required engineers to find and survey alternate routes through jungles, swamps and over mountains. They had to find ways around rock that was too hard to excavate and too expensive to blast their way through. They had to find ways of raising ships many hundreds of metres over the mountains. Rates of progress in building the Canal were excruciatingly slow at times. The project stopped and restarted several times because sufficient funds and workers could not be found. During construction, thousands of workers became ill and some 20,000 died. In the face of a mounting death toll the project was stopped until mosquito-free barracks could be built to protect workers from mosquito-borne disease and medical facilities provided to treat the sick so they could return to productive work.

The Channel Tunnel (Anderson and Roskrow, 1994) had its own false starts and work stopped on several occasions. Just when it looked like it would be finally finished, extensive re-work was required. For example, means had to be found to extract the heat which built up as a consequence of friction between molecules of air forced through the small gap between the tunnel walls and trains travelling at high speed.

Each of these projects involved substantial rectification, or rework. That is, work not done correctly or not producing the required performance or reliability had to be done again. What we have learned from centuries of complex projects is that the need for this rework is generally lessened as we gain experience about project-related processes or as we learn more about the technology involved. Also, of course, we should expect to do less rework in project tasks that have been successfully completed a number of times before.

Whilst conventional wisdom does not consider them to be so, many activities actually involve rework, and to consider them otherwise is risky. For example, the following involve rework because they are highly dependent upon iteration (after Boehm, 2007):

• incremental, iterative development of statements of requirement and confirmation by users;
• incremental or concurrent, iterative requirements definition in software engineering terms;
• incremental, iterative systems decomposition and definition;
• development of stakeholder trust and commitment;
• development of user trust and commitment;
• development of trust between sub-contractors and contractor or contractor and customer, noting that new partnerships between contractors and sub-contractors are increasingly frequent: they start with relatively little built-up trust;
• development of group performance, which is vitally dependent on development of robust trusting relationships through the experience of a history of honoured commitments: without trust, partners must specify and verify details which is increasingly untenable in a world of rapid changes.

Unfortunately, building trust and gaining experience are costly, often requiring us to fail so that we might succeed in the future.

Some types of contemporary projects are almost always delivered late, and prove to be much more expensive than the original estimates provided at the time funding for the project was sought. Consistently amongst projects that overrun budget and schedule are those that involve software development. This remains the case despite a growing body of knowledge and experience about the detailed processes followed to deliver technical outcomes.

Modern developmental projects

Modern developmental projects can be likened to the mega construction and aerospace projects mentioned, insofar as new ways of working or developing new technology to overcome unexpected problems have to be found. However, all developmental projects are continually under threat of slipping behind schedule, with consequent increases in cost, for surprisingly simple reasons. Prime amongst these reasons is almost universal inability to estimate reliably the extent of rework needed. Even when we are able to describe the processes involved in rework cycles with some precision, our judgment fails us. Estimates we make of time to complete any project involving rework at best are unreliable, and at worst are not much better than informed guesses. Further, this situation is exacerbated by linear-causal thinking embodied in our training, our education, and the prescriptive methods encapsulated in the body of knowledge about the management of projects.

Our inability to make reliable estimates of the extent of rework and implications this has for schedule and cost is an insidious problem. This is derives from:

• human cognitive limitations when faced with dynamic feedback (Dörner, 1980; Diehl and Sterman, 1995; Kleinmuntz, 1985; 1993; Kline, 1995; Richardson, 1991; Sterman, 1989a; 1989b; 1989c; 1994); and
• time-dependent behaviour being difficult to predict, with dynamic feedback response to remedial strategies being almost always counter-intuitive (Meadows, 1989; Forrester, 1961; 1971; 1975; 1987; Nuthman, 1994).

It might be argued that unlike building of the first pyramid, first-time construction of an inter-ocean canal, putting man on the Moon for the first time and bringing him back alive or developing novel tunnel-boring methods for building the world’s longest under-sea tunnel, developing software for military projects would be relatively straightforward. This might appear to be so when our accumulated knowledge about projects is considered. For example, undertaking military software development projects surely is a matter of applying well known methodologies for capturing requirements, designing the software, writing code, integrating modules and acceptance testing.

Systems engineering methodologies (Blanchard and Fabrycky, 1997; Stevens, et al., 1998) which provide the foundation for the engineering of software were spawned by NASA’s Apollo space program. These methodologies have become highly developed in the last 50 years, and continue to evolve. The software engineering body of knowledge is now comprehensively defined. Many engineers and project managers are trained in software engineering and software project management.

Billions of lines of commercial software code have been written. Some software applications we use every day such as Microsoft™ Word are exceedingly complex, contain millions of lines of code. There is a large pool of experience in writing software. Indeed, some of the world’s largest corporations have built their success on their declared ability to develop software.

Unfortunately, this commercial experience does not readily translate through to success in the development of software needed for military applications. This is particularly so if we measure success in terms of achieving planned cost and schedule. Whilst the software used in military applications often contains orders of magnitude fewer lines of code than commercial applications this does not improve success rates in delivering military software projects. By comparison with Microsoft™ Word which contains some 2,700 thousand lines of code (2,700 KLoC); the C code which controls the operation of the RAVEN VHF radio contains some 23 KLoC. Despite its apparent simplicity and small size, after the RAVEN VHF software had been subjected to field trials and its functionality demonstrated, it required substantial rework. Siemens Plessey Defence Systems originally estimated that this rework would take three to six months and costs incurred by them in doing this rework would not be passed onto the customer, the Commonwealth. Rework eventually took around three years and is understood to have cost a further ₤10 million sterling.

One attribute that differentiates military software from commercial software and increases the amount of rework is the need for military software to be very reliable. Fielding beta-version software for operational use in anticipation of a final release, then entering into an extended program of reliability growth, is not an option for military systems. If handling classified information, the level of trust required in the software considerably increases the need for reliability. Further, military software is frequently embedded in hardware and this brings unique demands for creating working interfaces between the software and hardware.

Each of these issues places particular demands on software engineering and project management skills needed by programmers and quality management. But, before any project can be approved, viable estimates must be made of the time and cost associated with expected successful delivery of the required functional performance. Further, if success is to be repeatable in subsequent projects there must be confidence in the likely achievement of schedule and budget, as well as functional performance.

The need to do rework is often unavoidable, or is inherent in the delivery of many types of project. Almost all engineering, systems engineering and software engineering deliberately involves iteration at some stage or other. Having to do rework is not the problem, though to do less would be a good thing, it is just that do we not account for it particularly well in our planning.

Perhaps the greatest threat to schedule and budget is not how to deliver technical performance, per se, but how to make reliable estimates of the rework needed. The project’s sponsor needs to be convinced of the veracity of these estimates made before the project is approved. Further, the basis for claiming the veracity of these estimates needs to be established for currently planned and future projects to be successful.

Rework and quality management – a simple metaphor

A bathtub is filled with a very large number of marbles. For our purposes, these marbles are all green in colour. Each marble represents an amount of work to be completed, such as the writing of a number of lines of software code. Marbles are scooped out of this first bathtub one bucketful at a time. Each bucketful represents, for example, the writing of code planned to be completed in a single day.

Unfortunately, a small fraction of the marbles is damaged as they are scooped up from this first bathtub. This damage is akin to defects (errors or bugs) inadvertently being added to the software as it is written. Fewer defects are likely to result when highly skilled programmers are employed in writing the code, but there will always be some defects. The second bathtub is progressively filled with our metaphorical green marbles, some of which have been damaged by the very method used to move them from the first to the second bathtub. All marbles, including the damaged ones, are placed in this second bathtub.

Damaged marbles are not easily identified except by close and careful inspection by skilled inspectors with special equipment. A selected sample of marbles from the second bathtub is carefully collected and inspected. Unfortunately, even the most rigorous inspection process will not find all damage on the marbles due to errors in the inspection process, which cannot be perfect. Each damaged marble found is painted red at this point. All marbles, both green and red are placed in a third bathtub by the inspectors.

Marbles are progressively taken from the third bathtub and the red ones are repaired. This is the rectification work carried out by programmers. When repaired, the red marbles are painted green. Now all marbles are green, appearing exactly as they did when they were placed in the second bathtub for the first time except that only a small number are defective, and those defects have not yet been discovered.

All the marbles so far inspected, those rectified and a small number of marbles having undetected damaged are progressively placed back into the second bathtub. We would expect the marbles that have been inspected and rectified not to be mixed with marbles placed in the second bathtub for the first time, but this would only occur under specific conditions where:

• each marble has a unique identity and is tracked accordingly; and
• rates of doing initial work, inspection and rectification are well matched to the rate of inspection leading to acceptance of rework, that is, when the last of these does not result in an accumulation of inspection to be done.

Unfortunately, in the general case and unless we carefully design inspection and rectification to specifically avoid it, mixing can occur.

Of the green marbles now placed back in the second bathtub, the portion likely to be green again, having been identified defective and therefore rectified, is hopefully nearly 100%, but always effectively less than 100%, of those that were defective before they entered the inspection process. How close we are likely to get to finding 100% of the defects, of that portion of the total actually inspected, depends on the skill, effort and tools applied during inspection. The effectiveness of the inspection process is highly non-linear, meaning that results obtained are not achieved simply in proportion to the effort expended. Further, there is a minimum threshold in terms of effort which must be applied to achieve any significant results, and above a certain higher level further incremental increases in inspection effort will return marginally diminishing results.

When they are sufficiently satisfied with the quality of the rectification work, our team of inspectors select marbles from the second bathtub and pass these on to the next stage of the process or phase of the project. The aim here is to empty the second bathtub so that work can proceed through the next phase of the project. It is important to note here that marbles from the first bathtub and the third bathtub appear the same. They are randomly mixed. One consequence of this is that some defects will never be found. In the general case, some of our marbles are passed on to the next phase without being inspected.

A further complication here, not considered before now, is that our green marbles are related to each other. Some marbles will be related to a large number of marbles. Some marbles will be related to a smaller number of marbles. All marbles are related in some way, through the functional structure of the software being built. Our metaphor does not extend to taking this functional structure into account, but we can accept this limitation for now. In practice we would use knowledge of this structure to direct the focus of work, inspection and rework effort on critical lines, modules or functional blocks of code.

Our metaphor applies to a single phase of a project. Typically, a project will involve a number of phases. Each of the phases is similar in that each involves rework. Rework can also occur as a consequence of defects discovered in a later, or downstream, phase of the project. This inter-phase rework demands input into one or other of the earlier, or upstream, phases of the project. So, whilst rework is an element of each phase, there is always the possibility that the need for some inter-phase rework, or rework of the rework, will arise.

Meeting the challenge of managing developmental projects

Unfortunately, conventional project management estimating is based on a simple mechanistic relationship between work to be done and rate of applying effort. Of course, the rate of applying effort is not uniform, and experienced project managers claim to know this. Indeed, this is well documented in the body of knowledge. Rates of progress are initially slow before ramping up to higher rates then slowing down as the project of phase nears completion. Regardless of the variations in rates of working as the project of phase progresses, the fundamental totality of work to work rate relationship is embedded in the way calculations of the estimate at completion (EAC) for the project, that is, what the whole project will cost to complete, are made.

In conventional project management terms, rework is almost universally accommodated by providing management reserves, time buffers, and contingencies, where the determination of the need for these is made on the basis of the experience of the estimator. Models such as the COnstructive COst MOdel (COCOMO), (Futrell, et al., 2001: 372-399) have been developed as a consequence of extensive empirical research and are used extensively for estimating the effort required to complete software projects. These models use a combination of parametric characterization and regression analysis based on historical performance in similar projects to enable predictions to be made. The validity of this approach is not questioned, per se, though these models assume ways of working will remain largely unchanged. Further, whilst various factors such as volatility of requirements specifications are included, the way causal factors impact upon the need for re-work are not specifically considered.

Infrequently, or rarely, is the extent of rework quantified with any rigorous understanding of rework mechanisms, with the consequence that projects such as those involving software development result in significant cost and schedule overruns. It does not require much imagination, and reflection on the rework metaphor, to realize that our usual methods of estimating the extent of rework will fail us except where rework is minimal such as that for routine, technically naïve projects. To make reliable estimates of the impact rework will have on software development projects, we need different techniques.

Conclusion

The way we take rework into account in making estimates of the time, effort and resources needed to deliver software development projects affects the reliability of our estimates and hence the confidence we might have in estimates made before committing to a particular project. The second part of this article explains how a combination system dynamics model-based simulations and scenario planning can be used both to enable improved estimating and to reveal the causal drivers of variations in time and effort required to deliver such projects.

References

Author

Dr Alan McLucas is a Senior Lecturer at the School of Information Technology and Electrical Engineering, University of New South Wales, at the Australian Defence Force Academy. E-mail: a.mclucas@adfa.edu.au.