The Assessment Of Staff Training Within A Context Of Acquiring Military Capability

Introduction

Background

The defence communities in the US and UK, and indeed other nations, are undergoing an almost revolutionary change as they endeavour to operate Capability Management, which can be defined as:

‘Acquiring the ability to undertake required tasks, without specifying exactly how the ability is to be acquired.’

To effect Capability Management significant organisational and process change initiatives have been employed. Although the move to a capability-based approach requires fundamental changes to be made across the whole of defence management, so far, change initiatives have been mainly focused on Equipment Acquisition.

For Capability Management to achieve its potential, a mechanism is required to enable all capability components to be managed coherently. Furthermore, it is our premise that unless you can measure capability you cannot manage it. This paper proposes a structure for managing the three generic components of capability, namely People, Process and Materiel, and we illustrate how this Capability Management framework can be used to manage People Capability and show how capability can be linked to Operational Risk.

Scope of the paper

As stated above most Capability Management is geared towards the delivery of equipment capability. It is clear that the delivery of overall capability is not achieved solely by equipment solutions and that manning is a considerable input into the equation. Hence we need a fresh approach to the evaluation of the people component of Military Capability and a mechanism to assess its contribution to defence capability.

This paper presents a method for the assessment of Military Staff Training within the context of Defence Capability and the impact of such training in terms of Operational Risk. Military Staff Training is now largely achieved through Computer Aided Exercises (CAX). It is imperative that the effectiveness of this training is evaluated in a rigorous manner that is both militarily and scientifically credible.

This paper specifically addresses:

• How Defence Capability can be defined and linked to Operational Risk.
• How to assess Staff training.
• How to connect the assessment of Staff Training to Defence Capability.
• How to estimate Staff training costs so that intra- and inter-Capability investment decisions can be supported.

The methods presented in this paper have been used on a number of major simulation-based staff-training exercises, including joint UK/US exercises.

Defining military capability

The business analogy

To help us describe and define the concept of Military Capability we will use an analogy with a commercial business operation. Most businesses can be defined in terms of three capabilities: People, Process, and Materiel.

These three capabilities interact with each other to produce the output of the business, profit (Figure 1).

The top-level business process, as shown in Figure 1, can be decomposed into a hierarchical set of such processes. For example, the materiel is provided through an acquisition enterprise as shown in Figure 2. The success of the business is dependent on how well the top-level process is made to work and this is entirely dependent upon the coherency down the hierarchy of sub-processes. In the example in Figure 2, coherency is required to ensure that the delivery of the Materiel is timely, within budget and fit-for-purpose.

Clearly, the organisation responsible for delivering a nation’s Defence Capability is not in the business of making a financial profit, but we show below how a ‘profit-equivalent’ can be constructed for the purposes of supporting defence expenditure decisions.

Defence ‘profit' and operational risk

The final output of the national defence, known in the UK as Operational Effectiveness, could be said to be ‘winning wars’; a more useful measure, however, would be ‘to meet a politically acceptable probability of winning those wars that the Government deems to be desirable’. We therefore define Operational Effectiveness (OE) as: “The probability of achieving an assigned mission in a specified scenario”.

We will use OE as the defence equivalent of profit. Clearly, Operational Risk (OR) is (1–OE) and therefore the containment of OR is achieved by delivering OE.

Returning to the analogy, successful businesses attempt to optimise the profit across all areas of the business by making best use of the available resources. We show in this paper how defence capability can be managed, in much the same way, by optimising Operational Risk. We also show how this defence ‘profit’ is likely to vary in response to changes in the resources, and particularly to indicate the likely time lag between the decision to vary resources and the impact in the field.

We must emphasise, however, that unlike commercial enterprises, this defence ‘profit’ can only be estimated and it will have an associated level of uncertainty. The uncertainty will be greatly compounded if the targets set to deliver the ‘profit’ are ambiguous or not quantified.

Our definition expresses OE in terms of a probability of success, and it must be recognised that whilst we would all like the probability to be 100% this is, in practical terms, impossible and certainly unaffordable. The decision on what level of OE is deemed acceptable is a political one in a democracy. The CEO of the business therefore, has to balance the priority of the mission, the predicted OE and the resources available (that is, in our analogy the importance of that element of the business, the likely profit and the resources available for maintaining/altering the profit). Following the business model described above, the CEO has to be able to change the Defence Capability mix within the politically agreed resources.

Following the Capability-based business approach described above, we need to have a method of estimating OE. This estimation results from a comparison of Own Defence Capability and Enemy Defence Capability in a given scenario (Figure 3). From the discussion above, Defence Capability is made up of a complex combination of People, Materiel and Process (Figure 4).

Costing

Clearly the approach described above can only be useful as a management tool if it is possible to estimate the likely financial implications of changes in these three elements of Defence Capability; we will describe how cost of delivering capability can be estimated.

People capability

Within the scope of this paper we illustrate our methodology by discussing the sub-process that delivers ‘People Capability’. The following discussion concentrates on the assessment method and the OE impact of Staff Training, and show how costs can be estimated. We can define People Capability in three basic categories as shown in Figure 5. In turn we can define Skills as shown in Figure 6.

For the purposes of this paper, we are interested in the collective skills of the Staff and how these can be related to OE.

Measurement of staff training

We use the discussions in the remainder of this paper to describe and illustrate a method for assessing Staff Training that is consistent with the capability management framework outlined above. It is our conjecture that in isolation a mechanism to measure and manage staff training is itself a valuable tool, but to realise its full potential requires:

• an understanding of the contribution of staff training to the delivery of OE; and
• integration within a mechanism that will support the defence planner in managing all capability components in coherent manner, and will enable him to understand the operational consequences of his decisions.

Definitions

We start with some definitions:

• The output of Collective Training is defined as Collective Performance (CP).
• Training Transfer (TT) is defined as the overall improvement in CP.
• Training Effectiveness (TE) is defined as the rate of change of CP.

Delivery of staff training

Staff officers learn the basic skills of their trade at Staff College, but it is collective Staff Training that turns these individuals into a team—that is, they achieve a level of Collective Performance.

Until recently staff training was achieved through two main types of exercises; paper-driven Command Post Exercises (CPX) and Field Training Exercises (FTX). With the advent of affordable computer simulation Computer Aided Exercises (CAX) have become accepted as the preferred method of training. Militarily, it is important to realise that CAX can teach collective skills and can readily be used to raise the stress levels as the Headquarters improves its skills. Moreover CAX readily lends itself to feedback to the Training Audience. What CAX cannot do, however, is provide real-world chaos and physical deprivation. There is no suggestion in this paper therefore, that CPX and FTX are no longer necessary. Moreover, the assessment methodology we propose is equally valid for all three types of staff training.

Assessment need

A rigorous and credible assessment of staff training would provide the evidence to justify investment in staff training; of course the need for such justification is growing as budgetary pressures continue to increase. In particular if a direct and consequential linkage can be made between staff training and operational risk then this would provide to defence planners the information they need to make informed decisions and to understand the downstream consequences of these decisions in terms of operational risk.

The measurement of staff training, described below, evaluates the Training Transfer and Training Effectiveness of a staff training exercise through the measurement of CP. A major component of this work has been to develop a method of assessing CP at staff level, which is both practical and credible to military audiences. Furthermore the approach developed can be applied to any Tier 3 training exercise independent of media, and can provide the data required to compare the training value of different exercise mediums.

It is not appropriate to use exercise outcome as a measure of Training Transfer as this is a function of the scenario and chance; a 'good' HQ can, through bad luck or exercise design, 'lose the war' and conversely a 'poor' HQ can 'throw straight sixes' and 'win'.

Therefore the evaluation of the CP can only be undertaken through the explicit application of objective judgement. It is therefore essential that evaluators have a military background at an appropriate rank and with appropriate experience, that they ‘own’ the process, and that as a group they provide consistency of judgement. Strict discipline and rigour is also needed by evaluators to ensure that individual performance and CP are not confused, otherwise the training audience will not have confidence in the process or in the evaluation itself.

Assessment method

The assessment approach was developed after discussions with the UK Staff College, operational units in Bosnia and the Battle Command Laboratories at the US War College. The method of CP evaluation used is based on the Command Estimate process and is designed to:

• Enable the judgement of the Evaluation Team to be quantified in a coherent and consistent way.
• Test how well individuals in the Battle Staff work as a team.
• To be neither ‘mission-dependent’ nor dependent on ‘mission success’.

The Estimate process is defined by the following cycle of activities:

• Assessment.
• Develop Plans.
• Choose Plan.
• Convert plans to orders.
• Issue orders

It is important to note that these tasks are undertaken irrespective of whether the phase of the exercise is deployment, execution or recovery of forces.

Essentially, the Estimate process was further broken down and a marking sheet with a six-point scoring system produced. In the scoring system 5 = Excellent and 1 = Poor. In addition a mark of 'X' was introduced where one of the serials shown below was intentionally and justifiably not performed. To illustrate this, a portion of the scoring sheet has been reproduced in Table 1.

Two further sections were added, which do not form part of the Estimate process, but allow essential aspects of good headquarters to be assessed. We name these Staff Quality1 (SQ1) and Staff Quality2 (SQ2) A section of SQ1 is reproduced in Table 2.

There are thus seven sections against which the CP is assessed. To help ensure consistency amongst the evaluators, guidance sheets for the evaluators have been produced and a sample is shown in Table 3.

Assessment results

The Output Assessment method has been used in the evaluation of training transfer during exercises PURPLE LINK (CAX) and MAUVE MARAUDER 1998 (CPX). During Exercise Purple Link the evaluators collected the CP data on a shift-by-shift basis giving a resolution of a data point about every 12 hours. Figure 7 presents the Training Transfer results.

Ex PURPLE LINK provided positive Training Transfer with the training audience being relatively more capable in the activities of a joint operational HQ after exposure to the exercise.

As mentioned above, our assessment method is independent of exercise media and the paragraphs below illustrate how, through example, the methodology can be used to compare training value of CAX, CPX and FTX. The scope of the study dictated that military judgement would provide comparative Training Transfer data for CPX and FTX (a CPX has been assessed using this method and we describe our findings below). The military judgement was acquired from sources that had relevant experience of FTX and CPX and had observed Ex PURPLE LINK. The data was collected through interviews that addressed the following:

• The shape of the FTX and CPX curves, and whether they could reasonably be expected to follow the same trends as the CAX equivalent.
• The time after which no further training benefits could be expected from FTX or CAX.
• The overall Training Transfer that could be expected from the three approaches.

The military judgement suggests that CAX can provide greater Training Transfer and greater Training Effectiveness than both CPX and FTX equivalents (Figure 8).

The Output Assessment method has subsequently been applied to Ex MAUVE MARAUDER—Figure 9 shows the results from the evaluation of that exercise. During the exercise, evaluators witnessed a smaller total Training Transfer but starting from a higher level of CP, than that assessed for Ex PURPLE LINK.

Relationship between staff CP and operational risk

We have emphasised above that the only practical way to assess CP is through the explicit use of objective judgement. Equally the effects of different levels of CP on OE can only be made through the explicit use of objective judgement and it is only combat-experienced personnel who can credibly provide this judgement. However, such judgement will only be useful if it is connected to analytical models of operations. It is therefore equally important that the combat-experienced personnel are supported by analysts who:

• understand the types and ranges of information required by senior decision-makers;
• understand how Capability can be quantified;
• are able to debate with the Military the relative importance of the various components of Capability; and
• can construct decision-support tools, which have the necessary level of detail and no more.

The process therefore consists of the following activities:

• Agreeing with the Military and the decision makers that the Capability matrices (Figures 4, 5 and 6) are acceptable.
• Establishing the kinds of Operations that the Armed Forces may be called upon to undertake.
• Establishing whether different operations lead to different requirements for the seven components of Staff CP. Our experience is that most operations are not dependent on the individual levels, so that an overall Staff CP score can be used.
• Establishing the detailed connection between overall level/levels of the CP components and the type of Operation.

We emphasise again this process can only be carried out if supported by analysts of the calibre described above.

Outputs

The outputs of this process are shown below. Firstly we connect Staff CP to overall CP, for each type of operation and for each level of Combat CP (Figure 10 and 11).

In a similar way Overall CP can be related to the other elements of Skills (Figure 6) and Skills in turn related to the other elements of People Capability (Figure 5). Clearly this represents a hierarchical set of models, from which the elements of People Capability can be traded off for each type of operation, the other two elements of Defence Capability being held constant.

Costing

Cost data for the three training methods is based on historical data gathered from exercise planners; Figure 12 summarises the costs for each training medium. Note the column labelled ‘FTX(a)’ includes the cost of deploying forces and while ‘FTX’ does not. The column labelled ‘CAX’ includes the annual cost of a dedicated facility used for one exercise a year, while ‘CAX(a)’ and CAX(b)’ are shared facility costs at a 50% and 25% share rate respectively.

Similar plots can be constructed for the other elements of People Capability, leading in turn to a hierarchical set of costing models.

Connection of people capability to operational effectiveness

In principle People Capability can be 'added' to current operational models by applying a degradation factor to each side based on their People Capability scores. However it is far better to develop a model, which includes People Capability from scratch. We know of only one such model, and that only deals with the Collective Performance at battlegroup level. Figure 13 shows typical results for three different types of operation, and it can be seen that to achieve only a 50% chance of success (OE) requires CP scores of about 4.

However having established the principles of Capability-based analysis, we are confident that models using overall People Capability are feasible.

Summary

Capability Management can only be truly achieved if we can firstly define and measure capability metrics, and secondly operate a mechanism to coherently manipulate the available resources that can impact the delivery of capability. We have shown in this paper how a Military Capability framework can be constructed that is linked to the top-level metric of Operational Effectiveness, which we have defined as, “The probability of achieving an assigned mission in a specified scenario”.

The principles of employing this framework have been illustrated in this paper by the example of delivering People Capability and in particular we have concentrated on the assessment of Staff Training in terms of Collective Performance emphasising that such evaluation can only be undertaken by the use of objective and trained military judgement, and we have shown how this can be achieved.

The causal linkage in our capability framework between the components of capability and Operational Effectiveness enables the defence planner to appreciate the impact of planning decisions in terms of operational risk, so that rational investment decision-making, that is, real capability management, can occur. In conclusion this we have shown, through example, how a capability-based investment decision culture could be made to work.

Mark Dumble has over 15 years experience with UK defence R&D both within the UK MOD and UK industry. He has been closely involved in supporting Modelling, Simulation and Synthetic Environment evaluation studies, and conducting research into validation and verification techniques for Synthetic Environments. Mark Dumble is the Director of Operations with Aegis Simulation Technologies UK Ltd.-mail: mdumble@aegissim.co.uk.

Dr Robin Miller joined the UK MOD in 1979, initially to co-ordinate Research on Military Vehicles. He then became the Technical Director of the major Army Armoured Warfare Study, Project Foresight. Seconded to DOAE in 1989, he initiated the Measurement of Defence Output that is in continuous use today by MOD Policy Branch. He then became Superintendent of the Field Studies Division, and was a leading authority on Collective Training and its assessment. During this period, he set up the field trials and conducted the analysis that led to the purchase of the Combined Arms Tactical Trainer. He was then transferred to DERA where he played a leading role in establishing the modelling and simulation requirements for supporting high-level decision making. Robin Miller is an Associate of Aegis Simulation Technologies UK Ltd.

.