A Two-Sided Contextual Technology Classification Framework For The Australian Army

Introduction

The Australian Army has a need for impact assessments of emerging technologies to determine how these technologies might impact Army’s structure and function. Early identification of new technologies with potential impact can allow the Army to prepare and position itself to either adopt a particular technology or to protect itself against its use by opponents. In order to undertake these assessments in a systematic and auditable manner, an internally consistent and robust classification framework was required to frame the technology space within the context of the Australian Army and underpin the impact assessment framework. The framework needed to be specific to the Land domain allowing coverage from strategic down to tactical levels, as that was the focus of the Australian Army and the requirement set by Army. This classification framework would avoid miscommunication and thereby improve analysis with consistent data collected from the broad range of experts required, and provide robust outcomes to Army.

Categorisations, such as this classification framework, provide common languages of communication for a given context. Their value lies in allowing different people to use a common language to ensure consistent and clear communication. Any given classification is not the only way to break down and communicate a common area but does provide a framework which can be built on over time or amended as required when new information for classification might become available. There are also weaknesses with classifications where reliance on a system becomes prevalent or new items for classification do not fit within the structures. This highlights the need for care and to construct systems which are flexible and adaptable to changing environments and which are recognised as one way to structure a topic. There are many different classifications (or taxonomies) for many different areas requiring categorisation. Indeed, the use of classification systems in the literature is as varied from the classification of biological systems [1] to the categorisation of webpage layout and design [2], to name but two.

Across both the broader military technology and more general emerging technology literature the use of researched and reported classification systems is limited. Within the military context, there is little consistency or commonality to the use of any formal classification or consistent terminologies for considering technologies, though the studies in emerging technologies are numerous and a selection are presented for discussion [3–13]. During the research presented it was found that none of the existing formal classifications or general terminologies used met the requirements (either in scope, definition or method of development) needed to underpin the technology impact assessments. They did however provide the foundation for the Delphi-like technique used to establish a classification framework of Army functions and emerging technologies for the Australian Army, allowing the assessment of emerging technologies impacts on Army functions.

Development of the framework

Across the military domain and more broadly across national-scale technology assessments, the terminology often used to represent technologies (either as a classification or as emerging technologies of importance) are reported inconsistently and with little definition. In many instances, no reference is provided on how the classification system was developed or how the language was chosen and defined for that context. Indeed, it is seen that results are reported of important emerging technologies with little or no comment on how or why the technology language was used [3–7,9,11–29]. Some may not present the terminology deduction process because of the specific nature of the context, such as the Australian Energy Technology Assessment 2012 [30]. This lack of either a classification or reporting of how the terminology was defined and developed, results in the outcomes of those studies having reduced internal consistency and lowering their broader applicability or scope for use in future contexts. Additionally, the definition of each technology term or category is of critical importance for future reference and is also often not provided. When scoping the terminologies used for formal classifications and language used in technology assessments, both the contexts of use, and ways of approaching the problem are wide and varied.

Table 1 shows two classifications used for emerging technologies in a military context and highlights the variety and inconsistency in levels of terminology used. Indeed in these examples, (as in many sources) no method or source is given to a classification system for the technology terminology used apart from an unpublished study within Defence, which briefly comments on the use of five broad technology areas to informally classify and present the technologies investigated. However, no further detail is given on how or why they were chosen and what definitions were associated with those classifications. The standard research area classifications of the Australian Bureau of Statistics [31] were also assessed for suitability but these were not considered feasible due to the detailed and unwieldy nature and covering many areas which were not relevant to the context of the Australian Army.

In four national-scale technology assessments from the literature, in particular, the Japanese 6^th [22] and 9^th [26] Delphi studies, the Korean 2^nd Delphi Study [28] and the United Kingdom Technology and Innovation Futures [23], the inconsistency of terminology used is seen clearly. In these examples, some limited discussion is given to why the terminology is used. Indeed, some of the terms were developed purely as the results of the technology assessments—without a specific language or classification being in place beforehand to ensure a common language is used. For example, In the 6^th Japanese Delphi, the term used was Energy/Resources, but in the 9^th Japanese Delphi that one category could now be mapped into at least four different categories now termed, promotion of diverse energy technology innovation; necessary resources including water, food, minerals; technologies for protecting environment and forming sustainable society; and fundamental technologies including substances, materials, nanosystems, processing, and measurements. The similar categories for the 2^nd Korean Delphi were termed energy, resources and atomic energy and for the United Kingdom program the same technologies could be categorised in energy materials and storage; energy scavenging; hydrogen economy; organic solar cells; and smart grids. These highlight the importance of the context for a particular study and the need for a clear and well defined classification or terminology of use for a particular context. With the evolution of technology assessments across changes in focus, time and language, the ability to track changes over time becomes difficult if there is no broader consistent framework allowing those changes to be monitored.

The classification framework sought for the Australian Army was required to be broad across the technology space, within the specific context of the Australian Army, robust, internally consistent and traceable. If it met these criteria then it would be flexible enough to be readily adapted to incorporate any future developments to the terminologies. This classification framework would allow iterations of technology assessments to be undertaken over time, and the commonality of language (even though specific technologies of interest will change) would ensure that these assessments can be compared against each other using a common and consistent framework. This provides insights to changes over time and also allows necessary changes to the classification framework to be tracked and documented.

There are two approaches, classification and clustering, which can be taken to group technologies, and Lee at al [32] provides a succinct summary of these approaches. In the classification approach, which Lee at al suggest is usually used for strategic purposes based on experts’ judgement, criteria, indicators or definitions are used to assess technologies and classify them. In the clustering approach, data is used to cluster technologies with similar characteristics.

Clustering was scoped as an alternative approach for this study, however, the limited availability to data sources on future technologies relevant to the Army, showed this to be infeasible. As such, the classification approach was used where the literature provided the terminologies and classifications previously used for both technologies and Army functions and these were then used as the foundation points for developing and refining the required classification framework.

For a classification framework of technologies it is critical that the technologies should be defined within their contexts of application [34], as the context of the technology applies directly to how it is classified and defined within the framework. Indeed for the classification framework of technologies for the Australian Army, the technology areas of interest are linked to their potential application to Army functions. It was determined that this information was required in parallel to that of the technologies, in a two-sided contextual classification framework.

The Australian Army has many ways to represent their functions through various related categorisations with different contexts of interest. The classification framework required a semi-independent and “generic” function set which was flexible enough to cover all the roles the Australian Army undertakes across five lines of operation and not just its primary role of joint land combat. This function set would broadly define and cover all those roles which Army’s mandate requires of it, now and into the future. After assessing each of the Army categorisations with specific applications, it was concluded that none fit the broader criteria required for this classification framework. Other classifications exist which were developed for land projects within the Australian Defence Department and international military organisations, however on assessment, they were found to be specific to a project context, or focussed on one line of operation and were also not suitable for this requirement [12,13,34]. One source did provide a reasonably generic set of core skills for the Australian Army [9], which were applicable for providing a foundation for further developing a set for this particular classification framework. This was used as the foundation for the Army functions.

The method discussed in this paper used participants with expertise in the context domains to refine a draft (literature researched) classification framework using a Delphi-like method, and is similar from the processes presented by others when eliciting expert knowledge [35–37]. Indeed the process described in this paper follows the process of determining how the information will be used; what to elicit; designing the process; performing the data collection and encoding the information. In this work the elicited information was used to develop the draft classification framework and provide sound transparent reasoning for its future application in Australian Army emerging technology impact assessments.

Principles for classification framework refinement

The technique used for the process of refinement of the classification framework followed the key principles of the Delphi technique including anonymity and iterative feedback, allowing confidence in the qualitative data collected. These principles are discussed in detail in another source [37]. The application of those principles differed in some areas here—so the technique used here is referred to as Delphi-like; however confidence in the results was maintained by applying those affected Delphi principles in the following manner.

The key elements ensuring a well structured and implemented Delphi-like process were based upon those used previously [37] in the implementation of a larger Delphi study. Internal Validity (or the credibility) was achieved in this study through measuring the improvement in the refinement to the framework using the participant feedback. Dependability was ensured through selection of invited participants from across a motivated expert group with skills across both the technology and Army function domains in a similar fashion to that of the earlier study. Auditability was ensured similarly through the detailed recording of the process used for data collection. External Validity which is defined as ensuring the findings make sense in a broader context was managed in two ways. Firstly, the data collection was controlled to ensure data was focussed and secondly, the findings at each stage were reviewed by analysts working in the domain. Ecological Validity was achieved through ensuring the questions were asked in a direct and defined manner. This was tested prior to deployment of the main survey by divisional staff not directly involved in data analysis or collection as was conducted previously [37]. Finally, in order to meet the Department requirements for research using humans and to contribute to a trustworthy and open process, the conduct of the surveys was subject to an approved ethical protocol guiding treatment of participants, delivery of the survey and subsequent analysis of data. These principles and analytical structure allowed maintenance of confidence in the expert input [36,37].

The aspects of Delphi which were not applied in this study were iteration of the survey until consensus was achieved, and statistical analyses of the data. These were both deemed to be not achievable due to expected low response rates and a small expert participant pool. These constraints to a full Delphi process were also considered inhibitors to developing the classification framework from first principles as in a previous study [37], hence the need to leverage from the literature and undertake a Delphi-like process. The data collection from the Delphi-like survey produced mostly qualitative input from the participants (in conjunction with agreement levels to questions). In order to capture and analyse this data avoiding bias, thematic analysis was used [38, 39]. This theming of the data allowed the information of interest to be captured and coded in a coherent manner. Analysing the data in this way, the refinements could each be assessed for applicability, significance, similarity and traced. The thematic analysis was concurrently and independently undertaken by two experienced analysts. These themed results were then compared for consistency and interpretation to minimise analyst bias in prospective changes to the classification framework. Improvements in consistency data, coupled with the results of the thematic analysis, was used as another gauge of correct interpretation of participant input.

Method

Following an intensive review of the literature and assessing different methods for classification development, an initial classification framework for Army Emerging Technologies was developed. The Operations Research Staff within Land Operations Division of the Defence Science and Technology Organisation were asked to provide comments. Following this process, a draft classification framework was proposed which would form the baseline from which to undertake developments on its composition, structure and definitions from a broader range of experts.

Developments to the draft classification framework were made by using a distributed anonymous online survey using the Delphi-like process, similar to a full Delphi process [38]. Using this method we could capture the required input from participants without the issue of hierarchy or “loudest voice” skewing the results [37]. This allowed participants to contribute their input at any time during the survey open times and also enabled targeting of a geographically distributed group of experts easily—saving both time and financial resources. The broad pool of experts covered the technology areas as well as the Australian Army functions.

First round of refinement

In the first round, the invited participant pool included staff from Land Operations Division and the Joint Innovation Centre from Joint Operations Division within the Australian Defence Science and Technology Organisation. This pool of participants provided the breadth and depth of both (emerging and current) science and technology knowledge coupled with Australian Army function expertise. The nature of the experts required to input to the classification framework limited our possible pool of invited participants.

142 participants were invited on an anonymous and voluntary basis, via an email invitation with a link to the online survey. Participants could access this survey any time during the survey open times and it could be completed in multiple sessions. Participants were given access to the survey by consenting to participate and were then presented with the initial “draft” classification framework. This draft classification framework presented Army Areas of Interest and Broad Fields of Study. Each term in the classification framework was given along with a definition and context of use.

For each area of the classification framework, the participants were asked if there was consistency between the categories for that level and if there were any that could be merged—on a three-point scale of yes, no and unsure, and were asked to comment on their choices. Additionally, participants were also asked to comment on whether they considered there was better terminology for the framework or if there was anything missing. Finally, the participants were asked to comment on the consistency, merging or other ideas on the classification framework as a whole.

The raw data was subjected to thematic analyses by two analysts independently. Modifications were made to the draft classification framework to create an amended framework based on those results. This new amended classification framework was then used as the basis for the next survey to undertake a further round of developments.

The thematic analysis from this round produced several focus areas where participants considered the draft classification framework required further development. These were grouped into the four areas of Terminology; New Categories; Definition and Structure; and Merge Topics.

The specific comments within each of these areas were themed to key information allowing developments to be made. Each theme was then assessed in detail, and its merit to developing the classification framework considered and the changes either implemented or documented and discarded.

Participants were questioned on the level of consistency between the categories for each level of the presented draft preliminary classification framework. The Army Functions presented were given an overall result of 48% responding yes for consistency with 35% and 17% of respondents saying no and unsure respectively. This result coupled with the thematic analysis showed that the respondents were not convinced that the presented Army functions were consistent across the levels categorised and indeed showed that many respondents wanted further consideration to other classifications used to represent Army functions compared with the list which had been presented.

The term Army Areas of Interest was re-termed the Army Generic Functions to highlight the functions which are endemic to Army undertaking all their roles regardless of the technologies available. In addition, the Broad Technology Areas were re-termed the Emerging Technology Areas (Shortened to Technology Areas) to focus on the future technology environments to which this classification framework was to be applied.

For the thematic analysis of the technologies component it was noted that there were often conflicting themed comments in the areas of: computing, networking, software, hardware, information, communications and robotics. These areas were of particular interest to participants in how they were represented and categorised and significant changes were made to the classifications to incorporate the issues raised. Many of the new areas suggested were using terms which were subsets of the levels of categories or at too low a level to be consistently categorised with the other terms.

Only 34% of respondents considered that the Technology Areas were consistent and when coupled with the thematic analysis showed that the Technology terminology presented at this stage required considerable work to make the categories more consistent and the definitions clearer. There were also several categories identified for merging. When respondents were asked to consider the overall consistency of the classification as a whole, 51% agreed that between the two levels of the contextual classification presented, there was consistency.

From these responses, the classification framework was amended to reflect the key development areas from the thematic analysis. The next version of the draft classification framework was then developed and re-presented to the participant pool for another round with the aim of improving the above consistency results and narrowing the input from the thematic analyses.

Second round of refinement

In this second round, the amended classification framework was presented back to the invited participant pool. Participants were again able to gain access to the survey by consenting to participate and then being shown the context of the application of the work and the new classification along with the revised terminologies and definitions. In this survey, participants were asked to assess if there was consistency within the levels of the classification framework and if there was anything critical missing on a three-point scale of yes, no, unsure along with comments on their choices. Finally they were asked to comment on the framework as a whole and provide input if they felt further developments were needed.

For this round, 165 participants were invited via an email invitation. The expanded invited participant pool included those from round 1 as well as the Force Development Group of the Australian Army at the Land Warfare Development Centre. The participants in this round were not necessarily the same participants in round 1 as the anonymity and low response rates made it likely some participants were different. However, this was not considered to be an issue as the breadth in response would lead to a more robust outcome. This issue has been encountered previously and was not found to significantly impact the outcome [38].

At the start of the process it was accepted that a consensus would not be achieved across the breadth of the participant pool. The investigators aim was to achieve a point where improvements in both consistency and thematic changes would indicate that further surveys would not provide significant developments to the classification framework (saturation). In this round, improved consistency responses were obtained for both the Army functions and the emerging technology areas. The same analysts and analytical procedure from the previous round were reapplied here.

Overall, 76% of respondents agreed that there was consistency between the categories of the Army Functions presented. From the thematic analysis, the greatest areas of concern among participants were the clarity of the definitions of the terms where other roles fit within those functions. It was highlighted that the definitions and terminology of the functions should ensure that they clearly encompassed all the roles and not just joint land combat, which the Australian Army is to undertake now and into the future. The breadth and scope of the focus areas from the thematic analysis were greatly reduced from the previous round and consisted of definitions. Coupled with the improvement in consistency, this showed that the developments made to the Army Functions part of the classification framework had made significant improvements in the opinion of the participants.

For the Technology areas, the thematic analysis also showed fewer focus areas consisting of Definitions and Structural changes of like technologies into groups. For both the Army Functions and the Technology areas there was greater correlation of the themes which presented fewer specific areas to develop. Indeed the themes arising from this round were considered to be “tweaking” the classifications, as opposed to the previous round which resulted in significant changes to both its structure and composition.

For the technology areas, 67% agreed that there was consistency and the themes of greatest interest to participants were protection technologies; resolving the categories of automation, robotics, ICT and training and simulation; questioning the need and definitions of basic sciences and new paradigms; and the resolution of the terms for mobility and propulsion within the transportation term. This improvement indicated that the language and definitions used were now progressing to be internally consistent between the categories. Further refinements from the thematic analysis of this round provided further specific improvements which were not tested in further rounds.

Thematic analysis showed that the items considered to be missing from the framework were not consistent as categories, and were clearly subsets of the previously presented levels or were considered enabling technologies. This indicated that critical items were now being captured or had definitions and categories in place which would capture those terms.

With the overall improvements in consistency results and the reduction in the scope and breadth of the themes for development, it was determined that further significant refinements would not be achieved by further rounds of surveying. It was found that the US Joint Capability Areas [40] provided Army function terminology which was consistent with those developed by the participants. This terminology was then mapped and adapted to that identified by the participants to arrive at a set of Army Capability Functions which were consistent with the participant input, applicable in a joint domain and clear to the Army. These final developments to the classification framework produced the two-sided contextual classification framework for emerging technology impact assessments for the Australian Army.

Final classification framework

The final two-sided contextual classification framework for Emerging Technology Impact Assessments for the Australian Army with associated definitions is defined below.

Army Capability Functions (adapted from [40])

Protection: includes prevent, mitigate, immediate security to population to allow re-establishment of law and order.

Force Application (manoeuvre/engagement): manoeuvre (Army and populations); engagement (includes Joint Land combat, information operations, cyber operations, psychological operations, EW).

Battlespace Awareness: includes ISR, fight for information and on local population; environment.

Information and Communications: includes information transport; enterprise services; network management; information assurance and security.

C2: includes organise, understand, planning, decide, direct, monitor.

Logistics: deployment and distribution; supply; maintain; logistics services; operational contract support; engineering; installations support.

Force Support: force management; force preparation; human capital management; health readiness and battlefield support.

Building Partnerships: includes communicate (allies and populations); shape; inform, shape perceptions, attitudes, behaviour and understanding of populations.

Corporate Management and Support: includes advisory and compliance; strategy and assessment; information management; acquisition; program, budget and finance.

Emerging Technology Areas

Power Technologies: includes all those technologies with the capacity provide (or contribute to the capacity to provide) power and energy (including storage and distribution) to systems.

Information Technologies: broad term covering both contemporary devices and software which enables users to create, access, store, manipulate and manage information.

Robotics and Autonomous Systems: technologies and algorithms allowing machines to operate remotely with a spectrum of independence from human control, ranging from total dependency on the human to complete self sufficiency.

Weapon Technologies and Methods: all technologies and methods which contribute to and form part of systems or processes which have the potential to act as a weapon (includes lethal, less than lethal, cyber-attack, electronic warfare, information warfare, psychological, and biological attack)

Sensor Technologies: all forms of sensor technologies including position, targeting, navigation, detection, chemical, biological etc.

Health Technologies: all aspects relating to health and wellbeing of Army personnel and those they are protecting on operations.

Transportation Technologies: includes the technologies and principles (and those which contribute to) Army’s transportation needs across the spectrum of operations.

Materials and Manufacturing Technologies: all materials which impact or give advantage to Army (or an adversary) as well as developments in manufacturing methods.

Protection Technologies: includes materials, methods, weapons for protection of Army capability and those they are supporting on operations.

Training Technologies: includes the technologies and methods developed to improve the training and learning outcomes of ADF personnel.

Mapping army functions and technology areas

As the classification framework is two-sided, it is possible to represent the relationships between the Army capability functions and the emerging technology categories visually. This representation of the possible areas of impact of the Technology areas on the Army functions is shown in Table 2. The black shading indicates where it is highly likely that advances will have opportunities to impact. Those with grey colour may or may not have opportunity to impact. Those with no colour show where there is not likely to be direct impact from the technology areas on the Army functions. This allows early observation on the possible key relationships between Army functions and potential technological areas prior to detailed technology impact assessments being undertaken. The value of this is the reduction in the scope of the detailed impact assessments required initially—saving resources and time.

For example, the technology areas of Information Technologies and Sensor Technologies were found to have a potential to impact across all areas of the Capability Functions meaning that the scope of work will be considerably greater with greater potential impact across multiple areas conducted concurrently.

The mapping shown was found to provide useful insights by providing likely areas of interest and focus for the impact assessments. The client was able to visually and quickly consider and determine the areas of priority for impact assessment.

On its own, the classification framework provides a consistent and robust set of categories providing the backbone of assessment of the possible impact of new technologies on the Australian Army and its functions. As the Army capability functions are derived from a joint domain, both the function terms and the technology areas are readily applicable to other military domains.

The analysis of the technology terms, showed that using discipline-like or enabling technology terms for the technology classifications—such as nanotechnology—were not useful when considered in terms of the impact assessments this classification framework would underpin. Terms which had this broad or enabling applicability were found to not allow sensible categorisation and could not be clearly defined. They were best broken down into specific technologies of application which would then fit into the technology application areas. This had the additional benefit of allowing better consistency across the terms used. As a result it became necessary to be able to classify emerging technologies into categories which would focus the application to Army.

Across the technology space there will be some overlap in larger emerging technology systems across the technology categories or there may be individual technology elements which have the potential to provide technological advancements in multiple categories. This was not considered to be an obstacle as the application of the classification framework to the impact analysis allows multiple labels to be applied to both technologies and systems. For the purposes of classification and comparison with Army functions, multiple labels provides the ability to track various courses of development and complexity in the system and also shows the breadth of potential impact from specific technologies or areas.

A database of outcomes from the larger study of impacts of technology on the Australian Army functions, will use the classification framework developed here as the labelling and categorising system for tracking and storing information retaining the consistency across the entire spectrum of work.

Conclusion

The development of this two-sided contextual classification framework for the Australian Army was motivated by the need for a consistent and well defined terminology and framework for future emerging technology impact assessments. The need for a specific context and the lack of consistency in both military and general technology classifications resulted in the need for this study. An open and auditable method for developing classification frameworks for technologies and their contexts of application, which has not previously been documented, has been detailed. The classification framework developed has been presented. This contrasts with the lack of method presented with many works presenting terminologies for technologies of interest, or work showing classifications of technologies. This classification framework is used with the Futures branch of the Australian Army in communicating and reporting impact assessments of emerging technologies on Army functions. The scope and use of this contextual classification framework is shown as the underpinning language of a larger study. The limitations of a classification framework are addressed through recognition of this being one way to structure the problem space and by the framework being designed such that it can be adapted and evolved over time to meet changing needs while still tracking the origin on the information. The potential application of this method of classification framework development to other domains or within other Military contexts is highlighted.

Acknowledgements

The author would like to thank Dr Brandon Pincombe for both his support and many discussions and ideas in the development and planning of this project. Many thanks also to all the participants from across DSTO and the Australian Army who contributed to the successful development of this classification framework.

References

Author

Patricia Dexter has a background in physical chemistry and spectroscopy. She has been involved in operations analysis of Land Operations for 14 years in a wide range of areas. She has a current focus on the impact of emerging technologies for the Australian Army. She has expertise in Land futures scoping and analysis.

patricia.dexter@dsto.defence.gov.au