Automatic Speaker Identification in C2 Centres: Challenges and Pitfalls

Introduction

In text-independent automatic speaker recognition (ASR), a computerized system identifies an individual by analysis of his/her speech. ASR comprises both verification (ASV) and identification (ASI). In ASV, an identity is claimed and the system must attempt to authenticate that claim. In ASI, the claimant is known to belong to a well-defined set of identities (closed set) or is suspected of belonging to that set, without this being known with certainty (open set). One of these technologies may be relevant to command and control (C2) centres where there is a need to label recorded utterances during conferences, briefings or planning sessions, for dissemination or archival purposes. This paper exposes the main technical hurdles which need to be overcome or adequately managed should ASI be successfully exploited in such environments.

An idealized ASI scenario

Those C2 centres considered here are medium to large in-door venues as opposed to automated mobile command posts. The centres are frequented by clerical staff and members of a well-defined pool of military decision makers. These regular staff include a few dozen people, at most. In addition, an indeterminate number of people visit the centres infrequently. Decisions are taken collectively and orders dispatched. Speech-recognition technology transcribes utterances to text and ASI is needed to label that text with speaker identities. An open-set ASI system is likely to be the best candidate for most C2 environments. Indeed the system would be trained to recognize the core regular staff. It would be tested for that core but also for infrequent visitors. The system would bunch the latter together under a single class ‘other’. Ideally, the different C2 actors walk around within an environment free of microphone headsets or throat microphones. The wideband signal is captured by an unobtrusive omni-directional microphone placed at a fixed point inside the centre.

ASI system overview

Figure 1 gives an overview of an ASI system tailored for a C2 centre. The speech signal is reduced to vectors of parameters in the front end. Those vectors are classified as belonging or not to a speaker, in the back-end. The signal is digitised and segmented. Silent as well as low-energy parts are automatically removed. (The signal may also, optionally, be high-frequency pre-emphasised to compensate for glottal and lip-radiation effects.) After a hamming windowing stage, vectors of cepstra are computed for each speaker.

The m^th cepstrum is given by:

$c_m={\theta }_c{\sum }_{j=0}^{J-1}\text{cos}\left(m\frac{\Pi }{J}\right(j+0\text{.}5\left)\right){\text{log}}_{\text{10}}{}_{}$ (1)

where θ_c respects the dynamic range of the c_m, and E_j is the energy in each channel associated with the J triangular filters used.

• The cepstrum is usually warped to a mel-scale (mel-cepstrum) [1]. The new parameter contains only static information. Vectors can be extended by the addition of dynamic information. This latter information is provided by the cepstrum’s delta. Many formulas exist for computing the delta parameter. The standard m^th cepstrum’s delta is given by [2]:

$d_t=\frac{{\sum }_{i=1}^{I}i(c_{m,t+i}-c_{m,t-i})}{2{\sum }_{i=1}^I{i}^2}$ (2)

where c_m,t+i is the static m^th cepstrum at time t+i, and i is the size of the window.

• The ASI system produces a model for each of the core staff using connectionist [3] or Gaussian Mixture Model (GMM) technology [4]. During the operational phase, an incoming signal is labelled with the speaker model which best represents it statistically, provided a threshold measure is exceeded. Otherwise the signal is labelled ‘other’. Such labels are then applied to the corresponding speech-to-text transcriptions.

Inherent limitations of ASI technologies

ASI systems need to be trained to recognize people of interest (core C2 staff, in the present case). (The process may be alleviated but not without some loss of recognition accuracy [5].) This pre-supposes that the core staff are cooperative and available to undertake at least one training session although several would be beneficial. These sessions are best undertaken within the specific C2 centre where ASI is scheduled to be used because of room acoustic considerations detailed in following sections. The training sessions should include at least one minute of quality conversational speech [4]. The ASI system performs best on long utterances (at least 5 seconds in length [4]). However, under operational conditions, conversations are peppered with shorter comments as well as interjections.

Speakers using an ASI system may not stay at a fixed location and/or close to a microphone. The energy that characterizes the signal may be too low or fluctuate within too broad a range.

The design of speech technologies has long been plagued by the fact that the distribution of the speech signal is, itself, unknown. ASI systems are not able to recognize a small percentage of speakers nicknamed “goats” nearly as well as the general population of speakers (“sheep”) [6] for reasons that are not fully understood. A means of identifying goats, prior to ASI being conducted, has not yet been found. Such a means would be of little use here, anyway, since the content of the speaker set is not negotiable.

Signal degradation due to environmental factors and the use of a single microphone

Reverberation

An acoustic signal captured within an enclosure (such as a C2 centre) is reverberant. The signal is comprised of a direct speaker-to-microphone component as well as an infinite number of other components which are partly absorbed in, and reflected off, walls, floor and ceiling. This phenomenon is akin to that of multi-path returns in radar. The reverberation problem is too frequently underestimated or overlooked altogether in speech problems. This is because experimentalists, like most humans, have an auditory system able to spontaneously de-reverberate all but the heaviest reverberation. However, computers do not have that capability. Consequently, signal reverberation will represent one of the prime causes of ASI degradation in C2 environments where meetings are held indoors and commonly in highly reflective media (glass windows or walls).

An acoustically reverberated speech signal x(n) can be expressed mathematically (time domain) as the convolution of a clean signal s(n) with the impulse response of the enclosure p(n) as:

$x\left(n\right)=s\left(n\right)\ast p$ (3)

The enclosure impulse response is described by the transmission properties between source and receiver contained in the enclosure (room). The enclosure may be modelled acoustically as a linear system whose characteristics are dependent on enclosure volume (and therefore size), wall surface reflection and physical content of the enclosure. Small rooms and high wall surface reflection coefficients result in reverberation times of several seconds with dense impulse responses [7]. The acoustic system also possesses characteristics which vary between and during measurements [8]. These variations are caused by changes in source (head) or receiver orientation and source and receiver positions, relative to one another in the enclosure. The fundamental obstacle which needs to be overcome, in optimally solving reverberation problems, is estimating accurately an enclosure’s impulse response, free of experimental constraints. Allen and Berkley [9] propose a method commonly used to solve a simpler problem: the simulation of acoustical properties associated with small, rectangular and empty, enclosures. The speaker-to-receiver impulse response is deduced from an image method in the time domain. A speaker is modelled as a point source, in the rectangular enclosure. The signal emitted is a pressure wave. Assuming a rigid wall, the associated boundary condition is satisfied by positioning an image symmetrically behind the wall. The image is, itself, a point source and therefore requires imaging. In the case of a six-wall system, the enclosure impulse response to the initial pressure wave is contributed to by an infinite number of images. Walls are non rigid in practice, resulting in wall surface reflection coefficients of less than 1 in value. A reflection coefficient is taken to be constant across the entire wall surface and independent of the angle of incidence of the pressure wave. Under these conditions, the enclosure impulse response is given by:

$p(t,X,X^{\text{'}})={\sum }_{p=0}^1{\sum }_{r=-\infty }^{+\infty }{\beta }_{x1}^{|n-q|}{\beta }_{x2}^{\left|n\right|}{\beta }_{y1}^{|l-j|}{\beta }_{y2}^{\left|l\right|}{\beta }_{z1}^{|m-k|}{\beta }_{z2}^{\left|m\right|}\cdot \frac{\delta [t-|R_p+R_r|/c]}{4\Pi |R_p+R_r|}$ (4)

where:

t: time;

p: is (q,j,k) (point coordinates in a three-dimensional lattice);

r: is (n,l,m) (point coordinates in a three-dimensional lattice),

X: is speaker location in a three-dimensional space (x,y,z);

X^’: microphone location in three-dimensional space (x’,y’,z’ );

R_p: (x–x’+2qx’, y–y’+2jy’, z–z’+2kz’);

R_r: is 2(nL_x,lL_y,mL_z), ((L_x,L_y,L_z) being enclosure dimensions),

C: is the speed of sound at sea level,

β_k: is (1-α_k)^1/2 (α being the Sabine energy absorption coefficient for wall k, k $\in$ [1..6]); and

δ ( ): is the delta function.

Allen and Berkley’s method allows for the above-listed range of parameters to be accurately controlled but is still unable to precisely simulate the reverberation characteristics of furnished rooms of arbitrary shape. Real C2 centres are bounded by walls across which reflectivity varies greatly since walls may host screen-display systems. Furthermore, the centres comprise furniture (ranging from a few chairs to consoles) as well as people. Both contribute to the impulse response and do so in a matter that changes constantly. (The introduction or removal of an item of furniture or laptop, the positioning of a chair or, more dramatically, the change in orientation of a speaker’s head will instantaneously impact on that response.) Previous studies [10,11] have confirmed that ASI system accuracy is very poor when the system is trained in an anechoic (non-reverberant) environment and tested in a reverberant one. This is so for medium and long reverberation times (greater than 0.2 seconds) and a wide range of β_k. Accuracy degrades very rapidly with speaker-to-microphone distance and room size. This is the case, independently of the chosen signal parameterisation scheme and even when other experimental parameters are constant for the duration of the experiment. However, ASI accuracy can be somewhat improved by repeating the experiment but convolving the training signal with a constant impulse response captured at the centre of the room (provided speakers and microphone are static) [10].

Noise, the cocktail-party and lombard effects

The addition of non-periodic noise (such as white noise) to speech increases intra-speaker variability significantly. This has a negative effect on ASR [12]. However, it has been recognized that low signal-to-noise ratios (SNRs) affect speech intelligibility more than ASR [13]. Attili et al [14] put forward an ASR system which, it is claimed, is not degraded by white noise down to 15 dB SNR. The system makes no attempt to either model or remove that noise from the signal. Other authors [15,12] believe, on the contrary, that noise is a serious obstacle to ASR accuracy. Noise appears to degrade the cepstrum’s delta more than the cepstrum itself, rendering the addition of dynamic information counterproductive in some ASR applications. Speech enhancement has been used in speech processing for a long time [16]. While improvements in SNRs have been made, the big issue in speech enhancement has traditionally been a quest for improved intelligibility [17]. However one cannot assume that a gain in speech intelligibility, through enhancement, translates into an improvement in ASR accuracy. Speaker characteristics may have been lost in the enhancement. Furthermore, speaker recognisability (defined as a machine’s ability to recognize a particular speaker) cannot be easily predicted from speech intelligibility testing although a correlation exists between bandwidth and recognizability [18].

In locations occupied by several speakers, conversations overlap so that a speech signal is effectively degraded by others. This is known as the cocktail-party effect [19]. This has a negative effect on ASR since the signal retained for analysis originates from several sources and, as such, is not highly characteristic of any one speaker. (A related problem is that of segregating speakers whose signals are intermixed without actually overlapping, as is the case in radio communications.) Methods exist for separating utterances spoken by several speakers, without a priori knowledge of any of those speakers [20], but these would struggle in a C2 environment.

The Lombard effect characterises the signal of a speaker attempting to communicate in a noisy environment [21]. In doing so, he/she tends to communicate in a louder voice than normal. That loudness is a function of ambient noise. The stressed voice is accompanied by an articulation variability. Both degrade ASR accuracies. Both noise and the Lombard effect could be significant within the pressure cooker which an operating C2 centre frequently is during military operations. The resulting noisy-Lombard speech has a spectrum given by (frequency domain):

$S_{\text{nl}}\left(\omega \right)=G\text{.}A\left(\omega \right)S\left(F\right(\omega \left)\right)+N\left(\omega \right)$ (5)

where:

S(ω) is the spectrum of clean speech;

F(ω) is a non-linear frequency warping (due to articulation variability);

A(ω) expresses amplitude scaling (due to articulation variability);

N(ω) is the spectrum of additive noise (due to ambient noise); and

G is an intensity variation factor (due to loudness).

Signal degradation and microphone arrays

Lin and Flanagan [22] have concluded that 2-D matched-filter microphone arrays are capable of enabling high ASI when acoustics are degraded by a number of factors including reverberation. Their study reported on a simulation using Allen and Berkley’s imaging technique to manufacture a reverberant signal, for large empty rooms and immobile speakers. A matched-filter is the time inverse of the impulse response of the enclosure of interest. One such filter is computed for each microphone in an array. Two microphone arrays are sufficient, provided they are orthogonally placed. This technique is promising in terms of noise suppression as well. Future work should begin by repeating their study in a C2 environment, involving furnished rooms and mobile speakers (including head movements). Studying the impact of reverberation on the delta-cepstrum would represent part of that study.

Conclusion

A number of technical pitfalls plague the introduction of ASI systems into C2 centres where staff are unburdened by close talking microphones (headsets). Such pitfalls may be inherent to room acoustics (reverberation and non-periodic background noise), speaker-to-microphone distance (low energy frames), speaker interaction (cocktail-party and Lombard effects), the nature of dialogue (which contains a percentage of short sentences) and the inability to identify a small minority of speakers (goats) even under ideal conditions. Each such pitfall has the potential to render ASI inoperable (and will adversely affect speech recognition as well). It is unlikely that these difficulties will ever be surmounted, using a single microphone. This is because a single channel does not convey enough information to distinguish between a pure speech signal from a single source and the elements that contaminate that signal. Furthermore, little can be done to address the problem of transient C2 staff (visitors) for which the ASI system will not be trained.

The use of microphone arrays, including beamforming technology, offers the hope of lessening the effects of most signal related problems (reverberation, noise, low energy frames and the cocktail-party effect). Beamforming microphone arrays are the focus of present research. This research will be pursued well into the next decade. However, this technology cannot completely solve the above-mentioned problems because the acoustical properties that characterise the problems are dynamic in nature, especially since staff are assumed to be mobile and head movements unconstrained. These properties change instantaneously and one of DSP’s great challenges is to build processors that operate closer and closer to real-time (although a truly real-time processor is, of course, not possible.)

One must not discount ASI, as a practical solution for C2 speech-to-text labelling, outright, despite the host of problems mentioned above. There are several reasons for this. Firstly, the performance of an ASI system is inversely proportional to the number of speakers modelled. Secondly, reverberation is proportional to enclosure dimensions. Consequently, ASI performance will be superior in large, sparsely populated C2 centres. ASI performance may be further improved by including anechoic features into the centres’ designs and using dense microphone arrays. Of all challenges, these are probably the easiest to meet because technological costs are coming down. Thirdly, there may be some room for compromise between operational constraints and ASI performance, in low security C2 centres. Equipping staff with wireless throat microphones would be relatively unobtrusive. Such microphones are already cheap and offer substantial protection against room acoustics effects. A study into how these particular microphones affect ASI through spectral filtering of the speech signal is warranted.

References

Author

Peter Chatelain is a Defence Worker in the Command and Control Division of the Australian Defence Science and Technology Organisation (DSTO), South Australia. He holds a doctorate in EE&E (DSP—Automatic Speaker Recognition), has worked with several police organisations and published close to 30 articles and reports on the topic.