The present invention generally relates to speech synthesis technology.

In most telecommunication systems, speech signals are not transmitted with their full analog bandwidth in order to use the available transmission channel more efficiently. For telephone communication, the signal bandwidth is typically limited to less than 4 kHz, even though there are signal components up to 8 kHz and higher in the original speech signal. This band limitation has little or no effect on the intelligibility of voiced sounds, but fricatives such as /s/, /sh/, /ch/, /z/ or /f/ may be lost completely. In Fig. 1 a spectrogram of the utterance "Sauerkraut is served" is depicted. The phoneme /s/ is spoken at approximately 0, 0.8, and 1.4 seconds. At these times almost 100% of the signal energy is above 4 kHz. With a bandwidth limited to less than 4 kHz the speech intelligibility may still be sufficient because the listener is often able to predict missing phonemes from syntax and the context of what is being said. However, errors arise if such a prediction is not possible, e.g., because names or unknown words of a foreign language are transmitted. Furthermore, the phoneme /s/ is important in the English language to indicate the plural and possessive pronouns.

Various spectral compression schemes exist that aim to present signals from one frequency range in another more useful range of the frequency spectrum. The side-effects of the schemes vary according to the limitations of the candidate identification techniques, artifacts resulting from the signal processing, as well as high computational complexity and delay in the signal analysis stage.

P. Patrick, R. Steele, and C. Xydeas, Frequency compression of 7.6 kHz speech into 3.3 kHz bandwidth, 31 (5):692-701, May 1983 describes a system for retaining information under frequency compression. This system consists of frequency mapping at the transmitter and demapping at the receiver side. According to a frequency compression factor c, every cth sample is retained in the compressed magnitude spectrum, so that it occupies only the frequency range that is available for transmission. At the receiver side, the received frequency components are spaced out to their correct locations and the magnitude of the missing components is found by linear interpolation. The phase is chosen randomly.

The compression factor c can also be set frequency dependently to give higher or lower compression in certain regions. Three different mapping laws, which are tailored for certain phonemes, and one that simply corresponds to a band limitation, are used in the system. For switching between these laws, the signal is first classified as voiced or unvoiced speech based on its autocorrelation. Mapping is only applied for unvoiced speech. Then the mapping and demapping procedure is applied according to the same laws.

It is important to note that this system needs to transmit side information about the mapping laws that have been used to ensure correct demapping. Additionally, the receiver must be able to handle this side information and to perform the demapping.

D. A. Heide and G. S. Kang, Speech enhancement for bandlimited speech. In Proc. IEEE International Conference on Acoustics, Speech and Signal Processing, volume 1, pages 393-396, May 12-15, 1998 describes a method that works without transmitting any side information. A classification of the speech sound is based on the spectral centroid of the spectrum, f_c. For voiced speech, f_c ≈ 2 kHz usually holds.

When the spectral centroid is greater, portions of the spectrum above 4 kHz are added into the 3 - 4 kHz bands according to the following rules:

• fc = 2 kHz: No spectral translation is performed.
• fc > 3 kHz: The 4 - 5 kHz band is added into the 3 - 4 kHz band.
• fc > 4 kHz: The 5 - 6 kHz band is also added into the 3 - 4 kHz band.
• fc > 5 kHz: The 6 - 7 kHz band is also added into the 3 - 4 kHz band.

Uncontrolled artifacts are resulting from this signal processing. There is no general improvement of the speech intelligibility.

Another application area of frequency mapping methods is hearing aids. People with hearing impairment usually suffer from a bad perception of high frequency sounds. The traditional approach is to strongly amplify these critical frequency regions. However, for some people, hearing sensitivity is so poor at high frequencies that sufficient gain for achieving audibility cannot be provided.

Andrea Simpson, Adam A. Hersbach, and Hugh J. McDermott, Improvements in speech perception with an experimental nonlinear frequency compression hearing device, International Journal of Audiology, 44:5:281-292, 2006 discloses further method of frequency mapping. The methods don't improve sound quality but enable the hearing-impaired to receive at least some information contained in the high frequency speech components.

Some further approaches are:

• to shift the complete spectrum to lower frequencies if high frequency sounds are detected (H. J. McDermott, V. P. Dorkos, M. R. Dean, and T. Y. Ching. Improvements in speech perception with use of the avr transonic frequency-transposing hearing aid. J Speech Hear Lang Res, 42(6):1323-1335, 1999). Problems with this scheme are the reliable detection of high frequency signals under noisy conditions and artifacts that occur during the on/off switching.
• to detect a range of high energy and to shift that region downwards (Kuk, Korhonen, Peeters, Jessen, and Andersen. Linear frequency transposition: Extending the audibility of high frequency information. The Hearing Review, 2006). Shifting means here, to overlap the identified frequency interval with a lower band and to add the two spectra.

This can lead to blurring of vowel sounds.

• to compress frequencies above a threshold frequency (described above). The compression is always switched on, regardless of the current speech sound.

US 5 771 299 aims to compress or expand the spectral envelope of an input signal. The goal is to move signal information from a region of the spectrum that is outside of the audible range of hearing aid users into another range that is still audible for the user. The system uses an LPC analysis filter and transmits the filter coefficients to the synthesis filter directly. Then, with the help of all-pass filters non-integer delays are introduced in the analysis and/or the synthesis filters. Delays larger than 1 compress the spectral envelope while delays smaller than 1 expand the spectral envelope.

The main problem with this system is that the voice specific characteristics such as formant positions are not preserved, since they are also compressed/expanded as part of the signal processing. This has the effect of enhancing audibility for the hearing impaired, but not of preserving the audio quality of the original input speech signal. Further disadvantages are delays of the output signal.

US 2009 0074197 discloses a similar transposition but with user-dependent information for spatial hearing. The goal is to perform a frequency transposition that moves frequency regions of the input signal to user specific ranges that are measured using built-in mechanisms of the hearing aid.

A method according to EP 1 333 700 A2 applies a perception-based nonlinear transposition function to the input spectrum. The same transposition function is applied over the whole signal so that artifacts resulting from switching between non-transposition and transposition processing are avoided. However, the use of a single function over the entire speech signal ignores time-varying phoneme-specific characteristics in the signal. Also, transforming the input signal to and then from the perception based scale to apply the transposition function reduces spectral resolution. As a result, characteristics of parts of the spectrum, which would otherwise only be subjected to a linear section of the transposition function, are not accurately preserved.

US 2006 0226016 aims to correct the phase of a transpositioned spectrum, wherein the transposition system is always active. Such processing of voiced segments has a negative effect on the phase and harmonic characteristics of the signal.

According to US 2009 0226016 the input signal is subjected to a high pass (or band pass) filter, after which the spectral envelope of the high pass signal is estimated using an all-pole model. A warping function is then applied to the all-pole model, translating the poles to lower frequencies using both non-linear and linear warping factors. An excitation signal (the prediction error signal) is then applied to the newly transposed all-pole model and shaped according to this transposed spectral envelope to create a synthesized transposed signal. The synthesized signal with an optional amplification and the original low-pass signal are then summed to create a signal containing compressed and transposed segments of the original spectrum. The LPC analysis and voice synthesis step is costly and the quality of such signals usually sounds quite artificial without proper postprocessing. This postprocessing is also expensive and this method is therefore not suitable for a system focusing on improved audio quality compared to systems for the hearing-impaired.

US 6 577 739 describes a proportional compression factor to the frequency spectrum generated by an FFT of the input signal. The factors are set to be between 0.5 and 0.99, and are applied using different methods. Using linear interpolation and assuming a compression factor of 0.5, the contribution of two input FFT bins are calculated and applied to one output IFFT pin. Another method would use an IFFT length double that of the input FFT length. By additionally placing the contributions from the input FFT bins in a higher or lower position in the output IFFT vector, a frequency transposition can be applied as well. This method again places a compressed frequency range of an input signal in another range in the output signal, which is still available to the communication channel or the end user of a hearing aid device. This system is however quite limited in scope by requiring FFT processing of the input and output signals. The additional time domain trimming of signals required by using FFT and IFFT vectors of different lengths also presents problems for time-domain synchronization and variability of the compression factors is not easily implemented.

CA 2 569 221 describes an attempt to retain information in frequency ranges outside of a band pass threshold. The input signal is converted to the frequency domain via the FFT algorithm or a polyphase filterbank. A compression function is then applied. Additionally, an amplification is applied in accordance to the energy level of the uncompressed portion of the input signal. The system can also comprise sending the compressed signal to an automatic speech recognition module. This method can have negative compression effects by giving portions of the input spectrum either too much or too little emphasis.

WO 2008/0123886 first defines a pass band for the output signal, and then defines a threshold, generally lower than the pass band, above which the frequency compression is applied. In the frequency region above the threshold, the highest frequency still of interest is identified and the appropriate compression is then applied. After the compression, the new peak power is normalized in a manner proportional to the compression, or is simply halved by -3 dB. Additionally, this method provides for the ability to expand a received signal, compressed or otherwise and synthetically reconstruct high frequency portions of the signal that may or may not have been present in the original transmitted signal. This can have a negative effect on the audio quality of the signal. This method still neglects the time-varying speaker dependent characteristics of speech sounds and applies the same compression function to each frame.

In view of the foregoing, the need exists for improved solutions that preserve and enhance voice specific characteristics. The object of the present invention is to improve at least one out of controllability, precision, signal quality, processing load, and computational complexity.

The inventive method for adaptive spectral transformation for acoustic speech signals comprises the steps of
receiving at least one spectral input representation corresponding to at least one window of a time domain input signal of acoustic speech,
selecting of the spectral input representations at least one selected spectral representation to be transformed,
assigning the at least one selected spectral representation to one of a set of cluster centres, wherein
the cluster centres are defined on the bases of spectral representations of windowed acoustic speech segments of a speech corpus by a clustering algorithm,
spectral class representations are assigned to the cluster centres and are elements of a code book and
the code book links to each spectral class representation at least one spectral transformation which enhances the corresponding spectral class representation,
transforming each selected spectral representation to a spectral output representation, wherein the applied transformation corresponds to the at least one spectral transformation linked to the cluster centre which is assigned to the respective selected spectral representation, and
providing the spectral output representations to synthesize an acoustic speech signal.

Selected spectral representations are classified by assigning them to spectral class representation. The spectral transformation applied to a selected spectral representation is adapted to this selected spectral representation because the applied spectral transformation enhances the assigned spectral class representation. The adaptations to enhance the spectral class representations are made in a setup procedure and include heuristic steps in order to find transformations which enhance the spectral class representations. The adapted advantageous transformations for each cluster centre respectively for each class of the code book ensure enhancement of the spectral representation of all the spectral representations assigned to this class of the code book. The controllability, precision and signal quality are enhanced while the processing load and computational complexity are reduced.

Best results can easily be found by the use of an appropriate code book, respectively by selecting an appropriate speech corpus and an appropriate clustering algorithm. Clustering of spectral representation with sufficient energy in the 4 to 8 kHz band allows specific enhancement of fricatives such as /s/, /sh/, /ch/, /z/ or /f/. The step of finding transformations which enhance fricatives of different classes allows linking of very specific transformations to the classes of the code book.

An English speech corpus can for example be taken from the TIMIT acoustic-phonetic continuous speech data base. This speech data base has been designed by the Massachusetts Institute of Technology (MIT), SRI International (SRI) and Texas Instruments, Inc. (TI) for acoustic-phonetic studies. It contains utterances of 630 male and female speakers in American English, divided into eight dialect regions. From each person, ten phonetically rich sentences have been recorded and tagged with phonetic information. The training data can be reduced to 70 speakers from all dialect regions and pauses can been removed with the help of phoneme tags. The reduced training data has a duration of approximately 23 minutes. For validation purposes, a second data set can been extracted that consists for example of 10 minutes speech data from 30 speakers. The sampling frequency is preferably fs = 16 kHz.

There are two different kinds of preferred transformations, frequency compression and formant boosting. Frequency compression is compressing the bandwidth for example from a bandwidth of 0 to 8 kHz to an bandwidth of 0 to 4 kHz preferably with a compression only at the upper or lower end of the bandwidth, optionally linear at least in the middle frequency range, corresponding to no compression when the slope is equal to 1. Formant boosting takes place using formant-dependent functions to increase the contrast between the formants and the non-formant frequencies in the frequency spectrum. The formant boosting gain function linked to each spectral class representation of the code book is amplifying at least one preferably two or three of the formants at low frequencies.

Assigning the at least one selected spectral representation to one of a set of cluster centres includes
calculating distance measures between the selected spectral representation and all the
spectral class representations of the code book, and
assigning the at least one selected spectral representation to the cluster centre with the shortest distance measures between the selected spectral representation and the spectral class representations of the cluster centre.

Calculating distance measures becomes very simple by first calculating feature vectors to the spectral representations. The distance measures are just distances between the feature vectors. The feature vectors are calculated from spectral envelope representations by a filtering transformation, preferably with a mel-filterbank, wherein the mel-filterbank optionally uses overlapping triangular windows with widths variable with frequency. With such feature vectors it can be sufficient to use a code book including at least eight (for example for fricative enhancement), preferably thirty-two, optionally 128 cluster centres. With higher class numbers more different enhancement problems can be solved each in a different way by the linked at least one spectral transformations.

The spectral class representations for the cluster centres are preferably averaged spectral representations averaged over spectral representations of corresponding cluster elements. A reduction to spectral class representations of special interest can be made by applying the clustering algorithm on the bases of a preselected sub-corpus of the speech corpus. The sub-corpus can for example be reduced to spectral representations of the speech corpus which have a spectral centroid lying above a given threshold frequency, preferably of 3 kHz.

Transformations are only needed for spectral representation which can be improved. A selection can be made by calculating the spectral centroid of each spectral input representation and selecting spectral input representations which have a spectral centroid lying above a threshold, frequency preferably above 3 kHz. Such a selection fits to a code book base on a sub-corpus of the speech corpus with the same threshold frequency. A selection can also be made by detecting at least one of speech activity and background noise level. Spectral input representations with speech to be transformed will be selected with appropriate selection criteria.

The method for adaptive spectral transformation for acoustic speech signals can be implemented in different fields. In some applications the acoustic speech signal is windowed and of each window a spectral representation is deduced. There are also applications where the spectral representations of windows of an acoustic speech signal are provided by a system. Therefore the method of this invention starts with the step of receiving spectral input representations, which can be provided by the same system or by another system. At least some of the received spectral input representations are selected and enhanced by adapted transformations and the enhanced spectral representations are provided in the form of at least one spectral output representation. The combination of untransformed and transformed spectral representations allow synthesizing an enhanced acoustic speech signal.

The invention can be implemented in a computer program comprising program code means for performing all the steps of the disclosed methods when said program is run on a computer.

The described solutions preserve and enhance voice specific characteristics such as formants and their respective positions. This has the effect of enhancing audibility, while preserving the audio quality of the original input speech signal. The current invention also avoids unnecessary delays of the output signal. Various compression functions are applied to the speech signal, which takes into account the time-varying phoneme-specific characteristics in the signal and allows for spectral sharpening through adaptive formant boosting. Also, unnecessary transformations of the input signal, which could reduce spectral resolution are avoided. The effect of phase distortion on the harmonic section of the input signal can be avoided in the current invention by limiting the compression processing to only unvoiced segments of speech.

A focus on efficient algorithms and output signal synthesis allows the current invention to also avoid costly postprocessing to achieve high quality audio output. Additionally, transforming the input signal from the time domain to the frequency domain is not bound to the FFT (Fast Fourier Transform) algorithm. Compression functions are intended to be designed such that they are able to be efficiently stored in memory and do not inadvertently give portions of the input spectrum undesired emphasis.

• Fig. 1 spectrogram of the utterance "Sauerkraut is served once a week". The plots below show the percentage of signal energy above 4 kHz and the amplitude of the signal
• Fig. 2 example of the LBG-algorithm for clusters of two dimensional feature vectors with (c) k = K = 8 classes, x- and y-axis correspond to the value ranges of the first and the second features
• Fig. 3 results of clustering with K = 8 in classes after pre-classification with a threshold of fT = 3kHz. The line with more details shows the average spectrum in dB, the line with less details shows a filtered spectral representation of the code book entry and the vertical line the spectral centroid fc
• Fig. 4 a block diagram of the adaptive transformation method
• Fig. 5 and 6, compression functions
• Fig. 7 a formant boosting gain function and spectral representations (cluster mean and filtered cluster mean), where the line with more details shows the average spectrum in dB, the line with less details shows a filtered spectral representation of the code book entry
• Fig. 8 a block diagram of the adaptive transformation method using feature vectors
• Fig. 9 examples for processing of phonemes: (a) spectral compression with compression characteristic below, (b) formant boosting with gain g(Ω_µ) between + -10dB. The processed signals are band limited to 4 kHz (shorter curve)
• Fig. 10 example of the sauerkraut utterance processed with the 128 class scheme (input signal above, processed signal below)
• Fig. 11 time series auf a noise reduction filter and its derivative
• Fig. 12 block diagram of the onset sharpening using an algorithm according to $G_{\mathit{rec}}^{\left(\mathit{mod}1\right)}\left({\mathrm{\Omega }}_{\mu }l\right)$ of the recursive Wiener filter

Fig. 1 shows for the acoustic speech "Sauerkraut is served once a week" the energy distribution in time and frequency, a time series of the energy above 4 kHz and a time series of the amplitude. The fricatives "s", "ce" and "k" have quite some energy in the frequency range from 4 to 8 kHz.

A time-domain input signal, as for example "Sauerkraut is served once a week", is windowed before being transferred to the frequency domain via a fast Fourier transform (FFT) algorithm, discrete Fourier transform (DFT), discrete cosine transform (DCT), polyphase filterbank, wavelet transform, or some other time to frequency domain transformation. In the frequency domain the windowed time-domain signal has the form of spectral representations. By filtering for example with a mel-filterbank the spectral representations can be reduced to feature vectors.

Fig. 2 shows an example of clusters of two dimensional feature vectors. With the LBG-algorithm for (c) k = K = 8 classes eight cluster centers + are found. The procedure is similar for feature vectors with higher dimensionality. The spectral representations or the feature vectors of the cluster centres are defined on the bases of spectral representations of windowed acoustic speech segments of a speech corpus by the clustering algorithm. A code book includes the spectral class representations for the cluster centres, which are averages over the elements of the cluster.

Fig. 3 shows results of clustering with K = 8 in classes after pre-classification with a threshold of fT = 3kHz in (b). The line with more details shows the average spectrum in dB, the line with less details shows a filtered spectral representation of the code book entry and the vertical line the spectral centroid fc . The filtered spectral representations of the code book entries are the spectral class representations for the cluster centres, which are averages over the elements of the cluster.

The spectral representations or feature vectors of windowed frames of an input signal are subjected to a classification technique from the field of pattern recognition, such as the minimum mean squared error estimator. The input frames are classified by finding the class corresponding to the smallest value of a cost function, d(v,c_k) in Equation 1, where v_d is the D-dimensional set of features of the current frame and c_dk is the set of features of a codebook entry k. $$d\left(\mathrm{v}{\mathrm{c}}_k\right)={\displaystyle \sum _{d=1}^D}{\left({\nu }_d-c_{\mathit{dk}}\right)}^2$$

In Fig. 4, this setup is illustrated using K=128 as a possible number of entries for such a codebook. The codebook can be trained using feature vectors from a training set and any number of vector quantization algorithms, such as k-means [MacQueen, 1967] or the Linde-Buzo-Gray (LBG) algorithm[Linde, Buzo, & Gray, 1980]. Linear discriminant analysis (LDA), principal component analysis (PCA), support vector machines (SVM), and other class enhancing algorithms can also be applied to decrease the intraclass variances and/or increase the interclass distances, which improves separability.

The feature vectors for training and testing are in this case a set of perception based features in the mel scale, although the feature vectors must not be limited to these specific features. The features can also be designed to emphasize signal characteristics in or near the region of interest in the frequency spectrum. The key concept here is that the feature vector has a reduced dimensionality with respect to the input signal frequency spectrum in order to reduce computational costs.

A spectral compression function and/or a formant boosting function are chosen from the relevant codebook entry.

To avoid possible switching artifacts, a smoothing procedure can be applied to the chosen compression function in the frequency domain but in the time direction. Using for example, an IIR-smoothing function, the time-variance of the applied compression functions can be reduced.

The spectral compression functions are functions- in the preferred case continuous and nonlinear - that apply compression rates in the frequency domain. Both the compression functions and the frequency spectrum can be either linearly or nonlinearly scaled, reflecting the occasional advantages of processing in more perception based scales like the logarithmic scale.

The operation of spectral compression maps a frequency interval of the input signal into a smaller frequency interval of the output. When working on a discrete representation of the spectrum, this is equivalent to mapping a set of frequency pins from the input spectrum X(e^jΩµ) into one frequency pin of the output spectrum Y (e^jΩµ). Mathematically, an operation that reduces the input frequency bandwidth by approximately 0.5 can be expressed by $$\begin{array}{cc}Y\left(e^{j{\mathrm{\Omega }}_{\nu }}\right)=\frac{Y\left(e^{j{\mathrm{\Omega }}_{\nu }}\right)1}{\left|X\left(e^{j{\mathrm{\Omega }}_{\nu }}\right)\right|{\mu }_u^{\left(\nu \right)}-{\mu }_l^{\left(\nu \right)}-1}{\displaystyle \sum _{\mu ={\mu }_l^{\left(\nu \right)}}^{{\mu }_u^{\left(\nu \right)}}}\left|X\left(e^{j{\mathrm{\Omega }}_{\mu }}\right)\right|& \nu =0,\dots ,\frac{N}{4}+1.\end{array}$$

The upper and lower boundaries of the interval that is projected onto output frequency pin v are represented by the variables µ_u^(v)and µ_l^(v), respectively. These boundaries are actually functions of v, allowing a variable amount of compression for different frequencies. Compression with this equation derives the magnitude of Y (e^jΩv) as the mean value of the magnitude of X(e^jΩµ) for µ = µ_u^(v), ... , µ_l^(v) while retaining the phase of input component v for output component v.

Fig. 5 illustrates an example for the relationship between input frequency f_in and output frequency f_out. The curved line shows the compression characteristic, the horizontal and vertical lines indicate the frequency interval that is mapped. The frequency in Hz can be converted to the pin-index with the following relationship $$\mu =\lfloor \frac{f}{f_3}{N}_{\mathrm{FFT}}\rceil$$

Equation 2 also performs energy normalization with respect to the amount of frequency compression. Energy normalization can take many forms, however, and could also be applied in a fashion based on the momentary broadband or frequency localized SNR. Another option is to maintain the form of the precompression estimated spectral envelope and apply a gain or dampening factor to correct the energy level to correspond to the slope of the envelope in the region of interest.

The equation also preserves the phase of the original uncompressed signal, but other phase corrections and adjustments are also possible, such as using a random phase.

The compression functions can be continuous in nature and have various compression rates over frequency. Preferably, the compression function is linear in lower frequencies, corresponding to no compression when the slope is equal to 1. In the higher frequencies of interest, the compression rate increases gradually, with the rate and degree of increase dependent upon the feature-based classification.

In Fig. 6 various examples of continuous nonlinear compression functions are shown. Those displayed with solid lines are linear with a slope of, or near, 1 in the lower frequency range, hence performing little to no compression. With increasing input frequency, the compression rate increases differently for each of the curves. The dashed curves perform little to no compression in the middle frequency range, rather than the low frequencies. These curves compress frequencies in the lower and higher frequency extremes, and can be better understood as performing a transposition of the middle frequencies down into a lower region with almost no compression.

The appropriate compression function for each class of the code book is to be defined either manually, e.g., using subjective listening criteria, or automatically, e.g., applying unsupervised learning methods to find a preferred mapping to an output characteristic. Different sets of class function assignments are generally possible, since the ideal output for improved audio quality does not always coincide with that providing improved speech recognition rates.

Speech can be made more accentuated, if formants are amplified. Especially in situations with background noise, we can expect a better localized SNR in these frequency regions, so the broadband SNR can be improved by applying a frequency dependent gain factor to the input spectrum. Ideally, the amplification should only be applied during speech activity. Otherwise, background noise will be amplified during pauses.

The current invention can receive information about speech activity from an external module, such as a noise detection and cancelation module.

The formant-boosting functions are also functions - (non)continuous, (non)linear, and on a (non)linear scale - designed such that a variable gain factor is applied to frequency ranges around formants in the spectrum. Fig. 7 shows a formant boosting gain function and spectral representations (cluster mean and filtered cluster mean), where the line with more details shows the average spectrum in dB, the line with less details shows a filtered spectral representation of the code book entry. A curve can be used to determine the percentage of the gain factor that is applied to the formant frequency range. Alternatively individual gain and dampening curves can be stored in a codebook and applied to those frequency ranges identified as formants in voiced speech.

For some spaces between formants, the gain factor just described can be transformed into a dampening factor using another set of curves. The boosting curves can be designed to have positive values for amplifying the formant frequencies and negative values for the dampening curves to reduce the magnitude of the valleys between formants.

When compression is applied an additional curve can be used in the frequency range of the compression results. This amplifies or dampens the spectrum in the region of the frequency compression and so can be used to amplify or dampen the effects of the compression.

A decision can be made for each codebook entry of the spectral compression, formant boosting signal processing about the order of the steps. In some instances it can be necessary to first apply spectral compression and then the boosting function. At other times it should be possible to first boost (or dampen) regions of the spectrum and then to apply the spectral compression. This could be important when the compression is designed such that one of the formants is located in the compressed frequency range. The prior boosting of the formant would serve to retain the formant shape even after compression.

An overall block diagram of one possible incarnation of the system is seen in Fig. 8. Windows of a signal in the time domain are transformed to spectral representations by an analysis filter-bank. By extracting feature vectors a classification in relation to elements of a code book a signal processing is realized with the transformations linked to the code book elements. The transformed spectral representations are transformed to windows of a signal in the time domain by a synthesis filter-bank.

Fig. 9 shows examples for processing of phonemes:

1. (a) with a spectral compression according to the shown compression characteristic ,
2. (b) with formant boosting according to the shown gain g(Ω_µ) between + -10dB. The processed signals are band limited to 4 kHz.

Fig. 10 shows an example of the sauerkraut utterance processed with the 128 class scheme. The energy of the fricatives "s", "ce" and "k" is transformed below 4 kHz.

The system is also capable of receiving information from other signal processing modules, such as silence/unvoiced/voiced decisions from the noise estimation and reduction module. Furthermore the module should be capable of sending information to other modules, especially an ASR module, which can use the enhanced signal to improve recognition rates. The ASR module could be retrained with the compressed output data. However the compressed signal alone achieves a change in recognition rates that can be useful.

A focus on efficient algorithms and output signal synthesis allows the current invention to also avoid costly postprocessing to achieve high quality audio output. Additionally, transforming the input signal from the time domain to the frequency domain is not bound to the FFT algorithm. Compression functions in the current invention are intended to be designed such that they are able to be efficiently stored in memory and do not inadvertently give portions of the input spectrum undesired emphasis.

Another invention is disclosed, which is new and inventive independent of the independent claims. This further invention is related to onset sharpening, which can be used independently but is of course advantageously combinable with the previously described speech enhancement methods.

The onset sharpening method performs an onset sharpening, i.e., to introduce attenuation immediately before and/or amplification after speech onsets in order to make them more accentuated. This is improving speech quality and intelligibility, especially for speech signals corrupted with background noise that are to be enhanced with noise reduction methods. Noise reduction filters and their derivatives (Fig. 11) tend to react too slow and therefore do remove desired signal components during speech onsets.

Before any onset-dependent signal enhancement can be performed, the speech onsets need to be found first. This is done based on a recursive Wiener noise reduction filter. There are no real-time constraints, so all of the following steps can be applied to the entire signal x(t) giving the necessary data for the next step.

An analysis filter bank is needed to transform the input signal x(t) into the frequency domain. The result is a function X(e^jΩµ, l) with µ= 0, ... , N_DFT=2 and I = 0, ... , M-1, where M is the number of signal frames. This could also be interpreted as a (N_DFT=2+1 ) x M matrix which is constituting a spectrogram.

Based on the spectral signal representation of the previous step, the attenuation factors of the recursive Wiener filter characteristic can be computed. The following, non-frequency dependent, parameters have been used as filter parameters:

• Maximum attenuation G_min(^Ωµ, l) = G_min = 0.25 (corresponding to 20 log₁₀(G_min) = -20 dB)
• Overestimation factor β̃(Ω_µ, l) = β̃ = 5

The result G_rec(Ωµ, l) can again be seen as a (N_DFT=2+1 ) x M matrix containing the attenuation factor for each sub-band for all time instances.

In order to smooth the filter coefficients in a perceptually meaningful manner over time, a mel-filterbank of 32 bands is applied to G_rec(Ω_µ, l), resulting in the $32\mathrm{x\; M\; matrix}{G}_{\mathit{rec}}^{\left(\mathit{mel}\right)}\left(\mathrm{m}\mathrm{l}\right)\mathrm{.}$ The actual onset detection is performed within each mel-band of $G_{\mathit{rec}}^{\left(\mathit{mel}\right)}\left(\mathrm{m}\mathrm{l}\right)\mathrm{.}$. Here, the moments when $G_{\mathit{rec}}^{\left(\mathit{mel}\right)}\left(\mathrm{m}\mathrm{l}\right)$ changes its value from G_min to 1 (or close to 1) are of interest, i.e., the points when the filter opens. These time instances can be found by taking the numerical derivative $$\frac{{\mathit{dG}}_{\mathit{rec}}^{\left(\mathit{mel}\right)}\left(ml\right)}{\mathit{dl}}\approx {\mathit{G}}_{\mathit{rec}}^{\left(\mathit{mel}\right)}\left(ml\right)-{\mathit{G}}_{\mathit{rec}}^{\left(\mathit{mel}\right)}\left(m,l-1\right)$$
and comparing the resulting value with a threshold $$\frac{d{\mathit{G}}_{\mathit{rec}}^{\left(\mathit{mel}\right)}\left(ml\right)}{\mathit{dl}}>\gamma$$

The mel-band m for time instance l is labeled as a speech onset. The derivative of the noise reduction coefficient lies in the range of d $G_{\mathit{rec}}^{\left(\mathit{mel}\right)}\left(\mathrm{m}\mathrm{l}\right)/\mathrm{dl}\in \left[{\mathrm{G}}_{\min }\mathrm{-}\mathrm{1}\mathrm{,}\mathrm{1}\mathrm{-}{\mathrm{G}}_{\min }\right]$ and positive values indicate times when the filter is opening. A threshold = 0.2 has proved to give good detection results for various speech signals and SNRs. Because it could happen that the derivative is greater than for several consecutive frames, also a sliding time window of 100ms duration is applied. Within this time window, only one detection is allowed. Furthermore, if an onset has been detected in a certain mel-band, the neighboring mel-band towards lower frequencies of the same frame l is also marked as a speech onset. The clean speech is mixed with background noise recorded in a car driving at a speed of 160 km/h to form an SNR of 1 dB during speech activity. Of course, the detection can be made more sensitive by taking a lower threshold, e.g. y= 0.1.

Based on the detections made with the method of the previous section, the onset sharpening can be performed. It consists of placing attenuation immediately before a detected speech onset and boosting the signal for a short time interval afterwards. The shape of the attenuation/amplification is determined by a prototype function that has been chosen to be $$f\left(x\right)=\{\begin{array}{c}-\varrho \frac{e}{\sigma }{\mathit{xe}}^{\frac{x}{\sigma }}\mathit{for}-20\le x<0\\ \frac{e}{\sigma }{\mathit{xe}}^{\frac{-x}{\sigma }}\mathit{for}0\le x\le 20\\ 0{}_{}\mathit{otherwise}\end{array}$$
with the attenuation = 0.5 and σ = 3. The term e/ σ is used to normalize f (x) to a maximum value of 1. Out of this prototype function, the onset sharpening gain function g_os(l) can be sampled. When deriving onset sharpening gain function, three parameters can be set:

1. 1. The width of the (negative) attenuation and the (positive) boosting part, defined by the variable τ_os in ms.
2. 2. The offset τ_offs in ms that defines at which time instance the gain function is placed. For τ_offs = 0 ms, the zero crossing is exactly at the detection point, for negative offset values the zero crossing will be earlier. This is desirable because then the noise reduction filter is forced to open earlier.
3. 3. A gain parameter α_os that controls the amount of attenuation and boosting. It is multiplied with the onset sharpening gain function g_os(l) which is interpreted as dB values.

Several other prototype functions could be devised for onset sharpening, e.g., a sinusoid. The prototype function has been chosen with the stated parameters because it decays smoothly towards the ends and offers a steep slope around the zero crossing. This prototype function is now placed at each detected speech onset time instance in the corresponding mel-band, giving the onset sharpening matrix for mel-bands $g_{\mathit{os}}^{\left(\mathit{mel}\right)}\left(ml\right)$, which then is expanded into the onset sharpening matrix for the subbands g_os(m,l). The parameters that have been used for the onset sharpening gain functions are τ_os = 75 ms, τ_offs = -10ms and α_os = 3, of course other parameters are possible as well. Using the notation of the mel-filterbank matrix A the expansion from melbands to subbands can be expressed as $$g_{\mathit{os}}\left({\mathrm{\Omega }}_{\mu }l\right)={\displaystyle \sum _{m=0}^D}\frac{A_{\mathit{m\mu }}}{{\mathit{max}}_{\mu }\left\{A_{\mathit{m\mu }}\right\}}*g_{\mathit{os}}^{\left(\mathit{mel}\right)}\left(ml\right)\mathrm{.}$$

The weighting of the filters contained in A that gives triangles of a broader bandwidth a lower amplitude is removed by the normalization containing the maximum operation.

Fig. 12 discloses an onset sharpening algorithm with the following recursive Wiener noise reduction filter being modified by the onset sharpening gain function. The simplest way is by multiplication $$\begin{array}{ll}{G}_{\mathit{rec}}^{\left(\mathit{mod}1\right)}\left({\mathrm{\Omega }}_{\mu }l\right)& =g_{\mathit{os}}^{\mathit{lin}}\left({\mathrm{\Omega }}_{\mu }l\right)\cdot G_{\mathit{rec}}\left({\mathrm{\Omega }}_{\mu }l\right)\\ & =g_{\mathit{os}}^{\mathit{lin}}\left({\mathrm{\Omega }}_{\mu }l\right)\cdot \max \left\{G_{\mathit{min}}\left({\mathrm{\Omega }}_{\mu }l\right),1-\frac{\tilde{\beta }\left({\mathrm{\Omega }}_{\mu }l\right)}{G\left({\mathrm{\Omega }}_{\mu },l-1\right)}\cdot \frac{{\left|\hat{B}\left(e^{j{\mathrm{\Omega }}_{\mu }}l\right)\right|}^2}{{\left|\hat{X}\left(e^{j{\mathrm{\Omega }}_{\mu }}l\right)\right|}^2}\right\}\end{array}$$
where $$g_{\mathit{os}}^{\mathit{lin}}\left({\mathrm{\Omega }}_{\mu }l\right)={10}^{{\alpha }_{\mathit{dB}}\mathrm{.}{g}_{\mathit{os}}\left({\mathrm{\Omega }}_{\mu }l\right)/20}$$
is the onset sharpening function in linear values. Since the noise reduction filter is applied multiplicative to the input spectrum, this filter modification could also be interpreted as a multiplication of the signal spectrum X(e^jΩµ, l) with the onset sharpening gain before (or after) applying the noise reduction filter.

In a second modification, the onset sharpening gain is built into the noise reduction characteristic to modify the spectral floor and the maximum gain of the filter (which is set to 1 in the recursive Wiener filter): $${\mathrm{G}}_{\mathrm{rec}}^{\left(\mathrm{mod}\mathrm{2}\right)}\left({\mathrm{\Omega }}_{\mathrm{\mu }}\mathrm{l}\right)\mathrm{=}\max \left\{{\mathrm{G}}_{\min }\left({\mathrm{\Omega }}_{\mathrm{\mu }}\mathrm{l}\right)\mathrm{\cdot }{\mathrm{g}}_{\mathrm{os}}^{\mathrm{att}}\left({\mathrm{\Omega }}_{\mathrm{\mu }}\mathrm{l}\right)\mathrm{,}{\mathrm{g}}_{\mathrm{os}}^{\mathrm{amp}}\left({\mathrm{\Omega }}_{\mathrm{\mu }}\mathrm{l}\right)\mathrm{-}\frac{{\left|\hat{\mathrm{B}}\left({\mathrm{e}}^{{\mathrm{j\Omega }}_{\mathrm{\mu }}}\mathrm{l}\right)\right|}^{\mathrm{2}}}{{\left|\hat{\mathrm{X}}\left({\mathrm{e}}^{{\mathrm{j\Omega }}_{\mathrm{\mu }}}\mathrm{l}\right)\right|}^{\mathrm{2}}}\right\}\mathrm{.}$$

For this description, the gain function has to be separated into the part responsible for the attenuation before speech onsets $${\mathrm{g}}_{\mathrm{os}}^{\mathrm{att}}\left({\mathrm{\Omega }}_{\mathrm{\mu }}\mathrm{l}\right)=\{\begin{array}{cc}{10}^{{\alpha }_{\mathit{dB}}\mathrm{.}{g}_{\mathit{os}}\left({\mathrm{\Omega }}_{\mu }l\right)/20}& {\mathit{for\; g}}_{\mathrm{os}}\left({\mathrm{\Omega }}_{\mathrm{\mu }}\mathrm{l}\right)<0\hfill \\ 1& \mathrm{otherwise}\hfill \end{array}$$
and the part for amplification after speech onsets $${\mathrm{g}}_{\mathrm{os}}^{\mathrm{amp}}\left({\mathrm{\Omega }}_{\mathrm{\mu }}\mathrm{l}\right)=\{\begin{array}{cc}{10}^{{\alpha }_{\mathit{dB}}\mathrm{.}{g}_{\mathit{os}}\left({\mathrm{\Omega }}_{\mu }l\right)/20}& {\mathit{for\; g}}_{\mathrm{os}}\left({\mathrm{\Omega }}_{\mathrm{\mu }}\mathrm{l}\right)\ge 0\hfill \\ 1& \mathrm{otherwise}\hfill \end{array}$$

A third possibility is to modify also the overestimation factor: $${\mathrm{G}}_{\mathrm{rec}}^{\left(\mathrm{mod}\mathrm{3}\right)}\left({\mathrm{\Omega }}_{\mathrm{\mu }}\mathrm{l}\right)\mathrm{=}\max \left\{{\mathrm{G}}_{\min }\left({\mathrm{\Omega }}_{\mathrm{\mu }}\mathrm{l}\right)\mathrm{\cdot }{\mathrm{g}}_{\mathrm{os}}^{\mathrm{att}}\left({\mathrm{\Omega }}_{\mathrm{\mu }}\mathrm{l}\right)\mathrm{,}\mathrm{1}\mathrm{-}\frac{\tilde{\mathit{\beta }}\left({\mathrm{\Omega }}_{\mu }l\right)}{{\mathrm{g}}_{\mathrm{os}}^{\mathrm{amp}}\left({\mathrm{\Omega }}_{\mathrm{\mu }}\mathrm{l}\right)\mathrm{\cdot }\mathrm{G}\left({\mathrm{\Omega }}_{\mathrm{\mu }}\mathrm{,}\mathrm{l}\mathrm{-}\mathrm{1}\right)}\frac{{\left|\hat{\mathrm{B}}\left({\mathrm{e}}^{{\mathrm{j\Omega }}_{\mathrm{\mu }}}\mathrm{1}\right)\right|}^{\mathrm{2}}}{{\left|\hat{\mathrm{X}}\left({\mathrm{e}}^{{\mathrm{j\Omega }}_{\mathrm{\mu }}}\mathrm{1}\right)\right|}^{\mathrm{2}}}\right\}$$

Inspection of noise reduction filters from the first and second modification shows that there are only small differences between the characteristics. The parameter settings for τ_os, τ_offs, α_os and the choice of the gain prototype function are of much greater influence. Therefore, the simpler modification $G_{\mathit{rec}}^{\left(\mathit{mod}1\right)}\left({\mathrm{\Omega }}_{\mu }l\right)$) will be used for the evaluation.

For the evaluation of the speech onset enhancement method, the recursive Wiener filter with $G_{\mathit{rec}}^{\left(\mathit{mod}1\right)}\left({\mathrm{\Omega }}_{\mu }l\right)$ has been compared to a standard recursive Wiener filter $G_{\mathit{rec}}^{}\left({\mathrm{\Omega }}_{\mu }l\right)$. This has mainly been done on the basis of a logarithmic spectral distance (LSD) measure. Comparison of the noise reduction filter coefficients for several characteristics gives a qualitative impression about the opening/closing properties of a filter. Finally, listening tests give a useful criterion that help to judge intelligibility and the amount of artifacts such as musical tones.

The idea in using an LSD measure is to create a signal x(t) = s(t)+ b(t) corrupted with background noise, where the speech component s(t) and the noise b(t) are known. A noise reduction filter is computed for the disturbed signal x(t) and the two signal components are passed through this filter separately. Then, the distortion measures LSD_speech and LSD_noise can be calculated between the original and the filtered signal. Ideally, the speech component is passed through the filter unchanged, leading to a distance close to zero, whereas large distortions can be expected for the noise component. Based on these two measures, it is possible to judge on the noise suppression ability and on how much speech components are affected. The LSD between two time varying spectra S(e^jΩµ, l) and Ŝ(e^jΩµ, l) is defined as $$\mathit{LSD}=\frac{10}{D}{\displaystyle \sum _{l=0}^L}\sqrt{{\displaystyle \sum _{\mu =0}^{N_{\mathit{DFT}{/}_2}}}\frac{K_{\mu ,l}}{{\bar{K}}_l}{\mathit{log}}_{10}^2\left\{\frac{\mathit{max}\left\{{\left|S\left(e^{j{\mathrm{\Omega }}_{\mu }}l\right)\right|}^2{\delta }_S\right\}}{\mathit{max}\left\{{\left|\hat{S}\left(e^{j{\mathrm{\Omega }}_{\mu }}l\right)\right|}^2{\delta }_{\hat{S}}\right\}}\right\}}$$

The two variables $${\delta }_S={10}^{-5}\underset{\mu ,l}{\max }\left\{{\left|S\left(e^{j{\mathrm{\Omega }}_{\mu }}l\right)\right|}^2\right\}$$ $${\delta }_{\hat{S}}={10}^{-5}\underset{\mu ,l}{\max }\left\{{\left|\hat{S}\left(e^{j{\mathrm{\Omega }}_{\mu }}l\right)\right|}^2\right\}$$
give lower bounds for the values that enter the measure. These components are selected by the binary mask $$K_{\mu ,l}=\{\begin{array}{c}1\mathit{if}{\left|S\left(e^{j{\mathrm{\Omega }}_{\mu }}l\right)\right|}^2\ge {\delta }_S\\ 0\mathit{otherwise}\end{array}$$

The normalization factor K̅_l counts the number of components that are used for the distance measure in each time frame. In order to avoid a division by zero, it is defined as $${\bar{K}}_l=\max \left\{{\displaystyle \sum _{\mu =0}^{N_{\mathit{DFT}{/}_2}}}{K}_{\mu ,l},0.1\right\}$$

The variable D gives the number of signal frames that are used for the calculation of the LSD, i.e., the number of signal frames with K̅_l > 0.

For evaluating the performance of the modified recursive Wiener filter $G_{\mathit{rec}}^{\left(\mathit{mod}1\right)}\left({\mathrm{\Omega }}_{\mu }l\right),$, a set of 616 filters has been computed with all possible combinations of the parameters τ_os ∈ [50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100] ms
τ _offs ∈ [0, -5, -10, -15, -20, -25, -30, -35] ms
α _os ∈ [0, 2, 4, 6, 8, 10, 12] .

The signal that has been used is the "Sauerkraut is serve once a week" utterance used throughout this text, mixed with background noise from a car driving at a constant speed of 160km/h. The SNR d ring speech activity is adjusted to 1 dB. Then, only the speech and only the noise components have been processed with these filters and the distances ${\mathit{LSD}}_{\mathit{speech}}^{\left(\mathit{os}\right)}$ and ${\mathit{LSD}}_{\mathit{noise}}^{\left(\mathit{os}\right)}$ have been calculated. As reference for the comparison, a recursive Wiener filter has been designed and the measures ${\mathit{LSD}}_{\mathit{speech}}^{\mathit{rw}}$ and ${\mathit{LSD}}_{\mathit{noise}}^{\left(\mathit{rw}\right)}$ have been evaluated.

As mentioned earlier, a good filter is characterized by a small LSD for speech and a large value for noise. Obviously, these two requirements are difficult to meet at the same time and an increase in one category usually falls together with an increase in the other one. For better comparison of the two filter types, the ${\mathit{LSD}}_{\mathit{speech}}^{\left(\mathit{rw}\right)}-{\mathit{LSD}}_{\mathit{speech}}^{\left(\mathit{os}\right)}$ and ${\mathit{LSD}}_{\mathit{noise}}^{\left(\mathit{os}\right)}-{\mathit{LSD}}_{\mathit{noise}}^{\left(\mathit{rw}\right)}$ could be calculated They are defined such, that a positive value means that the onset sharpening approach gives better results in the LSD sense. This is apparently not always the case and again it can be seen that an improvement in noise suppression leads to worse results for speech and vice versa. However, there are some combinations where a positive value can be achieved in both disciplines, e.g., for the parameter combination
τ _os = 100ms
τ _offs = -30 ms.

The gain factor α_os basically only scales the distance measure. For the special case of T_os = 0, the modified filter reduces to the standard recursive Wiener filter and thus gives the same LSD.

Comparing the noise reduction filter coefficients for a Wiener filter, a recursive Wiener filter and a recursive Wiener filter modified by onset detection for several subbands and different filter parameters gives the following result. The same signal as for the LSD measures has been used for the design of these filters.

The parameters that have been used are $$\left[{\tau }_{\mathit{os}}{\tau }_{\mathit{offs}}{\alpha }_{\mathit{os}}\right]=\left[75\mathrm{ms},-10\mathrm{ms},3\right]$$

But of course other parameters are possible, too.

The Wiener filter opens more often, which potentially results in musical tones. It has also been seen that the recursive Wiener filter opens a bit later, which can be corrected by the onset sharpening modification.

First of all it should be noticed that the offset τ_offs = -30ms is fairly large compared to the duration of τ_os = 100 ms. This causes that the modified filter coefficient near a speech onset increases, then decreases for a few frames and finally grows again with the opening of the recursive Wiener filter. Apparently, this is no optimal behavior even though this parameter combination gave the best results in the LSD. This gives rise to the assumption, that a different prototype function should be used. Good candidates would be more flat around their maximum, avoiding the "crash down" near 0.18 and 0.5 seconds. At any rate, a filter that is opening earlier seems to give benefits in terms of the LSD measure.

However, for larger gains α_os, the noise floor was decreased on the expense of increasing filtering artifacts.

Fig. 11 shows a procedure where the attenuation factor and its derivative together with the threshold are shown for mel-band 25 (corresponding to frequencies between 3.6 and 4.4 kHz). Because it could happen that the derivative is greater than y for several consecutive frames, also a sliding time window of 100ms duration is applied. Within this time window, only one detection is allowed. Furthermore, if an onset has been detected in a certain mel-band, the neighboring mel-band towards lower frequencies of the same frame I is also marked as a speech onset.

The signal that has been used is the utterance "Sauerkraut is served once a week" from the TIMIT database that has also been used before. The clean speech is mixed with background noise recorded in a car driving at a speed of 160 km/h to form an SNR of 1 dB during speech activity.