The invention relates to an audio apparatus and method of operation therefor, and specifically, but not exclusively, to representing and rendering stereo signals using Parametric Stereo based encoding.

Spatial audio applications have become numerous and widespread and increasingly form part of many audiovisual experiences. New and improved spatial experiences and applications are continuously being developed which result in increased demands on the audio processing and rendering.

A lot of research and development effort has focused on providing efficient and high quality audio encoding and audio decoding for spatial audio. A frequently used spatial audio representation is multichannel audio representations, including stereo representation, and efficient encoding of such multichannel audio based on downmixing multichannel audio signals to downmix channels with fewer channels have been developed. One of the main advances in low bit-rate audio coding has been the use of parametric multichannel coding where a downmix signal is generated together with parametric data that can be used to upmix the downmix signal to recreate the multichannel audio signal.

In particular, instead of traditional mid-side or intensity coding, in parametric multichannel audio coding, a multichannel input signal is downmixed to a lower number of channels (e.g. two to one) and multichannel image (stereo) parameters are extracted. Then the downmix signal is encoded using a more traditional audio coder (e.g. a mono audio encoder). The bitstream of the downmix is multiplexed with the encoded multichannel image parameter bitstream. This bitstream is then transmitted to the decoder, where the process is inverted. First the downmix audio signal is decoded, after which the multichannel audio signal is reconstructed guided by the encoded multichannel image/ upmix parameters.

An example of stereo coding is described in E. Schuijers, W. Oomen, B. den Brinker, J. Breebaart, "Advances in Parametric Coding for High-Quality Audio", 114th AES Convention, Amsterdam, The Netherlands, 2003, Preprint 5852. In the described approach, the downmixed mono signal is parametrized by exploiting the natural separation of the signal into three components (objects): transients, sinusoids, and noise. In E. Schuijers, J. Breebaart, H. Pumhagen, J. Engdegård, "Low Complexity Parametric Stereo Coding", 116th AES, Berlin, Germany, 2004, Preprint 6073 more details are provided describing how parametric stereo was realized with a low (decoder) complexity when combining it with Spectral Band Replication (SBR).

Parametric Stereo (PS) is a technology which is widely used to efficiently code a stereo signal as a mono downmix and a set of spatial parameters allowing an accurate reconstruction of the stereo image. PS has been used to substantially improve the compression efficiency for AAC and USAC at lower bit-rates, say 32kbps and below.

In addition to accurately reproducing a stereo signal, it has also been of interest to create high quality binaural rendering of (encoded) stereo signals to emulate a virtual loudspeaker playback.

Binaural rendering of content authored for multi-channel playback can be achieved by the sum of convolutions of the input channel signals with left and right Head Related Impulse Responses (HRIRs), where each HRIR pair corresponds to a measured/simulated impulse response from a loudspeaker location to the ears. This can be expressed compactly in the z-domain as: $${\mathrm{Y}}_{\mathrm{L},\mathrm{R}}\left(\mathrm{z}\right)={\displaystyle \sum _{\forall \mathrm{c}}{\mathrm{X}}_{\mathrm{C}}\left(\mathrm{z}\right)\cdot {\mathrm{H}}_{\mathrm{L},\mathrm{R}}^{{\mathrm{\phi }}_{\mathrm{c}}}\left(\mathrm{z}\right)}$$ where X_C(z) represents the z-transform of the time domain input signal x_c[n] with channel c, Y_L,R(z) represents the z-transform of the left and right time domain output signals l[n] and r[n] respectively and ${\mathrm{H}}_{\mathrm{L},\mathrm{R}}^{{\mathrm{\phi }}_{\mathrm{c}}}\left(\mathrm{z}\right)$ is the z-transform of the HRIR of the left and right channels h_l[n] and h_r[n] for the angle (and distance) corresponding to loudspeaker position φ_c. Approaches for rendering binaural stereo are disclosed in WO2010/122455A1 and WO2007/031896 A1.

A particular approach for rendering stereo audio using headphones is presented in J. Breebaart and E. Schuijers. "Phantom materialization: A novel method to enhance stereo audio reproduction on headphones.", IEEE transactions on audio, speech, and language processing 16.8 (2008): 1503-1511. The approach seeks to provide accurate rendering of both audio point sources and more ambient and background audio.

However, whereas current approaches for audio rendering may provide acceptable performance in many applications and scenarios, they tend to not be ideal and may exhibit suboptimal behavior in some scenarios. In particular, it may result in suboptimal perceived quality and/or a reduced user experience with e.g. perceived suboptimal spatial perception/audio scene in some cases. Complexity and/or resource usage may also be higher than desired and may in some case make the approach undesired or impractical for some implementations, such as applications based on small and cheap portable devices.

Hence, an improved approach would be advantageous. In particular an approach allowing increased flexibility, improved adaptability, improved performance, increased audio quality, improved perceived quality, an improved rendering/generation of a multichannel signal, improved spatial perception, reduced complexity and/or resource usage, reduced computational load, facilitated implementation, improved user experience, and/or an improved spatial audio experience would be advantageous.

Accordingly, the Invention seeks to preferably mitigate, alleviate or eliminate one or more of the above mentioned disadvantages singly or in any combination.

According to an aspect of the invention there is provided an audio apparatus comprising: a receiver arranged to receive a data signal comprising a set of spatial upmix parameters being indicative of relative signal properties of channels of the stereo signal, encoded data for a mono downmix audio signal being a downmix of a stereo signal, and a difference signal representing a difference between the mono downmix audio signal and a directional signal, the directional signal representing a point source audio component of the stereo signal; a store comprising directional transfer functions for different directions, a directional transfer function for a given direction representing a mapping of a mono audio signal to stereo channels such that the mono audio signal is positioned in the given direction in a stereo image of the stereo channels; a decoder arranged to generate the mono downmix audio signal and the difference signal by decoding the encoded data; a compensator arranged to generate a modified mono downmix audio signal by compensating the mono downmix audio signal as a function of the difference signal; a direction determining circuit arranged to determine a first direction from the spatial upmix parameters; a first renderer arranged to perform a first rendering of the modified mono downmix audio signal to generate a first intermediate stereo signal, the first rendering being a directional rendering using a first channel transfer function retrieved from the store for the first direction; and an output circuit to generate the output stereo signal to include the first intermediate stereo signal.

The approach may provide an improved audio experience in many embodiments. For many signals and scenarios, the approach may provide improved rendering of a stereo audio signal allowing improved generation/ reconstruction of a stereo audio signal with an improved perceived audio quality. The approach may provide improved representation of an audio scene by a stereo signal. The approach may in many embodiments allow an improved and/or more attractive spatial audio experience from a stereo signal.

The approach may provide an efficient implementation and may in many embodiments allow reduced complexity and/or resource usage. The approach may in many scenarios allow a reduced computational burden while providing a perceived high quality rendering of a stereo signal, and in particular based on an efficient representation of a stereo signal using a downmix and spatial upmix parameters, such as specifically a PS encoded stereo signal.

The processing may be in time frequency segments or tiles. Each time frequency segment/tile may represent a frequency interval in a time interval. In many embodiments, the mono downmix audio signal may be divided into time segments/intervals and a frequency representation of the signal in the time segment/interval may be provided by signal values representing different frequency segments of the signal in the time segment/interval. Some or all of the processing may be performed in the frequency domain/ frequency subbands.

The spatial upmix parameters may comprise sets of upmix parameters, each set of upmix parameters comprising at least one of: a level difference parameter indicative of a level difference between channels of the multichannel audio signal; a correlation parameter indicative of a coherence between channels of the multichannel audio signal; a timing difference parameter indicative of a timing difference between channels of the multichannel audio signal, and a phase difference parameter indicative of a phase difference between channels of the multichannel audio signal.

The first direction may be a desired/target rendering direction. A direction may be an angle and/or orientation from a listening position/in a stereo image.

The directional rendering may be a binaural rendering generating the first intermediate stereo signal as a binaural stereo signal comprising a point source positioned in the first direction, the binaural rendering comprising selecting binaural impulse response values as values for a binaural impulse response for a sound source in the first direction. The directional transfer functions may be parameterized transfer functions, and specifically may be represented in the frequency domain as weights for each of a plurality of subbands. A weight may be provided for each stereo channel. The weights may typically be complex valued.

The binaural impulse response values may be parametric values and may be frequency tile values. The binaural impulse response values may be values representing any suitable binaural impulse response in any suitable way, including HRIR, HRTF, BRIR values etc.

In many embodiments, the encoded data and the spatial upmix parameters are part of a Parametric Stereo encoding of the stereo signal.

The compensator may be arranged to generate the modified mono downmix audio signal by compensating the mono downmix audio signal as a function of the difference signal such that a difference between the modified mono downmix audio signal and the directional signal is reduced in comparison to a difference between the mono downmix audio signal and the directional signal. In some embodiments, the compensator may be arranged to generate the modified mono downmix audio signal by offsetting the mono downmix audio signal by the difference signal. The offsetting of the mono downmix audio signal may be to reduce a difference between the directional signal and the modified mono downmix audio signal. In some embodiments, the compensator may be arranged to generate the modified mono downmix audio signal by combining the mono downmix audio signal and the difference signal. The combining may be a weighted combination and specifically a weighted linear combination. In some embodiments, the compensator may be arranged to generate the modified mono downmix audio signal by summing the mono downmix audio signal and the difference signal.

The mono downmix audio signal may be any downmix that based on the spatial parameters may be upmixed to generate the stereo signal. The mono downmix audio signal and spatial parameters may together form a representation of the (output) stereo signal.

According to an optional feature of the invention, the audio apparatus further comprises: a decorrelator arranged to apply a decorrelation to at least one of the mono downmix audio signal and the modified mono downmix audio signal to generate a first decorrelated mono downmix audio signal; a second renderer arranged to perform a second rendering being a rendering of the first decorrelated mono downmix audio signal to generate a second intermediate stereo signal, the second rendering being a predetermined rendering employing a predetermined mapping of the decorrelated mono downmix audio signal to channel signals of the second intermediate stereo signal; and wherein the output circuit is arranged to combine at least the first intermediate stereo signal and the second intermediate stereo signal to generate an output stereo signal.

This may provide improved audio rendering in many embodiments. It may in many scenarios and applications allow an improved and/or more attractive spatial audio experience from a stereo signal. It may provide an advantageous approach for many scenarios, including e.g. providing an advantageous trade-off between complexity, computational resources, data rate and/or the perceived audio quality of the generated output stereo signal.

According to an optional feature of the invention, the difference signal is indicative of a difference between the directional signal and an estimate of the directional signal, the estimate of the directional signal being a predetermined function of the mono downmix audio signal and the spatial parameters; and the compensator is arranged to generate an estimate of the directional signal from the mono downmix audio signal and the spatial parameters (using the predetermined function) and to generate the modified mono downmix audio signal by compensating the predicted signal as a function of the difference signal.

This may provide improved audio rendering in many embodiments. It may in many scenarios and applications allow an improved and/or more attractive spatial audio experience from a stereo signal. It may provide an advantageous approach for many scenarios, including e.g. providing an advantageous trade-off between complexity, computational resources, data rate and/or the perceived audio quality of the generated output stereo signal.

According to an optional feature of the invention, the compensator is arranged to determine the estimate of the directional signal by scaling the mono downmix audio signal using scale factors determined from the spatial parameters.

This may provide improved audio rendering in many embodiments. It may in many scenarios and applications allow an improved and/or more attractive spatial audio experience from a stereo signal.

The scale factors may be applied at any stage in the rendering process/path including prior to or after rendering.

In some embodiments, the scaling value may be determined substantially as: $$g_x=\sqrt{\frac{\sqrt{{\left(\mathrm{IID}-1\right)}^2+4{\mathrm{ICC}}^2\mathrm{IID}}}{\mathrm{IID}+1}}$$ or $$g_x=\sqrt{\frac{{\left(\mathrm{IID}-1\right)}^2+4\cdot {\mathrm{ICC}}^2\cdot \mathrm{IID}+\left(1-\mathrm{IID}\right)\cdot \sqrt{4\cdot {\mathrm{ICC}}^2\cdot \mathrm{IID}+{\left(\mathrm{IID}-1\right)}^2}}{1-{\mathrm{IID}}^2+\left(\mathrm{IID}+1\right)\cdot \sqrt{4\cdot {\mathrm{ICC}}^2\cdot \mathrm{IID}+{\left(\mathrm{IID}-1\right)}^2}}}$$ where IID is an interchannel intensity difference and ICC is an inter-channel cross-correlation as described above.

The scale factors may be determined for and applied in frequency subbands.

According to an optional feature of the invention, a data rate of encoded data for the mono downmix audio signal is no less than two, five, ten, 25, 50, or even 100 times higher than a data rate of encoded data for the difference signal.

The approach may allow an improved audio quality to data rate relationship in many scenarios and applications.

According to an optional feature of the invention, the first rendering is a binaural rendering and the directional transfer functions are binaural transfer functions.

This may provide improved audio rendering in many embodiments.

According to an optional feature of the invention, the second rendering is arranged to generate the second intermediate stereo signal using a set of directional transfer functions retrieved from the store for a set of predetermined directions.

This may provide an advantageous approach for many scenarios, including e.g. providing an advantageous trade-off between complexity, computational resources, data rate and/or the perceived audio quality of the generated output stereo signal.

In some embodiments, the set of predetermined directions consists of one predetermined direction.

This may provide a particularly advantageous trade-off between complexity and user experience/perceived audio quality in many scenarios and embodiments.

In some embodiments, the set of predetermined directions comprises a plurality of predetermined directions.

This may provide a particularly advantageous trade-off between complexity and user experience/perceived audio quality in many scenarios and embodiments.

According to an optional feature of the invention, the spatial upmix parameters and the directional transfer functions are provided for frequency subbands and the first renderer is arranged to generate subband values for subbands of the first intermediate stereo signals from subband values of the modified mono downmix audio signal based on spatial upmix parameters and directional transfer functions for the subbands.

This may provide improved audio rendering in many embodiments. It may in many scenarios and applications allow an improved and/or more attractive spatial audio experience from a stereo signal.

According to an optional feature of the invention, the direction determining circuit is arranged to determine a point source direction in a stereo image of the stereo signal from the spatial upmix parameters, and to determine the first direction by applying a mapping function to the point source direction.

This may provide an advantageous approach for many scenarios, including e.g. providing an advantageous trade-off between complexity, computational resources, data rate and/or the perceived audio quality of the generated output stereo signal.

The mapping may be non-uniform. The mapping may be from one range to a different range (e.g. from [0,90°] to [-30°,30°]). The mapping may be non-linear.

In some embodiments, the direction determining circuit may be arranged to determine an indication of the point source direction as: $$\gamma =\arctan \left(\frac{1-\mathrm{IID}+\sqrt{{\left(\mathrm{IID}-1\right)}^2+4\cdot {\mathrm{ICC}}^2\cdot \mathrm{IID}}}{2\cdot \mathrm{ICC}\cdot \sqrt{\mathrm{IID}}}\right)$$ where IID is an interchannel intensity difference and ICC is an inter-channel cross-correlation. The point source direction may be derived by interpreting y as a relative angle between two loudspeakers (γ = 0° is one speaker, 90° is the other). In typical notation a front-left speaker is typically at 30 degrees, and front-right at -30 degrees.

According to another aspect of the invention, there is provided an audio apparatus comprising: a downmixer arranged to generate a mono downmix audio signal for a stereo signal; a parameter determiner arranged to determine spatial upmix parameters for upmixing the mono downmix audio signal to the stereo signal, the spatial upmix parameters being indicative of relative signal properties of channels of the stereo signal; a generator arranged to determine a directional signal for the stereo signal, the directional signal representing a point source audio component of the stereo signal; a difference signal circuit arranged to generate a difference signal indicative of a difference between the mono downmix audio signal and the directional signal; and a data signal circuit arranged to generate the audio data signal to comprise the spatial parameters, the encoded data for the mono downmix audio signal, and the difference signal.

This may provide improved representation of a stereo signal allowing audio rendering in many embodiments and scenarios. It may in many scenarios and applications allow an improved and/or more attractive spatial audio experience from a stereo signal. It may provide an advantageous approach for many scenarios, including e.g. providing an advantageous trade-off between complexity, computational resources, data rate and/or the perceived audio quality of the generated output stereo signal.

According to an optional feature of the invention, the difference signal circuit is arranged to generate an estimate/prediction of the directional signal from the mono downmix audio signal and the spatial parameters (using the predetermined function) and to generate the difference signal to indicate a difference between the estimate/prediction of the directional signal and the directional signal.

This may provide improved performance in many embodiments.

According to an optional feature of the invention, the data circuit is arranged to include the encoded data for the difference signal in an optional data field of the audio data signal.

This may provide improved performance in many embodiments.

According to another aspect of the invention, there is provided a method of operation for an audio apparatus, the method comprising: receiving a data signal comprising a set of spatial upmix parameters being indicative of relative signal properties of channels of the stereo signal, encoded data for a mono downmix audio signal being a downmix of a stereo signal, and a difference signal representing a difference between the mono downmix audio signal and a directional signal, the directional signal representing a point source audio component of the stereo signal; providing directional transfer functions for different directions, a directional transfer function for a given direction representing a mapping of a mono audio signal to stereo channels such that the mono audio signal is positioned in the given direction in a stereo image of the stereo channels; generating the mono downmix audio signal and the difference signal by decoding the encoded data; generating a modified mono downmix audio signal by compensating the mono downmix audio signal as a function of the difference signal; determining a first direction from the spatial upmix parameters; performing a first rendering of the modified mono downmix audio signal to generate a first intermediate stereo signal, the first rendering being a directional rendering using a first channel transfer function retrieved from the store for the first direction; and generating the output stereo signal to include the first intermediate stereo signal.

According to another aspect of the invention, there is provided a method of operation for an audio apparatus, the method comprising: generating a mono downmix audio signal for a stereo signal; determining spatial upmix parameters for upmixing the mono downmix audio signal to the stereo signal, the spatial upmix parameters being indicative of relative signal properties of channels of the stereo signal; determining a directional signal for the stereo signal, the directional signal representing a point source audio component of the stereo signal; generating a difference signal indicative of a difference between the mono downmix audio signal and the directional signal; and generating the audio data signal to comprise the spatial parameters, the encoded data for the mono downmix audio signal, and the difference signal.

These and other aspects, features and advantages of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter.

Embodiments of the invention will be described, by way of example only, with reference to the drawings, in which

• FIG. 1 illustrates some elements of an example of an audio distribution system;
• FIG. 2 illustrates some elements of an example of an audio apparatus in accordance with some embodiments of the invention;
• FIG. 3 illustrates some elements of an example of an audio render apparatus in accordance with some embodiments of the invention;
• FIG. 4 illustrates some elements of an example of an audio render apparatus in accordance with some embodiments of the invention; and
• FIG. 5 illustrates some elements of a possible arrangement of a processor for implementing elements of an audio apparatus in accordance with some embodiments of the invention.

FIG. 1 illustrates an example of an audio system wherein a stereo signal may be distributed/communicated for remote rendering. In the system, an audio apparatus referred to as the audio source device 101 generates an audio data signal including a representation of a stereo audio signal. The stereo audio signal may be one captured at the audio source device 101, may be received from another source, or indeed may e.g. be an artificially generated stereo signal (e.g. it may be a virtual audio stereo signal).

The audio source device 101 may generate the data signal to include encoded data that represents a mono downmix audio signal for the stereo signal. For example, the mono downmix audio signal may be generated as a weighted combination, and specifically as a weighted summation, of the channel signals of an input stereo signal. In many cases, the weights may be fixed and may specifically be the same for the two channel signals.

The generated mono downmix audio signal is encoded using a suitable mono audio encoding algorithm/standard to generate encoded audio data representing the mono downmix audio signal.

In addition to the mono downmix audio signal, the audio source device 101 generates spatial upmix parameters for upmixing the mono downmix audio signal to recreate the original stereo signal.

The spatial upmix parameters are generated to be indicative of/reflect relative properties of the channel signals of the stereo audio signal. In particular, the spatial upmix parameters may be indicated to include parameters that are indicative of at least one of relative intensities/levels of the stereo channels, relative (frequency domain) phases of the stereo channels, a relative time difference between the channels, and/or a correlation between the channels. Specifically, the audio source device 101 may generate spatial upmix parameters including one or more of an inter-channel intensity difference, inter-channel level difference, inter-channel time difference, inter-channel phase difference, and/or inter-channel correlation.

The data signal may specifically comprise a Parametric Stereo (PS) encoding of the stereo signal.

A classical PS downmix is calculated as: $$m=c\left(l+r\right)$$ where the parameter c is chosen such that the power of the stereo signal is preserved in the downmix, the power being defined using the 2-norm: $${‖m‖}^2={‖l‖}^2+{‖r‖}^2$$ and thus e.g.: $$c=\sqrt{\frac{{‖l‖}^2+{‖r‖}^2}{{‖l+r‖}^2}}$$

The PS parameters are specifically an Inter-channel Intensity Difference IID, an Inter-channel Correlation ICC, and in some cases an Inter-channel Phase Difference IPD parameter. These may specifically be defined as: $$\mathrm{IID}=\frac{{‖l‖}^2}{{‖r‖}^2}$$ $$\mathrm{ICC}=\frac{\left|〈l,r〉\right|}{\sqrt{{‖l‖}^2{‖r‖}^2}}$$ $$\mathrm{IPD}=\arg 〈l,r〉$$ where the complex-valued inner product is defined as: $$〈\mathrm{x},\mathrm{y}〉={\displaystyle \sum _{\forall i}{x}_i\cdot y_i^{\ast }}$$

The spatial upmix parameters are typically generated for specific time frequency tiles, and thus specifically each parameter value is generated/provided for a given frequency subband/interval and for a given time segment/interval.

The audio source device 101 may accordingly encode a stereo signal as encoded data representing a mono downmix audio signal of the stereo signal and associated spatial upmix parameters that are indicative of relative properties of the channels (channel signals) of the stereo signal. Specifically, the audio source device 101 may be arranged to generate a data signal comprising a conventional PS encoded stereo signal.

The system of FIG. 1 further comprises an audio apparatus which henceforth will be referred to as the audio render apparatus 103. The audio render apparatus 103 is arranged to receive the data signal generated by the audio source device 101. In the example, the audio render apparatus 103 and audio source device 101 are both coupled to a network 105 through which the data signal can be communicated and specifically through which it can be communicated from the audio source device 101 to the audio render apparatus 103. The network 105 may specifically be, or include, the Internet.

Thus, the receiver 101 receives a data signal which comprises encoded data for a mono downmix audio signal and spatial upmix parameters for upmixing the mono downmix audio signal to the stereo signal. The set of spatial upmix parameters comprises one or more parameters indicative of relative signal properties of channels of the stereo signal, and may specifically be indicative of a level/intensity difference between the channels of the stereo signal, a cross-correlation between the channels of the stereo signal, and/or a phase difference or time difference between the channels of the stereo signals. In many cases, the data signal may comprise ICC, IID and/or IPD parameters. The data signal may specifically comprise a PS (Parametric Stereo) encoded stereo signal. The audio source device 101 may be arranged to generate a data signal which comprises a stereo signal encoded in accordance with the Parametric Stereo (PS) specifications/standard. The receiver 201 may accordingly receive a representation of a stereo signal encoded by a mono downmix audio signal and spatial upmix parameters, and specifically a PS encoded stereo signal.

The audio render apparatus 103 is arranged to process the data signal to render a stereo signal. An audio render apparatus could render the stereo signal of the data signal using a conventional PS rendering approach based on a PS upmixing of the mono downmix audio signal using the PS spatial parameters. However, the audio render apparatus 103 of FIG. 1 uses a specific approach where point source components may be specifically considered, and where in many embodiments different/parallel paths process the received mono downmix audio signal in different ways to generate different stereo signal components which are then combined to generate the output stereo signal.

The approach is based on a signal model for the stereo signal and specifically is based on a consideration that the stereo signal can be represented by: $$l=\cos \left(\gamma \right)e^{{\mathit{j\varphi }}_l}x+n_l$$ $$r=\sin \left(\gamma \right)e^{{\mathit{j\varphi }}_r}x+n_r$$

This signal model essentially represents a consideration that the stereo signal corresponds to a combination of a directional signal and diffuse background audio. The directional signal x is an audio component that corresponds/reflects audio that is considered to originate from an audio source that has a point source property/characteristic and which thus has a well defined spatial origin. The audio of the directional signal thus corresponds to audio that can/should be rendered to be perceived to reach the listener from a specific direction. The directional signal may for each time frequency tile represent a single audio point source and thus may for each time frequency tile have a specific source position. The directional signal may in some cases/embodiments correspond to different audio point sources in different time frequency tiles, i.e. it is not required that all time frequency tiles of the directional source represent audio from the same point source. In some cases, the directional signal may correspond to different frequency components being reflected differently in the environment and thus may (e.g. in some frequency bands) correspond to early reflections of the point source within the acoustic environment.

In the signal model, the directional signal component x is phase shifted using two parameters ϕ_l and ϕ_r, and is further panned/positioned in the stereo image of the original stereo channels l and r. The panning is to an angle represented by the panning angle γ. Furthermore, a diffuse signal component is represented by diffuse residual signal components n_l and n_r of the respective left and right channels.

It is noted that the signal model description does not necessarily refer to a time-domain signal, but rather can alternatively or additionally refer to individual (potentially relatively small) frequency subbands. For example, the described signal model may individually apply to each of the frequency subbands for which separate spatial upmix parameters are provided.

In the described approach, the rendering of the stereo signal is based on a consideration of such a signal model and specifically the audio render apparatus 103 is arranged to render signal components corresponding to the directional signal x and the diffuse/residual signals n_l and n_r.

Accordingly, the audio render apparatus 103 may generate a signal representing the directional audio/signal, i.e. it may generate the directional signal component x from the received signal. A possibility for doing so might be to determine the directional signal component by processing the mono downmix audio signal based on the spatial parameters, and specifically could be to generate a directional signal by processing of a received Parametric Stereo mono downmix audio signal based on Parametric Stereo spatial parameters. Specifically, as will be described in more detail later, a directional signal may be estimated as a spectro-temporally shaped modification of the received mono signal. However, rendering of e.g. point audio sources based on such an estimated/derived directional signal tends to result in a suboptimal quality of the rendered audio.

The inventors have realized that a major cause of such degradation in quality may be due to it typically not being feasible to generate a sufficiently accurate estimate of the directional signal from the received mono downmix audio signal.

In particular, a time frequency directional signal may be estimated using two (typically complex-valued) coefficients for each time frequency interval where the coefficients are dependent on the inter-channel properties and thus are typically determined based on the spatial parameters: $$x=g_1\cdot l+g_2\cdot r$$

In most cases the gains g₁ and g₂ can be directly expressed as a function of the Parametric Stereo parameters (IID, ICC, IPD).

In a PS encoder, the mono downmix audio signal is typically constructed as: $$m=c\cdot \left(l+r\right)$$ where the parameter c is typically a function of the PS parameters.

So, whereas for the directional signal, the coefficients g₁ and g₂ may be, and typically are, different, the mono downmix audio signal for a PS encoder is typically based on an equal weighting of the channel signals. As such it is typically not feasible to recreate the desired directional signal from the mono downmix audio signal as it does not contain all information.

The issue could be addressed by the encoder side generating a downmix signal that is based on using different weights for the channels of the stereo signal, and which specifically is generated to correspond to the directional signal. However, the system of FIGs. 1-3 is arranged to use a different approach where the mono downmix audio signal is generated to be different from the directional signal, and which specifically may be generated as the mono downmix audio signal that is typical for PS encoding. Accordingly, in many embodiments, the encoder may be arranged to generate the mono downmix audio signal to have equal weighting for the channel signals of the stereo signal being downmixed. The mono downmix audio signal may specifically be a potentially scaled summation of the left and right channel of the stereo signal. However, further, the encoder is arranged to generate and include a difference signal which reflects differences between the mono downmix audio signal and the directional signal, and in many embodiments specifically differences between a prediction of the directional signal generated from the mono downmix audio signal and the directional signal. Such a difference signal may be generated and included in the data signal which is communicated to the renderer side. Typically, the difference signal may be generated to have a lower, and often much lower, data rate than the mono downmix audio signal and thus the additional overhead of including the difference signal may be very small.

At the renderer side, the difference signal may be utilized to generate an improved estimate of the directional signal which can be rendered e.g. as a point source thereby providing an improved perceived audio experience.

The approach may allow improved rendering of the stereo signal based on the indicated signal model. Further, this may be achieved while still allowing backwards compatibility and in particular in many cases while still using restricted downmix approaches and traditional spatial parameters. In particular, improved rendering of a stereo signal may often be achieved while using a PS compatible data signal. The data signal may thus be one that can be used for the improved rendering described in the following while still allowing rendering using standard conventional PS based renderings.

In more detail, the audio source device 101 of FIG. 1 is arranged to generate a data signal with data representing an audio stereo signal. The audio source device 101 is specifically arranged to generate the data signal to include encoded audio data for a mono downmix audio signal which represents a downmix of the stereo signal. In addition, spatial parameters that indicate relative properties between the channels of the stereo signal are included. Such spatial data may be appropriate for upmixing the mono downmix audio signal to recreate the downmix represented by the mono downmix audio signal.

As illustrated in FIG. 2, the audio source apparatus 101 comprises a receiver 201 which is arranged to receive audio components from which the mono downmix audio signal is generated. In many embodiments, the receiver 201 may directly receive a stereo signal which is to be represented by the data signal. In other embodiments, the receiver 201 may additionally or alternatively receive a number of audio components such as audio objects, mono signals, single source signals, multichannel signals etc. The audio components are fed to a downmixer 203 which proceeds to generate the mono downmix audio signal. In many embodiments and scenarios, the downmixer 203 may generate a mono downmix audio signal from a received stereo signal, e.g. simply by summing the channels of the stereo signal in accordance with a standard PS downmix approach. In other embodiments, the downmixer 203 may be arranged to generate a stereo signal from received audio components, such as e.g. by generating an intermediate stereo signal for each received audio component followed by a combination of the intermediate stereo signals. For example, a multi-channel signal may be downmixed to an intermediate stereo signal, an audio object may be panned to the stereo image of an intermediate stereo signal based on position data provided for the audio object, etc.

The audio source apparatus 101 further comprises a spatial parameter circuit 205 which is arranged to determine the spatial parameters for the stereo signal represented by the mono downmix audio signal. Specifically, the received or locally generated stereo signal may be fed to the parameter circuit 205 which may determine the spatial parameters from an analysis/processing of the stereo signal.

The spatial parameter circuit 205 may provide sets of frequency subband spatial parameters for the stereo signal where the sets of frequency subband spatial parameters are indicative of relative signal properties of the channels of the stereo signal. The frequency subband spatial parameters are provided for individual subbands of the stereo signal.

The spatial parameters are indicative of/reflect relative properties of the channel signals of the stereo audio signal. In particular, the spatial parameters may be indicated to include parameters that are indicative of at least one of relative intensities/levels of the stereo channels, relative (frequency domain) phases of the stereo channels, a relative time difference between the channels, and/or a correlation between the channels. Specifically, the spatial parameters may include one or more of an inter-channel intensity difference, inter-channel level difference, inter-channel time difference, inter-channel phase difference, and/or inter-channel correlation.

The spatial parameters may specifically be spatial parameters as used for encoding a stereo signal using a Parametric Stereo (PS) encoding of the stereo signal.

A classical PS downmix is calculated as: $$m=c\left(l+r\right)$$ where the parameter c is chosen such that the power of the stereo signal is preserved in the downmix, the power being defined using the 2-norm: $${‖m‖}^2={‖l‖}^2+{‖r‖}^2$$ and thus e.g.: $$c=\sqrt{\frac{{‖l‖}^2+{‖r‖}^2}{{‖l+r‖}^2}}$$

The PS parameters are specifically an Inter-channel Intensity Difference IID, an Inter-channel Correlation ICC, and in some cases an Inter-channel Phase Difference IPD parameter. These may specifically be defined/determined as: $$\mathrm{IID}=\frac{{‖l‖}^2}{{‖r‖}^2}$$ $$\mathrm{ICC}=\frac{\left|〈l,r〉\right|}{\sqrt{{‖l‖}^2{‖r‖}^2}}$$ $$\mathrm{IPD}=\arg 〈l,r〉$$ where the complex-valued inner product is defined as: $$〈\mathrm{x},\mathrm{y}〉={\displaystyle \sum _{\forall i}{x}_i\cdot y_i^{\ast }}$$

The spatial parameters are typically provided for specific time frequency tiles, and thus specifically each parameter value is generated/provided for a given frequency subband and for a given time segment.

The spatial parameter circuit 205 may determine spatial parameters that are indicative of relative properties of the channels (channel signals) of the stereo signal.

In many embodiments, the spatial parameter circuit 205 may receive the input stereo signal and process/analyze this to generate the spatial parameters. Specifically, the spatial parameter circuit 205 may calculate the IID, ICC, and IPD values in accordance with the formulas indicated above. Thus, in many embodiments, the audio render apparatus may simply receive a stereo signal and therefrom may generate spatial parameters. In such a scenario, e.g. instead of the IID, ICC and IPD, the following inner products may alternatively or additionally be used as spatial parameters: $$〈l,r〉={\displaystyle {\sum }_{\forall i}{l}_i\cdot r_i*}$$ $${‖l‖}^2=〈l,l〉={\displaystyle {\sum }_{\forall i}{l}_i\cdot l_i*}$$ $${‖r‖}^2=〈r,r〉={\displaystyle {\sum }_{\forall i}{r}_i\cdot r_i*}$$

The spatial parameters may comprise sets of spatial parameters, each set of spatial parameters comprising at least one of: a level difference parameter indicative of a level difference between channels of the multichannel audio signal; a correlation parameter indicative of a coherence between channels of the multichannel audio signal; a timing difference parameter indicative of a timing difference between channels of the multichannel audio signal, and a phase difference parameter indicative of a phase difference between channels of the multichannel audio signal.

The audio source device 101 further comprises a data signal circuit 207 which generates the audio data signal and specifically it generates the audio data signal to include data representing the mono downmix audio signal and data representing the spatial parameters. The data signal circuit 207 may specifically include an audio signal encoder arranged to encode the mono downmix audio signal. It will be appreciated that any suitable audio signal encoding algorithm and approach may be used, such as an Advanced Audio Coding (AAC) mono encoding algorithm. Likewise, the spatial parameters may be encoded using a suitable encoding approach.

The data signal circuit 207 may generate an output data signal in accordance with any suitable format, and may specifically generate the output data signal to follow a suitable standard for an audio signal.

The audio source device 101 further includes a directional signal generator 209 which is arranged to determine a directional signal for the stereo signal. The directional signal is generated to represent/estimate a point source audio component of the stereo signal. The directional signal may be generated to seek to represent parts of the stereo signal which can be considered to have an origin with point source properties. Thus, the directional signal is generated to include audio components that correspond to audio that have a specific position/direction of origin.

The directional signal may in some cases be generated to represent/estimate audio that has a specific position in the stereo image of the stereo signal, and thus which corresponds to audio reaching the capture position (for the stereo signal) from a single direction.

The directional signal may accordingly be generated to estimate audio that is linked with audio sources that are spatially limited/small/localized and which specifically are generated by point sources. It may thus seek to extract such audio components/parts from the stereo signal to separate point source audio from audio with a spatial extension, such as e.g. background audio, ambient audio, etc.

In some cases, the directional signal generator 209 may be arranged to estimate an audio component from a single point source and specifically the directional signal may be generated to represent/estimate a single point source. In such an approach, audio from one specific direction may be identified/estimated and represented by the directional signal.

In most embodiments, however, the directional signal is generated to represent/estimate audio that has a point source origin but not necessarily from the same single point source. The directional signal may accordingly in many embodiments represent/estimate audio from a plurality of point sources.

In particular, in many embodiments, the processing to generate the directional signal may be performed in time frequency intervals/tiles and thus different frequency time intervals/tiles may result in different point sources being detected. For example, two or more point audio sources may simultaneously be active in the audio scene with different point sources being dominant in different frequency bands. Accordingly, different point sources may be detected/extracted/estimated in different frequency bands and the directional signal may be generated to represent audio from different point sources in different frequency bands.

It will be appreciated that different algorithms and approaches may be used for determining the directional signal and that any suitable algorithm may be used without detracting from the invention.

As an example, the directional signal may be generated to correspond to a principal component determined by performing a principal component analysis on the stereo signal (and specifically on frequency domain samples of the stereo signal). In more detail, the directional signal may follow a PCA orthogonalization according to: $$\left(\begin{array}{c}x\\ d\end{array}\right)=\left(\begin{array}{cc}\mathit{cos\alpha }& \mathit{sin\alpha }\\ -\mathit{sin\alpha }& \mathit{cos\alpha }\end{array}\right)\left(\begin{array}{cc}{e}^{j\cdot {\varphi }_l}& 0\\ 0& e^{j\cdot {\varphi }_r}\end{array}\right)\left(\begin{array}{c}l\\ r\end{array}\right)$$ where the parameters α, ϕ_l and ϕ_r are chosen in such a way that the directional signal x (the principal component) and the residual signal d become orthogonal: $$〈x,d〉=0$$

It can be shown that these parameters can be expressed as a function of the inter-channel properties: $$\alpha =\frac{1}{2}\mathit{arctan}\left(\frac{2\mathit{ICC}}{\sqrt{\mathit{IID}}-1/\sqrt{\mathit{IID}}}\right)$$ $${\varphi }_r-{\varphi }_l=\mathit{IPD}$$

So, the directional signal is determined as: $$x=\mathit{cos\alpha }\cdot e^{j\cdot {\varphi }_l}\cdot l+\mathit{sin\alpha }\cdot e^{j\cdot {\varphi }_r}\cdot r$$

The signal to be transmitted thus becomes: $$x_{\mathit{res}}=x-m$$

In this way the directional signal can be reconstructed in the decoder and used for the purpose of rendering.

The directional signal generator 209 is coupled to a difference signal circuit 211 which is arranged to generate a difference signal which is indicative of the difference between the mono downmix audio signal and the directional signal. The difference signal circuit 211 is fed both the mono downmix audio signal and the directional signal and proceeds to determine a difference signal based on a comparison between these signals. In many embodiments, the difference signal may be generated directly by subtracting the mono downmix audio signal from the directional signal or vice versa.

It will be appreciated that in other embodiments other approaches may be used to generate a signal that reflects the difference between the mono downmix audio signal and the directional signal. For example, as will be described further later, instead of transmitting the difference signal: $$x_{\mathit{res}}=x-m$$ the difference between the directional signal and an approximation of the directional signal from the mono signal is transmitted: $$x_{\mathit{res}}=x-x\prime$$

The approximation of the directional signal is realized by a (time- and frequency-variant) gain parameter applied to the mono signal: $$x\prime =g_x\cdot m$$ with $$g_x=\sqrt{\frac{{‖x‖}^2}{{‖m‖}^2}}$$

For the case of the PCA directional signal, it can be shown that the gain parameter can be expressed again as a function of the inter-channel properties: $$g_x=\sqrt{\frac{\sqrt{4\cdot {\mathit{ICC}}^2\cdot \mathit{IID}+{\left(\mathit{IID}-1\right)}^2}}{\mathit{IID}+1}}$$

This means that the approximation of the directional signal can be made both at encoder side as well as decoder side without the need for transmitting additional parameters. The benefit of transmitting the difference of the directional signal and the approximation of the directional signal is that this signal has a significantly smaller variance than the difference of the directional signal and the mono signal. As a result the amount of bits to transmit is significantly smaller in the case of transmitting the difference of the directional signal and the approximation of the difference signal from the mono signal and the inter-channel properties.

The difference signal is fed to the data signal circuit 207 which is arranged to include it in the output data signal. The data signal circuit 207 typically includes a suitable encoder to generate encoded data representing the difference signal. In most cases, the encoding may be a non-lossy encoding, but in some applications and embodiments the encoding of the difference signal may be a lossy encoding.

There may typically be a strong correlation between the mono downmix audio signal and the directional signal and accordingly the difference signal may have a substantially lower power and amplitude than the mono downmix audio signal (and the directional signal). This may allow the encoding of the difference signal to have a much lower data rate than the mono downmix audio signal (or indeed that would be required to encode the difference signal with a corresponding quality). Indeed, the approach can be considered to generate an audio data signal representing a stereo signal by an encoded representation of a mono downmix audio signal and associated spatial parameters, and with the audio data signal additionally including an encoding of a difference signal supporting an encoding of a directional signal. Further, the overhead required to include encoding supporting the directional signal may be kept relatively low.

In many embodiments, an encoded data rate for the difference signal may be no more than 50%, 25%, 10%, 5%, or even 1% of the encoded data rate for the mono downmix audio signal.

In many embodiments, the audio source device 101 may accordingly be arranged to estimate the difference between the mono downmix audio signal and the directional signal, encode the difference at a low bit-rate, and transmit the data as a side channel to the audio render apparatus 103. In some embodiments, the left and right stereo input signals may for example be fed to two filterbanks, e.g. hybrid QMF banks, and the resulting left and right frequency domain signals may be used for stereo/spatial parameter estimation. Furthermore, the left and right frequency domain signals may be used to form a downmix signal and a directional signal (X), optionally making use of the extracted stereo parameters. The difference between the resulting mono downmix and the directional signal can be determined and both the mono downmix audio signal (M) and the difference between the mono downmix audio signal and the directional signal, the difference signal (X_res = X - M) may be fed through a synthesis filterbank, resulting in time domain signals m and x_res respectively. The time domain signals can then be coded using existing techniques, such as waveform coding. Finally, bit-streams, consisting of the encoded mono downmix audio signal, the encoded spatial parameters, and the encoded differential signal can then be multiplexed into a single output bit-stream/data signal.

It will be appreciated that in many embodiments, the mono downmix audio signal used to generate the difference signal may be a decoded version generated by decoding the encoded mono downmix audio signal of the output data signal. This may provide improved performance in many embodiments and may allow encoding/decoding errors to be reflected in the difference signal thereby allowing them to be compensated in a directional signal generated by the audio render apparatus 103. Such an approach may be particularly advantageous in embodiments where the decoding operation of the audio render apparatus 103 can be known and replicated by the audio source device 101.

FIG. 3 shows examples of elements of the audio render apparatus 103.

The audio render apparatus 103 comprises a receiver 301 which is arranged to receive the data signal from the audio source device 101. Thus, the receiver 301 receives a data signal comprising encoded data for a mono downmix audio signal of a stereo signal. In addition, the data signal includes spatial upmix parameters for upmixing the mono downmix audio signal to the stereo signal where the spatial upmix parameters are indicative of relative signal properties of channels of the stereo signal. The spatial upmix parameters may as mentioned specifically be inter-channel time, phase, level, intensity differences and/or inter-channel correlation measures. Further, the data signal includes data describing a difference signal representing/describing a difference between the mono downmix audio signal and a directional signal where the directional signal represents a point source audio component of the stereo signal.

The receiver 301 is coupled to a decoder 303 which is arranged to receive the encoded data representing the mono downmix audio signal and to decode this data to generate the mono downmix audio signal. It will be appreciated that any suitable method for encoding and decoding the mono downmix audio signal may be used and in particular that any suitable standardized encoding format and algorithm may be used.

The receiver 301 may be arranged to receive a time domain audio signal and/or a frequency domain audio signal version/representation of the mono downmix audio signal. In some cases, the received data signal may include the mono downmix audio signal in only one representation, i.e. the data signal may include only one of the frequency domain audio signal and the time domain audio signal. In such cases, the received data signal may be transformed to the other domain as appropriate. Thus, in some cases, a received data signal may include a time domain audio signal being the time domain representation of the mono downmix audio signal, and a time to frequency domain transformer may from this generate the frequency domain audio signal for the mono downmix audio signal. In some cases, a received data signal may include a frequency domain audio signal being the frequency domain representation of the mono downmix audio signal and a frequency to time domain transformer may from this generate the time domain audio signal for the mono downmix audio signal if necessary.

In particular, in some embodiments, the receiver may comprise a filter bank which is arranged to generate a frequency subband representation of a received time domain mono downmix audio signal. The receiver 301 may comprise a filter bank that is applied to the mono downmix audio signal such that it is divided into frequency subbands.

The filter bank may be Quadrature Mirror Filter (QMF) bank or may e.g. be implemented by a Fast Fourier Transform (FFT), but it will be appreciated that many other filter banks and approaches for dividing an audio signal into a plurality of subband signals are known and may be used. The filterbank may specifically be a complex-valued pseudo QMF bank, resulting in e.g. 32 or 64 complex-valued sub-band signals.

The processing is furthermore typically performed in time segments or time slots. In most embodiments, the audio signal is divided into time intervals/segments with a conversion to the frequency/subband domain by applying e.g. an FFT or QMF filtering to the samples of each signal. For example, each channel of the downmix audio signal may be divided into time segments of e.g. 2048, 1024, or 512 samples. These signals may then be processed to generate samples for e.g. 64, 32 or 16 subbands. Thus, a set of samples may be determined for each subband of the mono downmix audio signal.

It should be noted that the number of time domain samples is not directly coupled to the number of subbands. Typically, for a so-called critically sampled filterbank of N bands, every N input samples will lead to N sub-band samples (one for every sub-band). An oversampled filterbank will produce more output samples. E.g. for every N input samples, it would generate k*N output samples, i.e., k consecutive samples for every band.

In some embodiments, the subbands are generated to have the same bandwidth but in other embodiments subbands are generated to have different bandwidths, e.g. reflecting the sensitivity of human hearing to different frequencies.

For example, the receiver 301 may employ a hybrid filterbank with logarithmic filter band center-frequency spacings that follow that of human perception similar to equivalent rectangular bandwidths (ERBs). In order to compensate for the delay of the filtering by the small filter bank, a delay may be introduced for higher frequency subbands.

As a specific example, a time-domain signal x[n] may be fed through a downsampled complex-exponential modulated QMF bank with K bands. Each frame of 64 time domain samples x[n] results in one slot of QMF samples X[k, l] with k = (0, ..., K - 1) at slot l. The lower slots may then be filtered by additional complex-modulated filterbanks splitting the lower bands further. The higher slots are delayed ensuring that the filtered mono downmix audio signals of the lower bands are in sync with the higher bands as the filtering introduces a delay. This finally results in a structure where for every 64 time-domain samples x[n], one slot m of hybrid QMF samples Y[k, l] is produced with k = (0, ... , L - 1) at slot l, e.g. with a total number of hybrid bands M = 77.

Thus, in many embodiments, the signals and the processing may be performed in subbands and for individual segments. Such blocks of a frequency interval/subband in a given time interval/segment will also be referred to as time frequency segments/tiles.

The mono downmix audio signal is fed to a compensator 305 which also receives the difference signal. The compensator 305 is arranged to compensate/modify the mono downmix audio signal based on the difference signal and is specifically arranged to generate a modified mono downmix audio signal by modifying the mono downmix audio signal based on the difference signal such that it more closely resembles the directional signal. The compensator 305 may compensate the mono downmix audio signal based on the difference signal to generate a modified mono downmix audio signal having for which a difference to the directional signal is reduced, and thus the modified mono downmix audio signal is generated to more closely correspond to/resemble the directional signal.

As a specific example, the compensator 305 may in some embodiments simply add the difference signal to the mono downmix audio signal to generate the modified mono downmix audio signal. Thus, in some embodiments, the audio apparatus 101 may directly generate the data signal to include (low rate) data that describes the difference between the mono downmix audio signal and the generated directional signal. The audio render apparatus 103 may then apply this difference to the received mono downmix audio signal to generate a modified mono downmix audio signal which is a local replica of the directional signal that was generated by the audio source device 101. This local replica generated by the audio source device 101 will correspond closely to the directional signal generated at the audio source device 101, and typically will only differ by potential distortions due to encoding/decoding and communication.

As will be described in more detail later, in other embodiments, instead of transmitting the difference between the directional signal and the mono downmix audio signal , the difference between the directional signal and an estimate of the directional signal generated from the mono downmix audio signal may be transmitted. As shown above, the directional signal can be estimated from the mono signal as x' = g_x · m, where the parameter g_x is a function of the inter-channel properties that are transmitted to the decoder. The difference signal x - x' generally has a lower variance than the difference signal x - m, thereby requiring less bits to be transmitted.

In yet another embodiment, instead of transmitting the difference between the directional signal and the estimated directional signal, the aforementioned gain g_x may be applied to the directional signal, and the difference x/g_x - m may be transmitted.

The modified mono downmix audio signal is fed to a first renderer 307 which is arranged to render the modified mono downmix audio signal to generate a first intermediate stereo signal. The rendering by the first renderer 307 (also referred to as a first rendering) is a directional rendering which renders the first intermediate stereo signal with a given direction/position in the stereo image of the first intermediate stereo signal. The first rendering may specifically render the mono downmix audio signal as a point source with a given direction/position in the stereo image.

The first renderer 307 is coupled to a direction determining circuit 309 which is arranged to determine a direction γ' which is fed to the first renderer 307 resulting in this rendering the modified mono downmix audio signal from this position/direction. Thus, the first rendering is specifically such that the modified mono downmix audio signal in the first intermediate stereo signal is perceived as a point audio source positioned in the direction corresponding to the direction γ' determined by the direction determining circuit 309. The direction γ' will also be referred to as the rendering direction or rendering angle.

The direction determining circuit 309 is arranged to determine the direction from the received spatial upmix parameters. The spatial upmix parameters provide information on the relationship between the channels of the stereo signal that is downmixed and as such provide information of the position/orientation of the audio, and specifically of a dominant signal component in the stereo image of the stereo signal. For example, for a PS encoded signal, the spatial upmix parameters provide information of the position of the dominant signal component in the stereo signal, and specifically it provides information of an orientation angle for the dominant signal.

The direction determining circuit 309 may specifically determine the rendering direction γ' from the spatial upmix parameters. The rendering direction will typically be determined on a frequency tile basis, and specifically in frequency subbands and time segments matching those for which the spatial upmix parameters are provided.

The first renderer 307 may accordingly proceed to render the modified mono downmix audio signal such that is perceived from the given direction and it specifically achieves this directional rendering by applying a channel transfer function to the modified mono downmix audio signal with the channel transfer function generating the intermediate stereo signal from the mono downmix audio signal. The channel transfer function may specifically include a sub-transfer function for each channel, i.e. it may include one (sub)transfer function for generating a left channel signal and one (sub)transfer function for generating the right channel signal.

In many cases, the transfer function may be provided as a set of complex weights for the different subbands of a frequency representation of the modified mono downmix audio signal. The audio apparatus may perform many or all of the operations in the frequency domain and thus the transfer function may also be expressed and applied in the frequency domain. For example, for each frequency subband of the representation of the mono downmix audio signal, the transfer function may provide a complex weight for each of the output channels and a frequency representation of the first intermediate stereo signal may be generated by applying/multiplying the subband samples of the mono downmix audio signal by these weights to generate the subband samples of the first intermediate stereo signal.

The first transfer function is determined to correspond to the desired direction, i.e. it reflects the mapping from the mono downmix audio signal to the channels of the first intermediate stereo signal such that it is perceived as/corresponds to an audio source at a position in the stereo image corresponding the rendering direction/angle.

For example, in some cases, the transfer function for a given direction may correspond to a panning of the mono downmix audio signal to the given direction in the stereo image.

In many embodiments, the first rendering may be a binaural rendering and the first intermediate stereo signal may be a binaural stereo signal providing an enhanced spatial experience/perception when heard through headphones. Thus, the first renderer 307 may specifically be a binaural audio renderer which generates binaural audio signals for the left and right ear of a user. Binaural audio signals are generated to provide a desired spatial experience and are typically reproduced by headphones or earphones that specifically may be part of a headset worn by a user (the headset typically also comprises left and right eye displays).

Thus, in many embodiments, the audio rendering by the first renderer 307 is a binaural render process using suitable binaural transfer functions to provide the desired spatial effect for a user wearing a headphone. For example, the first renderer 307 may be arranged to generate an audio component to be perceived to arrive from a specific position using binaural processing.

Binaural processing is known to be used to provide a spatial experience by virtual positioning of sound sources using individual signals for the listener's ears. With an appropriate binaural rendering processing, the signals required at the eardrums in order for the listener to perceive sound from any desired direction can be calculated, and the signals can be rendered such that they provide the desired effect. These signals are then recreated at the eardrum using either headphones or a crosstalk cancelation method (suitable for rendering over closely spaced speakers). Binaural rendering can be considered to be an approach for generating signals for the ears of a listener resulting in tricking the human auditory system into perceiving that a sound is coming from the desired positions.

The binaural rendering is based on binaural transfer functions which vary from person to person due to the acoustic properties of the head, ears and reflective surfaces, such as the shoulders. Binaural transfer functions may therefore be personalized for an optimal binaural experience. For example, binaural filters can be used to create a binaural recording simulating multiple sources at various locations. This can be realized by convolving each sound source with the pair of e.g., Head Related Impulse Responses (HRIRs) that correspond to the position of the sound source.

A well-known method to determine binaural transfer functions is binaural recording. It is a method of recording sound that uses a dedicated microphone arrangement and is intended for replay using headphones. The recording is made by either placing microphones in the ear canal of a subject or using a dummy head with built-in microphones, a bust that includes pinnae (outer ears). The use of such dummy head including pinnae provides a very similar spatial impression as if the person listening to the recordings was physically present during the recording.

By measuring e.g., the responses from a sound source at a specific location in 2D or 3D space to microphones placed in or near the human ears, the appropriate binaural filters can be determined. Based on such measurements, binaural filters reflecting the acoustic transfer functions to the user's ears can be generated. The binaural filters can be used to create a binaural recording simulating multiple sources at various locations. This can be realized e.g., by convolving each sound source with the pair of measured impulse responses for a desired position of the sound source. In order to create the illusion that a sound source is moving around the listener, a large number of binaural filters is typically required with a certain spatial resolution, e.g., 10 degrees.

The head related binaural transfer functions may be represented e.g., as Head Related Impulse Responses (HRIR), or equivalently as Head Related Transfer Functions (HRTFs) or, Binaural Room Impulse Responses (BRIRs). The (e.g., estimated or assumed) transfer function from a given position to the listener's ears (or eardrums) may for example be represented in the frequency domain in which case it is typically referred to as an HRTF or BRTF, or in the time domain in which case it is typically referred to as a HRIR or BRIR. In some scenarios, the head related binaural transfer functions are determined to include aspects or properties of the acoustic environment and specifically of the environment in which the measurements are made, whereas in other examples only the user characteristics are considered. Examples of the first type of functions are the BRIRs and BRTFs.

The audio render apparatus 103 comprises a store 311 which stores directional transfer functions for different directions. The directional transfer function for a given direction represents the mapping of a mono audio signal to stereo channels such that the mono audio signal is positioned in the given direction in a stereo image of the stereo channels. Thus, applying the directional transfer function for a given direction to the mono downmix audio signal may generate a stereo signal representing the mono downmix audio signal as an audio source positioned in the given direction. The mapping may in some cases be a time domain mapping (such as a gain, filter or other transfer function) or may in many cases be a frequency domain mapping, such as a set of parameter values/scale values (typically complex values) for different subbands. In the latter case, a frequency domain intermediate stereo signal may be generated by for each subband multiplying the subband sample of the mono downmix audio signal with respectively a complex value for that subband for a first channel of the intermediate stereo signal and with a complex value for that subband for a second channel of the intermediate stereo signal.

For example, in examples where a panning is performed in the horizontal 2D plane, the store 311 may comprise panning parameters for different directions. For example, panning parameters for azimuth angles in a 0-360° interval may be provided for each 1° angle increment. The first renderer 307 may be coupled to the store 311 and be arranged to extract the directional transfer function for the rendering direction and then proceed to perform the rendering using the extracted directional transfer function. The rendering of the (potentially gain compensated) mono downmix audio signal may accordingly be rendered such that it is positioned/perceived in the stereo image to arrive from the rendering position.

It will be appreciated that the store 311 may not have directional transfer function stored for the desired rendering direction. In such cases, the first renderer 307 may be arranged to retrieve the nearest directional transfer function from the store 311 and use this for rendering. In such cases, the rendering direction may be considered to correspond to the direction for the retrieved directional transfer function, i.e. the rendered direction may be a quantized value γ' of the desired rendering direction determined by the direction determining circuit 309.

In other embodiments, the first renderer 307 may be arranged to estimate a desired directional transfer function for a desired rendering direction by interpolating between two directional transfer functions from the store 311 corresponding to the two rendering angles nearest to the desired rendering direction determined by the direction determining circuit 309.

In most embodiments, the first renderer 307 is as mentioned arranged to perform a binaural rendering and the directional transfer functions stored in the store 311 are binaural transfer functions. Thus, the store may store data describing binaural transfer functions for different directions. The binaural transfer functions may for example be HRTFs, BRIRs, or HRIRs. The store 311 may specifically store frequency subband complex values for each channel for each frequency subband for a range of different frequencies. The first renderer 307 may thus perform the binaural rendering by multiplying the subband samples of the mono downmix audio signal with the corresponding subband coefficients/complex values of the selected binaural transfer function to generate subband sample values of the intermediate binaural stereo signal.

It will be appreciated that in many embodiments, the directional transfer functions may be stored as a plurality of functions linked with different directions. For example, the store 311 may be a look-up table which can receive the rendering direction as an index and provide a set of values of the directional transfer function for that direction. The directional transfer function may for example be represented by individual subband values/coefficients, or may e.g. in other embodiments be represented by e.g. parameter values defining the directional transfer function operation (e.g. coefficients for the transfer function), a mathematical description/function from which suitable values of the transfer function can be generated etc.

Thus, the audio render apparatus 103 comprises a processing path which generates an intermediate stereo signal comprising the modified mono downmix audio signal represented as an audio source at a specific position in the spatial image of the first intermediate stereo signal. The mono downmix audio signal may typically be represented as a point audio source at the given direction. The rendering is adaptive with the direction being given by the spatial parameters and thus is dynamically adapted to reflect the characteristics of the stereo signal.

Further, the spatially definite and directional rendering is specifically of a directional signal representing point source audio. Thus, an improved spatial rendering of point source audio can be achieved with point source audio typically being rendered with a higher audio quality.

The generated intermediate stereo signal is fed to an output circuit 313 which is arranged to generate an output stereo signal which includes the intermediate stereo signal. As will be described more in the following, the output stereos signal may be generated to include other signal components, including audio components representing non-point source audio, such as background or ambient sounds.

In particular, the audio render apparatus 103 comprises a second processing path which generates a second intermediate stereo signal. The mono downmix audio signal is fed to a decorrelator 315 which is arranged to apply a decorrelation to the mono downmix audio signal to generate a first decorrelated mono downmix audio signal. It will be appreciated that a large number of different algorithms and functions for decorrelating an audio signal is known to the skilled person, and that any suitable approach or algorithm may be used without detracting from the invention.

The first decorrelated mono downmix audio signal is fed to a second renderer 317 which is arranged to perform a second rendering being a rendering of the first decorrelated mono downmix audio signal to generate a second intermediate stereo signal. However, in contrast to the first rendering process, the second rendering process is a predetermined rendering which is not dependent on the spatial parameters, and which typically is not depending on properties of the stereo signal. The second rendering may typically be a diffuse rendering seeking to generate the second intermediate stereo signal to provide a perception of a more diffuse and spatially less definite audio source. The second rendering is specifically a predetermined rendering employing a predetermined mapping of the decorrelated mono downmix audio signal to channel signals of the second intermediate stereo signal.

As a specific example, the second rendering may typically generate the second intermediate stereo signal by simply mapping the first decorrelated mono downmix audio signal to two phase inverse signals, i.e. the second intermediate stereo signal may be generated with the first decorrelated mono downmix audio signal being mapped to both channels but with a 180° phase offset between them (the first decorrelated mono downmix audio signal may specifically be inverted for one of the channels). For example, in some embodiments, the first decorrelated mono downmix audio signal may be mapped to the right and left signals of the second intermediate stereo signal but with the mapping being 180° out of phase for the two channels of the second intermediate stereo signal.

The first renderer 307 and second renderer 317 are coupled to a combiner 313 which is arranged to combine at least the first intermediate stereo signal and the second intermediate stereo signal to generate an output stereo signal. In many embodiments, the combiner 313 may be arranged to combine/sum the samples/values of the individual channels of the first and second intermediate stereo signals to generate the samples/values of the output stereo signal. In many cases, the combination may be performed by combining/summing subband values of the intermediate stereo signals. In other embodiments, the combination may be performed in the time domain by combining/summing time domain values of the intermediate stereo signals.

In many embodiments, the combination of the intermediate stereo signals may be by a (possibly weighted) combination/summation of corresponding channel signals for the first intermediate stereo signal and the second intermediate stereo signal.

The audio render apparatus 103 accordingly generates an output stereo signal which is the combination of a directional rendering putting an audio source at a desired position as determined from the received spatial upmix parameters, and of a predetermined rendering providing a more diffuse and decorrelated perception of the corresponding audio source. The approach provides two parallel rendering processes/paths for the mono downmix audio signal with the rendered results being combined to generate the output stereo signal.

In addition to the described flexible and adaptable generation of the output signal to provide an output stereo signal that includes both a directionally rendered component and a more diffuse/predeterminedly rendered component, the audio render apparatus 103 is arranged to flexibly and adaptively adapt/control the relative level between these components.

An example of such an approach is illustrated in FIG. 3. In addition to the audio render apparatus of FIG. 2, this apparatus includes a second path for rendering the diffuse/ non-directional component.

In many embodiments, the audio source device 101 may be arranged to generate multiple signals representing non-directional signal components. In particular, the decorrelator 315 may be arranged to generate a second decorrelated mono downmix audio signal which is then rendered by the second renderer 317 generating a third intermediate stereo signal. The third intermediate stereo signal is then combined with the first intermediate stereo signal and the second intermediate stereo signal.

The second decorrelated mono downmix audio signal is generated to have low correlation with both the mono downmix audio signal and the first decorrelated mono downmix audio signal, and typically will be uncorrelated with both of these. Accordingly, the output stereo signal includes two components for the residual signal. The two signals may be rendered differently such that an improved diffuseness is perceived. For example, the second decorrelated mono downmix audio signal may be rendered from one position (e.g. fully in a first channel of the output stereo signal or from a first virtual speaker position for a binaural signal) with the first decorrelated mono downmix audio signal being rendered from a second position (e.g. fully in a second channel of the output stereo signal or from a second virtual speaker position for a binaural signal).

The approach may typically provide an improved performance and perception relative to what can be achieved with a single decorrelated signal representing the non-directional component. In particular, the approach may more accurately represent the signal model indicated previously with different and uncorrelated diffuse signal components n_l and n_r.

The predetermined rendering of the second renderer 317 may as previously mentioned simply be achieved by rendering the corresponding decorrelated signal in one channel of the corresponding intermediate stereo signal, and with no signal being included in the other channel. For example, the second decorrelated mono downmix audio signal may be rendered in the left channel of the second intermediate stereo signal and the first decorrelated mono downmix audio signal may be rendered in the right channel of the third intermediate stereo signal.

In some embodiments where binaural processing is used, each of the decorrelated signals may be rendered from a specific position, such as each decorrelated signal being rendered from a different virtual position, such as for example from different virtual positions.

In some embodiments, the rendering for a decorrelated signal, such as the rendering of the first decorrelated mono downmix audio signal, may be to position the signal at a specific position.

In many embodiments, the rendering of a decorrelated mono downmix audio signal may be performed by the renderer 317 retrieving a set of directional transfer functions from the store 311 and rendering the decorrelated mono downmix audio signal using the retrieved transfer function(s).

In many embodiments, the audio render apparatus 103 may be arranged to extract a directional transfer function for a single predetermined direction and render the decorrelated mono downmix audio signal using this directional transfer function. Accordingly, (each of) the decorrelated mono downmix audio signal(s) may be rendered from one predetermined direction/position, such as a direction/position corresponding to a virtual speaker position.

An example of subband parametric rendering may e.g. result in left and right signals: $$l=g_x\cdot m_d\cdot G_l\left[f\left(\gamma \right)\right]\cdot e^{{\mathit{j\varphi }}_l\left[f\left(\gamma \right)\right]}+g_n\cdot H_1\left\{m\right\}\cdot G_l\left[{\beta }_l\right]\cdot e^{{\mathit{j\varphi }}_l\left[{\beta }_l\right]}+g_n\cdot H_2\left\{m\right\}\cdot G_l\left[{\beta }_r\right]\cdot e^{{\mathit{j\varphi }}_l\left[{\beta }_r\right]}$$ $$\begin{array}{l}r=g_x\cdot m_d\cdot G_r\left[f\left(\gamma \right)\right]\cdot e^{{\mathit{j\varphi }}_r\left[f\left(\gamma \right)\right]}+g_n\cdot H_1\left\{m\right\}\cdot G_r\left[{\beta }_l\right]\cdot e^{{\mathit{j\varphi }}_r\left[{\beta }_l\right]}+g_n\cdot H_2\left\{m\right\}\cdot G_r\left[{\beta }_r\right]\\ \cdot e^{{\mathit{j\varphi }}_r\left[{\beta }_r\right]}\end{array}$$ where G_l, G_r, ϕ_l, ϕ_r form the parametric HRIRs, f(γ) is a mapping function converting the estimated angles (orientation direction y) to HRIR direction angles, β_l and β_r are two pre-determined angles and H₁{.} and H₂{.} are two mutually independent decorrelators, m is the received mono downmix audio signal and m_d is the directional signal generated by the compensator 305, and g_x and g_n are suitable scaling factors.

In some embodiments, the renderer 217 may retrieve directional transfer functions for a plurality of predetermined directions and it may use multiple directional transfer functions in performing the predetermined rendering. For example, different directional transfer functions may be used for different frequency subbands. This may provide a more diffuse perception with the audio being generated such that it is perceived from different directions for different subbands thereby resulting in a perception of a more distributed and spread audio source.

Such approaches may be used both in embodiments in which a single decorrelated mono downmix audio signal is generated and rendered, or indeed in cases where multiple decorrelated mono downmix audio signals are generated and rendered. In the latter case, the sets of predetermined directions for the different decorrelated mono downmix audio signals are different in order to enhance the perceived diffuseness of the non-directional signal component.

The renderer 317 may generate the second intermediate stereo signal using a first set of directional transfer functions retrieved from the store 311 for a first set of predetermined directions, and may generate the third intermediate stereo signal using a second set of directional transfer functions retrieved from the store 311 for a second set of predetermined directions where the first set of set of predetermined directions is different from the second set of predetermined directions.

In particular, the directional transfer functions may be binaural transfer functions and the renderer 317 may be arranged to perform binaural rendering to generate the first intermediate stereo signal using binaural impulse response values for a first set of predetermined directions and may be arranged to perform binaural rendering to generate the first intermediate stereo signal using binaural impulse response values for a second set of predetermined directions where the first set of predetermined directions are different from the second set of predetermined directions.

In many cases, the use of multiple directional transfer functions may be achieved by using directional transfer functions for different directions in different frequency subbands.

Thus, instead of rendering the diffuse/ non-directional signals using fixed angles, e.g. mimicking a virtual stereo speaker setup, the diffuse signals may also be rendered using composite, e.g. pre-calculated HRIRs for many sources/directions, e.g. spread over a (part of a) circle, or (part of) a sphere. $$\begin{array}{l}l=g_x\cdot m_d\cdot G_l\left[f\left(\gamma \right)\right]\cdot e^{{\mathit{j\varphi }}_l\left[f\left(\gamma \right)\right]}+g_n\cdot H_1\left\{m\right\}\cdot G_{l,\mathit{comp}}\cdot e^{{\mathit{j\varphi }}_{l,\mathit{comp}}}+g_n\cdot H_2\left\{m\right\}\cdot G_{l,\mathit{comp}}\\ \cdot e^{{\mathit{j\varphi }}_{l,\mathit{comp}}}\end{array}$$ $$\begin{array}{l}r=g_x\cdot m_d\cdot G_r\left[f\left(\gamma \right)\right]\cdot e^{{\mathit{j\varphi }}_r\left[f\left(\gamma \right)\right]}+g_n\cdot H_1\left\{m\right\}\cdot G_{r,\mathit{comp}}\cdot e^{{\mathit{j\varphi }}_{r,\mathit{comp}}}+g_n\cdot H_2\left\{m\right\}\cdot G_{r,\mathit{comp}}\\ \cdot e^{{\mathit{j\varphi }}_{r,\mathit{comp}}}\end{array}$$ where e.g.: $$G_{l,\mathit{comp}}=g_{\mathit{norm}}\cdot \left|{\displaystyle {\sum }_{\beta \in {\mathrm{B}}_l}{G}_l\left[\beta \right]}\cdot e^{{\mathit{j\varphi }}_l\left[\beta \right]}\right|$$ $${\varphi }_{l,\mathit{comp}}=\angle \left\{{\displaystyle {\sum }_{\beta \in {\mathrm{B}}_l}{G}_l\left[\beta \right]}\cdot e^{{\mathit{j\varphi }}_l\left[\beta \right]}\right\}$$ $$G_{r,\mathit{comp}}=g_{\mathit{norm}}\cdot \left|{\displaystyle {\sum }_{\beta \in {\mathrm{B}}_r}{G}_r\left[\beta \right]}\cdot e^{{\mathit{j\varphi }}_r\left[\beta \right]}\right|$$ $${\varphi }_{r,\mathit{comp}}=\angle \left\{{\displaystyle {\sum }_{\beta \in {\mathrm{B}}_r}{G}_r\left[\beta \right]}\cdot e^{{\mathit{j\varphi }}_r\left[\beta \right]}\right\}$$ with B_l being a set of angles at which the left diffuse signal is to be rendered, B_r a set of angles at which the right diffuse signal is to be rendered, and g_norm a normalisation factor.

In some embodiments, the diffuse signal component may be directly rendered onto left and right channels without any HRIR processing.

In some embodiments where only a single decorrelation path is used, the left and right diffuse signals may be approximated by an out-of-phase approximation, such as $$n_l^{\prime }=g_n\cdot H\left\{m\right\}$$ $$n_r^{\prime }=-g_n\cdot H\left\{m\right\}$$

As described, the operation may typically be performed in a frequency domain representation and most or all of the described processing may be performed on a subband basis. In particular, the spatial upmix parameters may be provided for frequency subbands and the first renderer 307 may be arranged to generate subband values for subbands of the first intermediate signals from subband values of the mono downmix audio signal based on spatial upmix parameters and directional transfer functions for the subbands. The processing may be performed in time intervals/segments with a time interval/segment of the output stereo signal being generated for the corresponding time interval/segment of the mono downmix audio signal.

Similarly, in many embodiments, the second renderer is arranged to generate subband values for subbands of the second intermediate signals from subband values of the decorrelated mono downmix audio signal, and specifically based on spatial upmix parameters and directional transfer functions for the subbands (if these are used).

The output signal may specifically be a binaural stereo audio signal and the audio render apparatus 103 may be arranged to perform a binaural rendering where an improved point source audio rendering can be achieved by generating a local directional signal from the received mono downmix audio signal and a received difference signal (X̂_res). Specifically, a received bit-stream/data signal may be de-multiplexed into a bit-stream for the mono downmix audio signal, the spatial parameters, and the difference signal. The mono downmix audio signal and the differential signal may be decoded to the time domain and transformed to the frequency domain. The differential signal may be added to the mono downmix audio signal to generate the directional signal X̂ = X̂' + X̂_res. The resulting signal may be rendered, and specifically may be rendered as an audio point source. This point source audio may further (optionally) be supplemented by rendering other audio that may specifically represent non-point source audio, such as background or ambient sounds. These signal components may be generated from decorrelated signals generated from the mono downmix audio signal.

The audio apparatus may accordingly be arranged to generate an output stereo signal, and often an output binaural stereo signal from the received mono downmix audio signal, spatial upmix parameters, and a difference signal. The audio apparatus specifically implements two different rendering paths with one being a directional (binaural) rendering of a directional (e.g. a dominant) signal component which is generated from the mono downmix audio signal and the difference signal. The other rendering path may employ a predetermined rendering/mapping of a decorrelated audio signal generated from the mono downmix audio signal. The rendering of the output stereo signal is typically not a conventional adaptive upmixing of the received and decorrelated mono signals, and is specifically not a conventional 2x2 matrix upmixing of the mono signal and a decorrelated signal, but rather is a direct generation of a stereo signal by parallel processing of respectively the mono downmix audio signal and one or more decorrelated versions of this, with the former rendering being directional dependent on the spatial upmix parameters and the latter rendering being a predetermined rendering.

The processing seeks to render direct/dominant/directional point source components using a direct rendering with a direction that is given by the spatial parameters. The rendering employs a directionally dependent transfer function to the left and right stereo output signal for that purpose. The approach further seeks to render a residual/remaining signal component as a more diffuse signal, and specifically it uses a predetermined rendering where a decorrelated signal is mapped directly to the channels of the output binaural signal using a transfer function. The mapping is predetermined and may specifically be such that it allows a more diffuse and non-directional perception of this signal component. The rendering process thus uses fundamentally different approaches to provide different signal components in the output binaural signal, but does so without specifically decomposing the mono downmix audio signal into a dominant and diffuse/residual signal component with these subsequently being individually rendered. Rather a direct rendering of respectively the mono downmix audio signal and a decorrelated version thereof is performed to generate the binaural output signal.

The approach allows low complexity and computationally efficient rendering of a stereo signal encoded as a downmix and spatial upmix parameters, such as a PS encoded signal. It may further allow high performance rendering with a perceived improved audio quality. In many cases, a substantially improved spatial perception and user experience may be achieved, and indeed can be provided using headphones. The approach may in many cases provide a user perception of an audio scene where individual/dominant audio sources are well defined at specific directions/positions in a stereo image whereas other sources (e.g. ambient or background audio sources) are perceived more diffuse.

The approach may typically allow a very efficient operation and rendering with reduced complexity. A particular advantage of the approach is that it does not require a decomposition of the received mono downmix audio signal into different components with different specific properties.

In many embodiments, the difference signal is generated to directly describe or define the difference between the directional signal and the mono downmix audio signal, such as by directly corresponding to the difference between these.

However, in some embodiments, the difference signal may be generated to be indicative of a difference between the directional signal and a prediction/estimate of the directional signal where the prediction/estimate of the directional signal is determined as a function of (at least) the mono downmix audio signal and the spatial parameters. The function may typically be a predetermined function that when applied to the mono downmix audio signal and the spatial parameters will provide an output of an estimated/predicted directional signal. The predetermined function may in many embodiments be applied in the frequency domain. The estimated directional signal may then be compared to the generated directional signal to generate the difference signal. Specifically, the difference signal may be generated as the difference between the estimated/predicted directional signal and the directional signal.

At the audio render apparatus 103, the compensator 305 may be arranged to perform the corresponding operation to generate a local estimate of the directional signal. It can apply the, typically predetermined, function to the received mono downmix audio signal and spatial parameters to generate the local replica of the difference signal. The audio source device 101 and the audio render apparatus 103 may be arranged to apply identical functions resulting in (at least substantially) the same estimated directional signal being generated at the audio render apparatus 103 as is generated by the audio source device 101. The function may be predetermined and may for example be defined/specified by a suitable audio encoding standard, or it may e.g. otherwise be known/shared by the audio source device 101 and the audio render apparatus 103. In some embodiments, a fixed predetermined function may be specified by the audio source device 101 and included in the data signal such that the audio source device 101 can reconstruct the function and apply it to the received mono downmix audio signal as part of the rendering process.

The compensator 305 may proceed to generate the modified mono downmix audio signal by compensating the predicted signal as a function of the difference signal. Specifically, in many embodiments, the compensator 305 may generate the modified mono downmix audio signal by adding the difference signal to the generated local estimate of the directional signal.

In many embodiments, instead of directly calculating the difference between the mono signal and the directional signal, the audio source device 101 may generate an estimate of the directional signal which may be used to reduce the differences that are to be represented by the difference signal thereby allowing a lower data rate for a given audio quality. In such cases, the difference between the directional signal x and the approximation/estimate x' may specifically be encoded by the difference signal: $${\widehat{X}}_{\mathit{res}}=X-X\prime$$

The complementary operation may be performed by the audio render apparatus 103 using the modified differential signal (X̂_res). Using the spatial parameters, such as specifically PS parameters, an estimate of the directional signal X̂' is made and the differential signal X̂_res is added, resulting in the modified mono downmix audio signal corresponding to the directional signal estimate X̂ = X̂' + X̂_res. The modified mono downmix audio signal corresponding to the resulting directional signal estimate X̂ may be rendered by a directional rendering. Further, decorrelated signals may be generated from the modified mono downmix audio signal/directional signal estimate X̂ (or possibly directly from the received mono downmix audio signal) and rendered as more diffuse and spatially non-specific audio.

A specific example of a block diagram of an implementation of the audio renderer device 103 which follows such an approach is illustrated in FIG.4. In the example, the received bit-stream/data signal is de-multiplexed into a bit-stream for the mono downmix audio signal (the lower path), the stereo parameters (the middle path) and the difference signal (the upper path). The mono downmix audio signal and the difference signal are decoded to the time domain and transformed to the frequency domain by filter banks FB. Using the spatial (e.g. PS) parameters, an estimate of the directional signal X̂' is made. The difference signal is added to the directional signal X̂', resulting in the directional signal X̂ = X̂' + X̂_res. From the directional signal, (or e.g. directly from the mono downmix audio signal) one or more residual components may be estimated. Finally, rendering of the components is performed as previously described after which left and right binaural signals are synthesized by a conversion to the time domain using complementary filter banks FB^-1.

It will be appreciated that different approaches, algorithms, and functions may be used to estimate the directional signal from the mono downmix audio signal using the spatial parameters.

For example, a rudimentary estimate of the directional signal from the mono signal may follow from: x' = ICC · m. This example may follow the consideration that for fully decorrelated signals all of the signal power of the original stereo signal is captured by the directional signal, whereas for fully decorrelated signals none of the signal power of the original stereo signal is captured by the directional signal.

In a particular approach, the directional signal is estimated as a spectro-temporally shaped version of the received mono downmix audio signal, where the shaping is done in such a way that it approximates the signal power of the directional signal of the signal model as previously described, i.e. where the original stereo signal is considered to represent a main component x and residual/more diffuse component n (which typically has a component for each channel, i.e. n can be considered a stereo component): $$l=\cos \left(\gamma \right)e^{{\mathit{j\varphi }}_l}x+n_l$$ $$r=\sin \left(\gamma \right)e^{{\mathit{j\varphi }}_r}x+n_r$$

The audio render apparatus 103 may seek to generate an estimate of the directional signal component x by scaling of the mono downmix audio signal (with the scaling typically being separate for different frequency band). The estimated direct signal component may be represented by: $$x\prime =g_x\cdot m=\sqrt{\frac{{‖x‖}^2}{{‖m‖}^2}}m$$ where the resulting approximation x' is the estimate of the directional signal in a given subband and g_x is determined from the spatial parameters, and specifically with $$g_x=f\left(\mathit{IID},\mathit{ICC},\mathit{IPD}\right)$$

In this case, the directional signal may (in the frequency/subband domain) be determined as: $$\widehat{X}=\widehat{X}\prime +{\widehat{X}}_{\mathit{res}}=g_x\cdot \widehat{M}+{\widehat{X}}_{\mathit{res}}.$$

The spatial parameters are indicative of the relative properties of the channel signals for the stereo signal and, as has been realized by the inventors, this may also provide information on the relative levels for respectively a directional component (and for a non-directional component). The compensator 305 may specifically determine a scaling factor that scales the effective gain of subbands of the mono downmix audio signal to generate the directional signal estimate.

In many embodiments, the compensator 305 is accordingly arranged to generate the gains dependent on the spatial parameters. In particular, the gain for the directional rendering may be set in dependence on a relative power level of a directional component of the stereo signal relative to a power level of the mono downmix audio signal. The spatial upmix parameters provide information on the relative properties of the channel signals of the stereo signal, and specifically may provide information on both the interchannel levels/intensity differences as well as on the interchannel correlation. Accordingly, the spatial upmix parameters can be considered to provide information on the directional signal component x and the diffuse/residual component n. Accordingly, the gains g_x may be estimated/calculated from the received spatial parameters.

The gains may in many embodiments be determined to ensure power preservation. For the indicated signal model of the stereo signal, the gains for an actual point source with no other audio being present may be 1 for the directional rendering path and 0 for the predetermined/diffuse rendering path. For a completely diffuse signal, the gains would be the opposite, i.e. 1 for the predetermined rendering path and 0 for the directional rendering path. In practice, when two diffuse signals are considered, one for the left channel and one for the right channel, the gains would typically be $1/\sqrt{2}$.

The gains for estimating the directional signal may in particular in many embodiments advantageously be determined in line with one or more of the following: $$g_x=\sqrt{\frac{\sqrt{{\left(\mathrm{IID}-1\right)}^2+4{\mathrm{ICC}}^2\mathrm{IID}}}{\mathrm{IID}+1}}$$ $$g_x=\sqrt{\frac{{\left(\mathrm{IID}-1\right)}^2+4\cdot {\mathrm{ICC}}^2\cdot \mathrm{IID}+\left(1-\mathrm{IID}\right)\cdot \sqrt{4\cdot {\mathrm{ICC}}^2\cdot \mathrm{IID}+{\left(\mathrm{IID}-1\right)}^2}}{1-{\mathrm{IID}}^2+\left(\mathrm{IID}+1\right)\cdot \sqrt{4\cdot {\mathrm{ICC}}^2\cdot \mathrm{IID}+{\left(\mathrm{IID}-1\right)}^2}}}$$ where IID is an interchannel intensity difference and ICC is an inter-channel cross-correlation, and specifically $$\mathrm{IID}=\frac{{‖l‖}^2}{{‖r‖}^2}$$ $$\mathrm{ICC}=\frac{\left|〈l,r〉\right|}{\sqrt{{‖l‖}^2{‖r‖}^2}}$$ and $$〈\mathrm{x},\mathrm{y}〉={\displaystyle \sum _{\forall i}{x}_i\cdot y_i^{\ast }}$$

Thus, in many embodiments, one or more of the gains may be determined based on received interchannel intensity differences, and interchannel correlations (being part of the spatial upmix parameters).

Different approaches for determining the rendering direction from the spatial upmix parameters may be used in different embodiments. In particular, the signal model as indicated above is based on directional component x being at a direction γ in the stereo image of the stereo signal, henceforth also referred to as the orientation direction. In many embodiments, the direction determining circuit 309 may determine the orientation direction y and then determine the (desired) rendering direction γ' from the orientation direction y. Indeed, in some embodiments or scenarios, the rendering direction γ' may simply be set equal to the orientation direction y.

The determination of the orientation direction y may be based on the signal model indicated above. The spatial upmix parameters provide information on the relative properties of the channel signals of the stereo signal and specifically they may provide information on both the interchannel levels/intensity differences as well as on the interchannel correlation. Accordingly, the spatial upmix parameters can be considered to provide information on the directional signal component x and on the position of this in the stereo image of the stereo signal, i.e. the spatial upmix parameters provide information on the orientation direction y allowing this to be determined from the provided parameter values.

The direction determining circuit 309 may determine the orientation direction y as a direction to a directional signal component in a stereo image of the stereo signal from the spatial upmix parameters, and to map this to a direction in a stereo image of the output stereo signal. The directional signal component may be a dominant signal component. The direction determining circuit 309 may be arranged to determine the orientation direction y as a direction of a dominant sound source in the stereo signal where the direction of the dominant sound source is represented by the spatial upmix parameters.

The directional signal component may specifically be a signal component (estimated/determined) to originate from a point source. Specifically, the direction determining circuit 309 may be arranged to determine the orientation direction y as a direction for which a single point source audio source will result in spatial upmix parameter values matching the spatial upmix parameters of the data signal.

In some embodiments, the direction determining circuit may be arranged to determine the first direction in line with: $$\gamma =\arctan \left(\frac{1-\mathrm{IID}+\sqrt{{\left(\mathrm{IID}-1\right)}^2+4\cdot {\mathrm{ICC}}^2\cdot \mathrm{IID}}}{2\cdot \mathrm{ICC}\cdot \sqrt{\mathrm{IID}}}\right)$$ where IID is an interchannel intensity difference and ICC is an inter-channel cross-correlation, and specifically with these given by the equations provided above in connection with the equations for determining gains. The determination of the orientation direction y above results from an assumption that the signals x, n_g and n_r of the signal model are mutually decorrelated, and that the power of the left and right diffuse signals n_l and n_r are equal.

The direction determining circuit 309 may, as previously mentioned, in some embodiments be used directly as the rendering direction γ', i.e. γ = γ'. However, in many embodiments, a mapping may be included which for at least some values of the orientation direction y may result in a dfferent rendering direction γ'.

Thus, in many embodiments, the direction determining circuit 309 may be arranged to apply a mapping function to the orientation direction y to determine the rendering direction γ'.

For example, the mapping may map the position in the stereo image of the original stereo signal as represented by the orientation direction y to a desired position in the stereo image of the output stereo signal as represented by the rendering direction γ'. In many cases, where the output stereo signal is a binaural signal, the mapping may include a consideration/ determination of a distance to the audio sources. For example, a range of the orientation direction y in the interval of [0,180°] may be mapped to a location between two virtual stereo speakers in the audio scene created by the binaural rendering. Such speakers may for example be positioned at angles of -30° and +30° relative to a center direction for the binaural signal. Thus, in such situations, the direction determining circuit 309 may include a mapping between an orientation direction y in the range of [0,180°] to a rendering direction γ' in the range of [-30°,+30°].

Thus, in some embodiments, the directional component (the mono downmix audio signal) may be rendered to a virtual angle in the range of a virtual loudspeaker angle range generated by a binaural rendering. The rendered directional component may be combined with a diffuse rendering of the residual signal(s).

In many embodiments, the direction determining circuit 309 may be arranged to map an orientation direction γ representing an angle in one interval/range to a rendering direction γ' representing an angle in a different interval/range.

In the previous examples, the rendering has been based on one intermediate stereo signal representing the diffuse signal component. However, in many embodiments, there may be two (or possibly more) parallel paths for the rendering of the residual/non-directional signal components.

The audio source device 101 is arranged to include data representing the difference signal in the data signal. Typically, the difference signal will be suitable encoded, e.g. using an audio encoding algorithm, or in many embodiments using a non-lossy encoding.

The data signal may be generated to include the different types of data in a suitable structure and with suitable data overhead (such as headers etc).

In some embodiments, the data signal circuit 207 is arranged to include the encoded data for the difference signal in suitable data fields of the generated bitstream, and in particular it may include the encoded data for the difference signal in one or more optional data fields of the audio data signal. An optional data field may be a field that can include data which is optional for the data signal. For example, some traditional data structures for PS data signals include data fields in which proprietary or non-PS data can be included and in some embodiments the difference signal data may be included in such data fields. In such cases, a conventional PS decoder/renderer may simply ignore the difference data and proceed to perform a conventional PS decoding and rendering. However, an audio render apparatus as previously described may instead extract the difference data and proceed to perform a rendering based on a generated directional signal as previously described. The approach may accordingly provide improved rendering for suitably capable renderers while providing backwards compatibility to support legacy decoders/renderers.

The processing may be performed in subbands and may be performed in time segments. The processing in each subband may for some (any) or all steps be performed separately/independently in each subband (with respect to the processing in other subbands). The processing in each time segment may for some (any) or all steps be performed separately/independently in each time segment (with respect to the processing in other time segments).

The processing may be time interval/segment based with all processing being performed for each time segment. Equivalently, the signal(s) for each segment may be considered a signal (and in particular signals of different time segments, may be considered different signals).

The audio apparatus(s) may specifically be implemented in one or more suitably programmed processors. An example of a suitable processor is provided in the following.

FIG. 5 is a block diagram illustrating an example processor 500 according to embodiments of the disclosure. Processor 500 may be used to implement one or more processors implementing an apparatus as previously described or elements thereof (including in particular one more artificial neural network). Processor 500 may be any suitable processor type including, but not limited to, a microprocessor, a microcontroller, a Digital Signal Processor (DSP), a Field ProGrammable Array (FPGA) where the FPGA has been programmed to form a processor, a Graphical Processing Unit (GPU), an Application Specific Integrated Circuit (ASIC) where the ASIC has been designed to form a processor, or a combination thereof.

The processor 500 may include one or more cores 502. The core 502 may include one or more Arithmetic Logic Units (ALU) 504. In some embodiments, the core 502 may include a Floating Point Logic Unit (FPLU) 506 and/or a Digital Signal Processing Unit (DSPU) 508 in addition to or instead of the ALU 504.

The processor 500 may include one or more registers 512 communicatively coupled to the core 502. The registers 512 may be implemented using dedicated logic gate circuits (e.g., flip-flops) and/or any memory technology. In some embodiments the registers 512 may be implemented using static memory. The register may provide data, instructions and addresses to the core 502.

In some embodiments, processor 500 may include one or more levels of cache memory 510 communicatively coupled to the core 502. The cache memory 510 may provide computer-readable instructions to the core 502 for execution. The cache memory 510 may provide data for processing by the core 502. In some embodiments, the computer-readable instructions may have been provided to the cache memory 510 by a local memory, for example, local memory attached to the external bus 516. The cache memory 510 may be implemented with any suitable cache memory type, for example, Metal-Oxide Semiconductor (MOS) memory such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), and/or any other suitable memory technology.

The processor 500 may include a controller 514, which may control input to the processor 500 from other processors and/or components included in a system and/or outputs from the processor 500 to other processors and/or components included in the system. Controller 514 may control the data paths in the ALU 504, FPLU 506 and/or DSPU 508. Controller 514 may be implemented as one or more state machines, data paths and/or dedicated control logic. The gates of controller 514 may be implemented as standalone gates, FPGA, ASIC or any other suitable technology.

The registers 512 and the cache 510 may communicate with controller 514 and core 502 via internal connections 520A, 520B, 520C and 520D. Internal connections may be implemented as a bus, multiplexer, crossbar switch, and/or any other suitable connection technology.

Inputs and outputs for the processor 500 may be provided via a bus 516, which may include one or more conductive lines. The bus 516 may be communicatively coupled to one or more components of processor 500, for example the controller 514, cache 510, and/or register 512. The bus 516 may be coupled to one or more components of the system.

The bus 516 may be coupled to one or more external memories. The external memories may include Read Only Memory (ROM) 532. ROM 532 may be a masked ROM, Electronically Programmable Read Only Memory (EPROM) or any other suitable technology. The external memory may include Random Access Memory (RAM) 533. RAM 533 may be a static RAM, battery backed up static RAM, Dynamic RAM (DRAM) or any other suitable technology. The external memory may include Electrically Erasable Programmable Read Only Memory (EEPROM) 535. The external memory may include Flash memory 534. The External memory may include a magnetic storage device such as disc 536. In some embodiments, the external memories may be included in a system.

The invention can be implemented in any suitable form including hardware, software, firmware, or any combination of these. The invention may optionally be implemented at least partly as computer software running on one or more data processors and/or digital signal processors. The elements and components of an embodiment of the invention may be physically, functionally and logically implemented in any suitable way. Indeed, the functionality may be implemented in a single unit, in a plurality of units or as part of other functional units. As such, the invention may be implemented in a single unit or may be physically and functionally distributed between different units, circuits and processors.

Although the present invention has been described in connection with some embodiments, it is not intended to be limited to the specific form set forth herein. Rather, the scope of the present invention is limited only by the accompanying claims. Additionally, although a feature may appear to be described in connection with particular embodiments, one skilled in the art would recognize that various features of the described embodiments may be combined in accordance with the invention. In the claims, the term comprising does not exclude the presence of other elements or steps.

Furthermore, although individually listed, a plurality of means, elements, circuits or method steps may be implemented by e.g. a single circuit, unit or processor. Additionally, although individual features may be included in different claims, these may possibly be advantageously combined, and the inclusion in different claims does not imply that a combination of features is not feasible and/or advantageous. Also, the inclusion of a feature in one category of claims does not imply a limitation to this category but rather indicates that the feature is equally applicable to other claim categories as appropriate. Furthermore, the order of features in the claims do not imply any specific order in which the features must be worked and in particular the order of individual steps in a method claim does not imply that the steps must be performed in this order. Rather, the steps may be performed in any suitable order. In addition, singular references do not exclude a plurality. Thus references to "a", "an", "first", "second" etc. do not preclude a plurality. Reference signs in the claims are provided merely as a clarifying example shall not be construed as limiting the scope of the claims in any way.