Technical field
[0001] The invention relates generally to hearing aids, and particularly to systems and methods
for improving directional hearing.
Background information and prior art
[0002] Speech understanding in noisy environments is a significant problem for the hearing-impaired
person. Hearing impairment is usually accompanied by a reduced time resolution of
the sensorial system in addition to a gain loss. These characteristics further reduce
the ability of the hearing-impaired to filter the target sound source from the background
noise and particularly to understand speech in noisy environments. It can also be
challenging for non-hearing-impaired person.
[0003] Some newer hearing aids offer a directional hearing mode to improve speech intelligibility
in noisy environments. This mode makes use of an array of microphones and applies
beamforming technology to combine multiple microphone audio inputs into a single,
directional audio output channel. Beamforming is a signal processing technique in
which multiple channels of audio can be processed to generate a beamformed audio signal
in which audio from different directions may be amplified, attenuated or cleaned.
Therefore, the audio output channel has spatial characteristics that increase the
contribution of audio waves arriving from the target sound source direction which
has the user's attention relative to those of the audio waves from other directions.
[0004] To determine the user's attention, some existing solutions rely on an eye-tracking system
that determines the gaze direction of a user. For example,
US2016/0080874A1 discloses a hearing assistance system incorporating an eye tracker and microphones
which audio outputs are conditioned to obtain an enhanced audio. In crowded environments
with multiple stimuli, maintaining control over user attention through gaze alone
can be challenging, potentially limiting its effectiveness in improving audio localization.
[0005] Other existing solutions rely on audio systems using head tracking to guide audio
beamforming toward a specific target sound source.
[0006] No evidence-based support was provided to suggest that any of the identified alternatives
would be more successful in determining the user's attention.
[0007] Moreover, these systems often face challenges in accurately identifying a user's
point of attention, particularly in dynamic or intricate visual environments.
[0008] One object of the present invention is to enhance the accuracy of the capture of
attention of the user, especially in dynamic of intricate visual environments.
[0009] Another object of the present invention is to determine the best combination of sensors
to use in different contexts of beamforming.
[0010] Another object of the invention is to improve the accuracy of the selection of the
target sound source the user would like to listen to and tailor the beamforming audio
experience accordingly, especially in dynamic of intricate visual environments, the
user being hearing impaired or not.
Summary
[0011] These goals are obtained according to the present disclosure by proposing a system
for hearing assistance as set forth in claim 1, a head mountable device as set forth
in claim 10, an eyewear as set forth in clam 11 and a method as set forth in claim
12.
[0012] Further characteristics of the systems, devices and method are the objects of the
dependent claims.
Description of the drawings
[0013] For a more complete understanding of the description provided herein and the advantages
thereof, reference is now made to the brief descriptions below, taken in connection
with the accompanying drawings and detailed description, wherein like reference numerals
represent like parts.
FIG. 1 represents is a schematic illustration showing a system for hearing assistance
to a user shown as an eyewear;
FIG. 2 is a schematic illustration of a processing unit implemented within the system
of FIG. 1 with its inputs and outputs;
FIG. 3 is a schematic illustration of another part of the processing unit implemented
within the system of FIG.1 ;
FIG. 4 is a scheme illustrating the main steps of a computer-implemented method for
hearing assistance to a user, according to the disclosure.
Detailed description of embodiments
[0014] In the description which follows the drawing figures are not necessarily to scale
and certain features may be shown in generalized or schematic form in the interest
of clarity and conciseness or for informational purposes. In addition, although making
and using various embodiments are discussed in detail below, it should be appreciated
that as described herein are provided many inventive concepts that may be embodied
in a wide variety of contexts. Embodiments discussed herein are merely representative
and do not limit the scope of the disclosure. It will also be obvious to one skilled
in the art that all the technical features that are defined relative to a method can
be transposed, individually or in combination, to a device and conversely, all the
technical features relative to a device can be transposed, individually or in combination,
to a method.
Hearing assistance system
[0015] FIG.1 is a schematic illustration of a hearing assistance system 20 that is at least
in part integrated into an eyewear. The eyewear may include at least a frame 22 with
a front piece 30 and temples 32 connected to respective edges of the front piece 30.
[0016] In other embodiments, the hearing assistance system 20 can be at least in part integrated
in other head mountable devices. Head-mountable devices encompass any type of head
mountable device designed to be worn in front of/in at least one eye of a user and/or
head mountable hearing device, in any suitable form factor. It includes head-mounted
display device (e.g. near eye display device, augmented reality devices, virtual reality
devices and/or mixed reality devices), helmet, headset, eyewear smart or not (e.g.
eyeglasses, googles, masks, visors, clip-on, contact lens, intraocular implant). It
includes also earwear (e.g. headphones, earbuds), hearing aid, speakers, bone conduction
transducers.
[0017] The hearing assistance system 20 comprises an array of microphones 23, 24 mounted
at predetermined distance of each other on the frame 22.
[0018] The microphones 23,24 are configured to detect and capture at least one audio signal
from sound sources in the environment of the user. Each microphone 23, 24 in the array
of microphones can be of the same type or different types. In particular, all microphones23,
24 in the microphone array may be omnidirectional or directional. A microphone array
having one dimension (for example, either the array of microphones 23 or the array
of microphones 24) is capable of beamforming in one dimension, while a microphone
array in two-dimensions (namely where for example there is both the array of microphones
23 and the array of microphones 24) may be capable of beamforming in one or two dimensions.
[0019] In the illustrated example, microphones 23 are mounted on the front piece 30 of the
frame 22, while microphones 24 are mounted on temples 32. Although the extensive array
of microphones 24 that is shown in FIG. 1 is useful in some applications of the present
disclosure, the principles of hearing assistance that are described herein may alternatively
be applied, mutatis mutandis, using smaller of higher numbers of microphones. For
example, these principles may be applied using an array of microphones 23 on the front
piece 30, alternatively or additionally to the array of microphones 24 that are on
the temples. In some embodiments, preferably, a single microphone 23 on the front
piece 30 and a single microphone 24 on each temple 32 may be used to triangulate in
a simple manner the audio signal sources.
[0020] The hearing assistance system 20 comprises one or more speakers 28, which are in
the illustrated embodiment mounted on the frame 20 and more precisely on the end of
each temple 32, typically in proximity to the user's ears. Although only a single
speaker 28 is shown on each temple 32 in FIG. 1, the system 20 may alternatively comprise
two or more speakers that can be grouped in arrays mounted on or attached to one or
both temples 32. In other embodiments, only one speaker 28 is mounted on or attached
to the frame 22.
[0021] Preferably, the speakers 28 are monodirectional or omnidirectional.
[0022] In other embodiments, the system 20 comprises alternative hearing devices 28. Such
hearing devices can be headphones, earbuds, headsets or any audio transducers like
bone conduction transducers.
[0023] The hearing assistance system 20 also comprises one or more physiological sensor
devices.
[0024] The system comprises at least an eye-tracking device including at least an eye tracking
sensor 21,21' and an eye tracking module operatively coupled to the sensor to provide
gaze direction data and eye movements data of the user's eyes.
[0025] The eye tracking module comprises at least a processor and a memory.
[0026] The eye tracking sensor can track the eye using any eye tracking technique. In particular,
it may comprise imaging systems like cameras including depth camera, LIDAR, time-of-flight
sensor, infrared sensor, wavefront sensor and/or a combination thereof.
[0027] In one example, as shown in FIG. 1, the eye-tracking sensors 21,21' are cameras.
The cameras 21,21' may be embedded or attached to the frame 22 especially in the front
piece 30. One single camera 21,21' may be sufficient.
[0028] The eye tracking module is configured to determine gaze direction data and eye movements
of the user's eyes based on the eye images (including pupil, corneal and/or light
reflection images) and/or videos captured.
[0029] Preferably, the eye tracking module further processes the eye movements and in particular
the saccadic movements of the user's eyes and determine the fixations of the eyes.
The module is configured to generate features relative to the stability of the gaze
of the user's eyes. Preferably, it generates features like focus intensity and duration
of the gaze on target sound source(s) in the environment the user is looking at.
[0030] In embodiments, the eye tracking module further process the eye movements and in
particular the pupil movements i.e. dilation/constriction variation of the eyes to
generate features relative to a level of cognitive and emotional engagement of the
user on target sound source(s) the user is interested in.
[0031] The eye tracking module uses any algorithm and in particular machine learning based
algorithm to process data output from the eye tracking sensors and determine gaze
direction vectors, gaze stability vectors as well as user cognitive engagement vectors
associated with target sound source(s) the user is interested in.
[0032] In addition, the system comprises at least one vergence sensor and a vergence module
operatively coupled to the sensor to provide vergence data of the user' eyes, including
vergence movements.
[0033] Vergence movements align the fovea of each eye of the user with attentional target
sound sources located at different distances from the user. Unlike other types of
eye movements in which the two eyes move in the same direction, vergence movements
are disconjugate i.e. they involve either a convergence or divergence of the lines
of sight of each eye to see an object that is nearer or farther away from the user.
[0034] The vergence sensor may be any eyetracking systems already described above. It may
be the same eyetracking sensor or an additional one.
[0035] The vergence module comprises at least a processor and a memory.
[0036] The vergence module is configured to determine vergence movements, in particular
the direction of each eye as well as the vergence angle of the user's eyes i.e. the
angle of convergence of the lines of sight of the eyes as they focus at variable distances
from the user. The vergence module is configured to estimate the 3D distance or depth
of attentional target sound source(s) the user is interested in based on these vergence
angle data.
[0037] The vergence module uses any algorithm and in particular machine learning based algorithm
to process vergence data to output depth of focus of user's eyes vectors.
[0038] The set of physiological sensor devices may also include at least one additional
device configured to provide cortical activity data of the user. This device comprises
at least a cortical activity sensor and cortical activity module operatively coupled
to the sensor.
[0039] The cortical activity module comprises at least one processor and a memory.
[0040] Cortical activity signals captured by the sensor can be characterized by their frequency,
amplitude and phase and can give rise electrical impulses indicative of brain activity
pattern. The signal properties can be analyzed using time frequency analysis.
[0041] The cortical activity sensor(s) may comprise electroencephalography (EEG) sensor
and/or magnetoencephalography MEG sensor to measure cortical activity signals.
[0042] In one example shown in FIG.1 the cortical activity sensor comprises at least a pair
of electrodes 29,29' configured to detect at least one EEG response signal to a scene/target
sound source(s) viewed and listened by the user.
[0043] The electrodes 29,29' are arranged at a predetermined distance to each other and
at least partially in contact with the user's head when the user wears the eyewear.
[0044] Preferably, the electrodes 29,29' are temporal electrodes placed in temporal regions
of the user's head to detect in a simple and accurate way the EEG response signals.
[0045] The electrodes 29,29' can be embedded in the frame 20 or attached to said frame 20.
[0046] The cortical activity module processes the cortical activity signals to extract features
in different frequency bands (e.g. alpha, beta and gamma oscillations) including features
related to amplitude variations, spectral power densities, power ratio, and event-related
potentials, in particular Visual Evoked Potential (VEP) and Auditory Evoked Potential
(AEP).
[0047] In an embodiment, the cortical activity signals are filtered to remove artifacts
and generate useful signals to be processed to extract features.
[0048] Based on the features extracted, the cortical activity module is configured to infer
cortical activity patterns associated to several states representative of user's level
of engagement with attentional target sound source(s) in his environment.
[0049] For example, the states may be whether the user is focusing intently on a single
target sound source (concentration, sustained attention) or switching focus between
various sound sources (selective attention or distraction). It may also encompass
any emotional state like whether the user is relaxed, stressed, excited, sad or not.
[0050] The cortical activity module uses any algorithm and in particular machine learning
based algorithm to process cortical activity signal data to output user's level of
engagement with target sound source(s) in his environment.
[0051] The set of physiological sensor devices may also include an additional head device
configured to provide head data of the user. The device comprises a head tracking
sensor and a head tracking module operatively coupled to the sensor.
[0052] The head data can be head movement (e.g. head fixed, tilted, up/down/tilt, left movement,
right movement) or head orientation (e.g. pitch, yaw, and roll) and/or position.
[0053] The head tracking module comprises a processor and a memory.
[0054] The head tracking sensor may comprise an inertial measurement sensor, an acceleration
sensor, a gyroscope, a depth camera or a combination thereof.
[0055] In another embodiment, the head data and in particular head orientation data may
be inferred based on the geometry of the microphone array to infer the head orientation
data.
[0056] The head tracking module process outputs signals from the head tracking sensor to
extract features related to the user's physical orientation and movement in the environment,
providing context to the direction and range of potential attentional targets sound
sources the user is looking at.
[0057] The module uses any algorithm and in particular machine learning based algorithm
to process head data to output user's physical orientation and/or movement vectors
representative of the attentional targets sound sources the user is looking at.
[0058] In one embodiment, head orientation data may also be used to optimize depth of focus
estimation.
[0059] In embodiments, the set of physiological sensor devices may also include a device
to provide context information relative to the environment of the user. For example,
the context information may include the objects and the person's location in the environment,
the type of sound source the user is interested in (e.g. person speaking, conversation,
device, etc..). Preferably, the context information device may include at least a
camera.
[0060] A context information module of the device uses any algorithm and in particular machine
learning based algorithm to process context information detected and provide context
information vector.
[0061] In embodiments, the set of physiological sensor devices may also include heart rate
sensor, pulse rate sensor, temperature sensor, blood pressure sensor, ambient light
sensor to collect additional vectors representative of the user' level of engagement
on attentional targets sound sources in his environment.
[0062] The sensors can be embedded or attached in the frame or be part of a wearable device
on any device in the user's environment.
[0063] In embodiments, all the sensors described can provide signals in real time. In other
embodiments, the sensors provide signals based on an event triggered by the user or
triggered by a predetermined threshold value.
[0064] In addition, the system 20 comprises a processing unit 26.
[0065] The processing unit 26 comprises at least one memory to store data and computer executable
instructions and at least one processor.
[0066] As illustrated in the embodiment of FIG.1 the processing unit is embedded in or attached
to at least one of the temples of the frame 22. In other embodiments, the processing
unit is embedded in the front piece 30 of the frame 22.
[0067] The processing unit 26 is configured to identify the user's attention state based
on data gathered from at least the eye tracking device and the vergence device.
[0068] In addition, the processing unit 26 is configured to enhance the audio signals from
the array of microphones by applying a beamforming function based on the user's attention
state estimated. For example, the processing unit emphasizes the sounds that originate
from the attentional target sound source the user is looking at. while suppressing
background noise originating outside this target sound source.
[0069] The processing unit 26 is described in greater detail hereinbelow, with the aid of
FIG.2.
[0070] As shown in FIG.2, the processing unit 26 comprises an attention estimation module
260.
[0071] The attention estimation module 260 is configured to generate attention data of the
user representative of a point of attention of the user. Attention data are at least
the 3D localization of the target sound source(s) the user is interested in and the
associated level of attentional engagement of the user.
[0072] In embodiments, the attention estimation module 260 integrates and processes at least
as inputs gaze direction and head data and depth of focus data determined from vergence
data. Based on these information, the attention estimation device 260 is configured
to apply a corresponding weight to the gaze direction data, head data and to the depth
of focus data, in order to generate the attention data of the user representative
of a point of attention of the user.
[0073] Preferably, the head data are head orientation data.
[0074] Moreover, the attention estimation module 260 is configured to process the vergence
data to decide which weight to apply respectively to the gaze direction vector and
to the head vector. The vergence data taken into account in the attention estimation
module helps to identify the best combination of sensors based on the region of interest
the user is looking at. For example, in specific situations, the head vector may have
a higher weight than the gaze vector or the contrary.
[0075] Preferably, context information data are used to smooth gaze direction and head vectors
to adjust these vectors to point at the closest object and/or person of interest within
the environment of the user.
[0076] In other embodiments, user level of engagement data defined from cortical activity
data are added to module 260 with a corresponding weight to generate the attention
data of the user representative of a point of attention of the user. Adding cortical
activity data allows to optimize the identification of the level of engagement of
the user on target sound source(s).
[0077] Preferably, gaze stability data are also considered to generate attention data.,
The module 260 is configured to receive said gaze stability data and to apply a corresponding
weight to the gaze stability data.
[0078] The module 260 may consider further physiological data or context information data
if needed, as a function of the type of sensors implemented.
[0079] The processing unit determines attention data of the user representative of a point
of attention of the user using any model-based algorithm, and in particular machine
learning-based algorithm.
[0080] In one embodiment, the attention estimation module 260 comprises a neural network.
Gaze direction vectors, user orientation and/or movement in space vector defined from
head data and the depth of focus vectors are joined as inputs in a neural network,
fused in the network whose output is the attention vector representative of the point
of attention of the user.
[0081] In another embodiment, an additional input corresponding to the user level of engagement
vector defined from cortical activity data is added to the set of input vectors in
the network.
[0082] All these data are combined to infer with accuracy the attention of the user in his
environment.
[0083] In other embodiments, an additional input corresponding to the gaze stability vector
is added to the set of input vectors in the network.
[0084] The module 260 may consider further data if needed, in particular context information
vector, pupil data, user's level of fatigue thanks to eyetracking monitoring.
[0085] As also shown in FIG. 2, the processing unit 26 comprises an audio beamforming module
262. The audio beamforming module 262 is configured to receive audio signals from
the microphone array 23, 24 as well as attention data generated by the attention estimation
module 260. Based on the attention data and the audio signals, the audio beamforming
module 262 generates at least beamformed audio signal intended to enhance sounds for
the user through said at least one speaker 28. For that, beamformed signal outputting
from the beamforming module 262 is directed towards a drive signal module 264, typically
comprising a numerical - analogic converter and filters, which drive signal (output)
is sent to said at least one speaker 28.
[0086] The audio beamforming module 262 is described in greater detail hereinbelow.
[0087] In one embodiment, the audio beamforming module 262 is configured to select a set
of beamforming processing settings to perform with respect to the audio signals captured
by the microphones. In particular, the audio beamforming module 262 comprises a beamforming
model that receives as inputs at least the attention vector and features associated
to the audio signals captured from the microphones. The model identifies as output
a selection of beamforming processing settings to apply to the signals to spatially
filter the audio signal inputs.
[0088] In several embodiments, the settings may include dynamic noise cancellation algorithms
to isolate attentional target sound sources and/or real time 3d spatial audio algorithms
to localize and enhance multiple target sound sources simultaneously.
[0089] For example, the settings may include parameters relative to beamforming directionality
and/or intensity to apply to the audio signals associated with the target sound source's),
to the amplification of sounds from the point of attention of the user and/or the
suppression of background noise and unrelated audio sources. In another example, it
may relate to a general enhancement of auditive acuity of people suffering hearing
loss and/or the decision to broaden the beamforming to help the user to listen to
the whole environment. All the settings enhance the listening experience in alignment
with the user's attention determined.
[0090] The audio beamforming module 262 uses any algorithm and in particular machine learning
based algorithm to process data to output the beamforming processing settings.
[0091] In one embodiment, it uses a deep neural network trained to map attention vectors
to optimal beamforming settings. For example, this network determine the directionality
and intensity of the beamforming focus. The audio signals captured by the microphones
are then processed using the beamforming processing settings identified.
[0092] Preferably, the audio signal processing is made in real time as the user's attention
shifts or auditory conditions of the environment change.
[0093] Within the beamforming module
262, any beamforming function known in art may be considered. For example, a time delay
algorithm may be used to combine the audio signals from the microphones
23, 24, with time shifts between the signals that are equal to the propagation times of the
acoustic waves between the microphone locations with respect to the desired beam direction
and to the location of the speaker
28. Alternatively, a Minimum Variance Distortionless Response (MVDR) beamforming algorithm
may be applied in order to achieve better spatial resolution. Other applicable beamforming
techniques are based on Linear Constraint Minimum Variance (LCMV) and General Sidelobe
Canceller (GSC) algorithms.
[0094] FIG.3 shows a part of the processing unit 26 comprising a beamforming activation
device 50.
[0095] The hearing assistance system 20 may comprise an ON/OFF activation device 50 for
the audio beamforming module 262. The user has therefore the choice to activate or
deactivate the beamforming processing of audio signals.
[0096] Alternatively, or additionally, the processing unit 26 may comprise an activation
device 51 for the audio beamforming module 262 configured to:
- receive at least gaze direction data of the user and/or
- receive at least vergence angle data of the user and/or
- receive at least the head orientation data
- activate the audio beamforming module when gaze direction data are above a first preset
threshold value THR1 and/or the vergence angle data are above a second preset threshold
value THR2, and/or head orientation data are above a fourth present threshold value
THR4.
[0097] Where cortical activity data are provided, the activation device 51 is also configured
to receive cortical activity data of the user and to activate the audio beamforming
module 262 where, additionally, said cortical activity data are above a third preset
threshold value THR3.
[0098] The system 20, the processing unit 26, the different devices and modules may use
diverse techniques and models to carry out the determination and estimation described
above. In particular, they can use any algorithm to determine the outputs based on
the inputs provided. The algorithms may include fuzzy logic, classification models,
clustering algorithms, data regressions algorithms, image segmentation algorithms,
Bayesian belief networks, data fusion engines machine learning algorithms like Support
Vector Machines SVMs ( linear, cubic, ..), neural network like Medium Neural Network
(MNN), wide Neural Network (WNN), recurrent neural networks (RNNs) like long short
term memory (LSTM), deep learning algorithms, convolutional neural network (CNN) and/or
Canonical Correlation Analysis (CCA) algorithm and so forth.
[0099] In alternative embodiments, all the modules described above may be implemented in
the processing unit 26.
[0100] In other embodiments, the processing unit 26 can be at least part of any external
computing device or can be a virtual machine located on a cloud or edge server. All
the devices or modules of the processing unit 26 described previously do not need
to be all implemented within the processing unit 26. Indeed, to optimize the memory
and power capabilities, at least some of the devices/modules may be implemented in
an external computing device communicating with the devices present in the frame through
a network.
[0101] Computing devices may include mobile devices (e.g., smartphones, tablet computers.)
and/or wearable devices (e.g., smartwatches), laptops, server computers and/or head-mounted
display devices.
[0102] The network can be a physical or wireless connection, such as Bluetooth
®, Wi-Fi
®, a GSM network or any global positioning system (GPS), private data network, near
field communications (NFC), LAN or WAN network.
[0103] Examples of processors include microprocessors, microcontrollers, graphics processing
units (GPUs), central processing units (CPUs), application processors, digital signal
processors (DSPs), reduced instruction set computing (RISC) processors, systems on
a chip (SoC), baseband processors, field programmable gate arrays (FPGAs), programmable
logic devices (PLDs), state machines, gated logic, discrete hardware circuits, and
other suitable hardware configured to perform the various functionality described
throughout this disclosure.
[0104] Memory refers to refers to any type or form of volatile or non-volatile storage device
or medium capable of storing data and/or computer-readable instructions. Memory may
include a random-access memory (RAM), a read-only memory (ROM), an electrically erasable
programmable ROM (EEPROM), optical disk storage, magnetic disk storage, other magnetic
storage devices, combinations of the aforementioned types of computer-readable media,
or any other medium that can be used to store computer executable code in the form
of instructions or data structures that can be accessed by the different modules and
processing units.
Computer implemented method for hearing assistance to a user
[0105] FIG.4. is a schematic diagram of the main steps of a computer implemented method
100 for hearing assistance to a user according to the disclosure.
[0106] In particular, the method 100 can be implemented by means of a computer program or
software loaded into the memory of the processing unit 26. Such a computer program
thus comprises instructions that induce the processing unit 26 to implement the method
100 when the processing unit runs the program.
[0107] The method 100 comprises the steps of:
- receiving 110 at least an audio signal captured with a microphone array,
- providing 120 gaze direction of the user,
- providing 130 vergence data of the user,
- providing head data of the user,
- determining 150 attention data of the user representative of a point of attention
of the user within his environment,
- generating 160 a beamformed audio signal from features associated with the audio signal
and from the attention data,
- broadcasting 170 the beamformed audio signal to the user through a hearing device.
[0108] Preferably, the head data are head orientation data.
[0109] In step 110, the processing unit 26 determines also features associated with the
audio signals. For example, the features can be the intensity, and the power of the
audio signals received from each microphone and/or the direction of arrival of the
audio signals.
[0110] In some embodiments, in step 150, the method determines the attention data by using
a model. More precisely, the method comprises steps of
- inputting at least a gaze direction vector, head vector and a depth of focus vector
in a model and
- outputting at least an attention vector representative of the attention of the user.
Preferably, the model is a convolutional neural network CNN.
Preferably, the head data are head orientation data.
[0111] In some embodiments, the computer implemented method may comprise before step 150:
- a step 140 where cortical activity data are provided,
- a step 141 where a user level of engagement data is determined from cortical activity
data.
[0112] In step 140, cortical activity signals are collected, and the cortical activity module
determines features associated with the cortical activity signals. For example, the
features can be the signal amplitude and power. A step of applying a frequency domain
filtering to the cortical activity signal may be implemented to select useful signals
about user's attention.
[0113] In these embodiments, in step 150, the method determines the attention data by using
a model. More precisely, the method comprises steps of:
- inputting at least a gaze direction vector, a head vector, a depth of focus vector
and a user level of engagement vector in a model and
- outputting at least an attention vector representative of the attention of the user.
Preferably, the model is a convolutional neural network CNN.
Preferably, the head vector is a head orientation vector.
[0114] In step 160, the method comprises the steps of selecting a set of beamforming processing
settings to perform with respect to the audio signals captured by the microphones
by using a model. More precisely, the method comprises steps of :
- Inputting at least the attention vector and features associated to the audio signals
captured from the microphones
- outputting a set of beamforming processing settings to apply to the signals to spatially
filter the audio signal inputs.
Preferably, the model is a deep learning network.
[0115] In other embodiments, the method comprises a step of activating the audio beamforming
experience when gaze direction data and/or vergence data and or cortical activity
data determined respectively in steps 120, 130 and 140 are above a respective first
preset threshold value THR1, a second preset threshold value THR2 and/or a preset
threshold value THR3.
[0116] In some embodiments, the method comprises a step of providing feedback about the
beamforming processing settings selected by determining a user's level of satisfaction
of the audio signals broadcasted.
[0117] In embodiments, the level of satisfaction may be computed from the cortical activity
data and/ or eye tracking data like pupil dilation data to define the level of satisfaction
of the user. Any method to determine the level of satisfaction can be used. In other
embodiments, a feedback answer from the user by voice command and/or user movement
is provided.
[0118] In embodiments, If the level of satisfaction is below a threshold THR5, the method
comprises a step to determine again the attention data in step 150 and/or to select
new beamforming processing settings in step 160.
[0119] Although representative methods, devices and modules have been described in detail
herein, those skilled in the art will recognize that various substitutions and modifications
may be made without departing from the scope of what is described and defined by the
appended claims.
1. A system (20) for hearing assistance to a user, said system comprising:
- a microphone array (23, 24) providing audio signals,
- at least one hearing device (28),
- an eye-tracking device (21) configured to provide user's gaze direction,
- a vergence device (21) configured to provide user's vergence data,
- a head tracking device configured to provide head data,
- a processing unit (26) comprising:
- an attention estimation module (260) configured to generate attention data of the
user representative of a point of attention of the user based on the user's gaze direction,
head data and vergence data,
- an audio beamforming module (262) configured to generate a beamformed audio signal
intended to enhance sounds for the user through said at least one hearing device (28)
based on the audio signals and the attention data of the user representative of a
point of attention of the user.
2. System according to claim 1 further comprising a cortical activity device (29,29')
configured to provide cortical activity data of the user, the attention estimation
module (260) being configured to generate attention data based on the cortical activity
data provided.
3. System according to claim 2, wherein the cortical activity device (29,29') is configured
to determine user level of engagement data within his environment.
4. System according to one of claims 1 to 3 further comprising a device configured to
provide gaze stability data of the user, the attention estimation module (260) being
configured to generate attention data based on the gaze stability data provided.
5. System according to one of the preceding claims wherein the head data comprise head
orientation data.
6. System according to one of the preceding claims wherein the audio beamforming module
(262) comprises a beamforming model configured to receive as inputs at least the attention
data and audio signals of the microphone array (23, 24) and to determine as output
a selection of beamforming processing settings to apply to the audio signals.
7. System according to claim 1 wherein the attention estimation module (260) comprises
a model configured to receive as inputs the user's gaze direction, the head data and
the vergence data and to determine as output attention data of the user representative
of a point of attention of the user.
8. System according to one of the preceding claims, comprising an activation device (50,)
for the audio beamforming module (262), said activation device (50) being configured
to:
- receive gaze direction data of the user,
- receive at least vergence data of the user,
- activate the audio beamforming module (262) when gaze direction data are above a
first preset threshold (THR1) and/or vergence data are above a second preset threshold
(THR2).
9. System according to claim 8 wherein the activation device (50) is also configured
to receive cortical activity data of the user and to activate the audio beamforming
device where, additionally, said cortical activity data are above a third preset threshold
(THR3).
10. Head mountable device for hearing assistance to a user comprising:
- a microphone array (23, 24) providing audio signals,
- at least one hearing device (28),
- an eye-tracking device (21) configured to provide user's gaze direction data,
- a vergence device configured to provide user's vergence data,
- a head tracking device configured to provide user's head data,
- a processing unit (26) comprising:
- an attention estimation module (260) configured to generate attention data of the
user representative of a point of attention of the user based on the user's gaze direction
data, head data and vergence data,
- an audio beamforming module (262) configured to generate a beamformed audio signal
intended to enhance sounds for the user through said at least one hearing device (28)
based on the audio signals and the attention data of the user representative of a
point of attention of the user.
11. Eyewear for hearing assistance to a user comprising:
- a microphone array (23, 24) providing audio signals,
- at least one hearing device (28),
- an eye-tracking device (21) configured to provide user's gaze direction and stabilization
data,
- a vergence device configured to provide user's vergence data,
- a head tracking device configured to provide user's head data
- a processing unit (26) comprising:
- an attention estimation module (260) configured to generate attention data of the
user representative of a point of attention of the user based on the user's gaze direction
data, head data and vergence data,
- an audio beamforming module (262) configured to generate a beamformed audio signal
intended to enhance sounds for the user through said at least one hearing device (28)
based on the audio signals and the attention data of the user representative of a
point of attention of the user.
12. A computer implemented method for hearing assistance to a user, said method comprising
the steps of:
- receiving (110) at least an audio signal captured with a microphone array,
- providing (120) gaze direction data of the user,
- providing (130) vergence data of the user,
- providing head data of the user,
- determining (150) attention data of the user representative of a point of attention
of the user within his environment based on the user's gaze direction data, head data
and vergence data,
- generating (160) a beamformed audio signal from the audio signal and the attention
data,
- delivering (170) the beamformed audio signal to the user through a hearing device.
13. A computer implemented method according to claim 12, comprising the following steps:
- providing (140) cortical activity data of the user,
- determining (150) attention data of the user representative of a point of attention
of the user within his/her environment based on the cortical activity data.
14. A computer implemented method according to one of the claims 12 to 13, wherein steps
150, 160 and/or 170 are implemented with machine learning based algorithms.
15. A non-transitory computer-readable medium storing computer-executable instructions
that, when executed by at least one processor, cause the processor to perform operations
comprising:
- receiving (110) at least an audio signal captured with a microphone array,
- providing (120) gaze direction data of the user,
- providing (130) vergence data of the user,
- providing head data of the user,
- determining (150) attention data of the user representative of a point of attention
of the user within his environment based on the user's gaze direction data, head data
and vergence data,
- generating (160) a beamformed audio signal from the audio signal and the attention
data,
- delivering (170) the beamformed audio signal to the user through a hearing device.