Granted Patent B2

US 12,739,576 · App. 18/617,089 · Granted Sep 15, 2026

Inventors: Olaf Strelcyk (Cincinnati, OH); Charlotte Vercammen (London, GB); Laurent Simon (Stäfa, CH); Stefan Klockgether (Hombrechtikon, CH); Gilles Courtois (Uerikon, CH)

Assignee: Sonova AG

H04R25/505G06F3/012H04R2225/43

Abstract

Claims (56)

1 . A system comprising:

a memory storing instructions; and

one or more processors communicatively coupled to the memory and configured to execute instructions to perform a process comprising:

receiving, from an input transducer included in a hearing device configured to be worn by a user, audio data representative of one or more audio signals presented to the user, the audio data including one or more speech signals representative of a speech from one or more speech sources different from the user;

acquiring one or more of:

motion data received from a motion sensor included in the hearing device, the motion data representative of head movements of the user while the user wears the hearing device; or

own-voice data included in the audio data, the own-voice data representative of an own-voice activity of the user;

determining, based on at least one of the motion data or the own-voice data, a listening state of the user with respect to the one or more speech signals, wherein the determining the listening state comprises:

determining a paying attention state representative of whether or to which degree the user intends to pay attention to any of the one or more speech signals; and

determining a comprehension state representative of whether or which degree the user comprehends the one or more speech signals; and

performing, based on the listening state, an operation associated with the hearing device, wherein the operation comprises:

activating one or more sound processing properties of the hearing device when the listening state includes both of an attentive state representative of the user intending to pay attention to the one or more speech signals and a comprehending state representative of the user comprehending the one or more speech signals; and

deactivating the one or more sounds processing properties of the hearing device when the listening state includes one or both of an inattentive state representative of the user not intending to pay attention to any of the one or more speech signals or an uncomprehending state representative of the user not comprehending the one or more speech signals.

2 . The system of claim 1 , wherein the determining the listening state is based on one or more of a frequency of the head movements, a direction of the head movements, a magnitude of the head movements, an acceleration of the head movements, a timing of the head movements, or a duration of the head movements.

3 . The system of claim 1 , wherein the determining the listening state includes classifying the listening state as one or more of an inattentive uncomprehending listening state, an attentive uncomprehending listening state, or a comprehending listening state.

4 . The system of claim 1 , wherein the determining the comprehension state includes determining that the user comprehends the speech based on a frequency of the head movements being within a predetermined frequency range representative of the speech comprehension.

5 . The system of claim 4 , wherein the frequency range representative of the speech comprehension comprises frequencies larger than 2 Hertz.

6 . The system of claim 4 , wherein the determining the paying attention state includes determining that the user is paying attention to one or more of the speech sources without comprehending the speech based on a proportion of the head movements having a frequency within the frequency range representative of the speech comprehension and the head movements having a frequency within a predetermined frequency range representative of the user paying attention to one or more of the speech sources.

7 . The system of claim 6 , wherein the frequency range representative of the user paying attention to one or more of the speech sources comprises frequencies smaller than the frequencies in the frequency range representative of the speech comprehension.

8 . The system of claim 4 , wherein the determining that the user comprehends the speech is further based on a proportion of time during which the head movements are within the frequency range representative of the speech comprehension.

9 . The system of claim 1 , wherein the determining the comprehension state includes one or more of determining that the user comprehends the speech based on a frequency of the head movements corresponding to a frequency of the speech or determining that, when the audio data includes one or more music signals representative of a music, the user is listening to the music based on a frequency of the head movements corresponding to a frequency of the music.

10 . The system of claim 1 , wherein the determining the comprehension state includes determining that the user comprehends the speech based on one or more backchannels to the speech in the own-voice data fulfilling a predetermined property representative of the speech comprehension.

11 . The system of claim 10 , wherein the determining the paying attention state includes determining that the user is paying attention to one or more of the speech sources without comprehending the speech based on a proportion of the one or more backchannels fulfilling the property representative of the speech comprehension and one or more backchannels fulfilling a property representative of the user paying attention to one or more of the speech sources.

12 . The system of claim 1 , wherein the operation includes selecting one or more of the speech signals for one or more of:

an enrollment as an audio sample in an audio processing algorithm configured to provide for a processing of the audio data based on the enrolled audio sample; or

a determining a direction of arrival (DOA) of the speech, wherein the direction of arrival is employed in an audio processing algorithm configured to provide for a processing of the audio data based on the direction of arrival.

13 . The system of claim 1 , wherein the activating the one or more sound processing properties includes one or more of increasing a gain of the hearing device, increasing a volume of an output signal transmitted by an output transducer included in the hearing device, modifying a property of a beamforming performed by the hearing device, selecting one or more microphones included in the hearing device to detect the one or more audio signals, or extracting or separating one or more of the speech signals from the audio data.

14 . The system of claim 1 , wherein the activating the one or more sound processing properties is further based on determining a target speech signal from the one or more speech signals included in the audio data.

15 . The system of claim 1 , wherein the activating the one or more sound processing properties is further based on determining a listening effort exerted by the user.

16 . The system of claim 1 , wherein the operation further includes deactivating the one or more sound processing properties based on one or more of when the listening state indicates that a comprehension of the speech by the user decreases or when the listening state indicates that an attention payment of the user to the one or more speech sources decreases.

17 . The system of claim 1 , wherein the operation includes providing a notification indicating one or more of the listening state or information derived from the listening state.

18 . A hearing device configured to be worn by a user, the hearing device comprising:

an input transducer;

a motion sensor; and

a processing unit communicatively coupled to the input transducer and the motion sensor, the processing unit configured to:

receive, from the input transducer, audio data representative of one or more audio signals presented to the user, the audio data including one or more speech signals representative of a speech from one or more speech sources different from the user;

acquire one or more of:

motion data received from a motion sensor, the motion data representative of head movements of the user while the user wears the hearing device; or

own-voice data included in the audio data, the own-voice data representative of an own-voice activity of the user;

determine, based on at least one of the motion data or the own-voice data, a listening state of the user with respect to the one or more speech signals, wherein the determining the listening state comprises:

determining a paying attention state representative of whether or to which degree the user intends to pay attention to any of the one or more speech signals; and

determining a comprehension state representative of whether or which degree the user comprehends the one or more speech signals; and

perform, based on the listening state, an operation associated with the hearing device, wherein the operation comprises:

activating one or more sound processing properties of the hearing device when the listening state includes both of an attentive state representative of the user intending to pay attention to the one or more speech signals and a comprehending state representative of the user comprehending the one or more speech signals; and

deactivating the one or more sounds processing properties of the hearing device when the listening state includes one or both of an inattentive state representative of the user not intending to pay attention to any of the one or more speech signals or an uncomprehending state representative of the user not comprehending the one or more speech signals.

19 . A method comprising:

receiving, from an input transducer included in a hearing device configured to be worn by a user, audio data representative of one or more audio signals presented to the user, the audio data including one or more speech signals representative of a speech from one or more speech sources different from the user;

acquiring one or more of:

motion data received from a motion sensor included in the hearing device, the motion data representative of head movements of the user while the user wears the hearing device; or

own-voice data included in the audio data, the own-voice data representative of an own-voice activity of the user;

determining, based on at least one of the motion data or the own-voice data, a listening state of the user with respect to the one or more speech signals, wherein the determining the listening state comprises:

determining a paying attention state representative of whether or to which degree the user intends to pay attention to any of the one or more speech signals; and

determining a comprehension state representative of whether or which degree the user comprehends the one or more speech signals; and

performing, based on the listening state, an operation associated with the hearing device, wherein the operation comprises:

activating one or more sound processing properties of the hearing device when the listening state includes both of an attentive state representative of the user intending to pay attention to the one or more speech signals and a comprehending state representative of the user comprehending the one or more speech signals; and

deactivating the one or more sounds processing properties of the hearing device when the listening state includes one or both of an inattentive state representative of the user not intending to pay attention to any of the one or more speech signals or an uncomprehending state representative of the user not comprehending the one or more speech signals.

Assignments (1)

Continuity (1)

References Cited (17)