IP Library Granted Patent US 11,412,333
Granted Patent B2
US 11,412,333 · App. 16/192,333 · Granted Aug 9, 2022

Interactive system for hearing devices

Inventors: Tao Zhang (Eden Prairie, MN); Eric Durant (Milwaukee, WI); Dean G. Meyer (Mound, MN); Martin McKinney (Minneapolis, MN); Matthew D. Kleffner (Eden Prairie, MN); Dominic Perz (Bloomington, MN); Karrie Recker (Edina, MN)
Assignee: Starkey Laboratories, Inc.
H04R25/505G06F3/165G10L15/063H04R25/407H04R25/70G10L2015/225H04R25/30H04R25/552H04R25/554H04R2225/39H04R2225/41H04R2225/43H04R2225/55
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,412,333
App. No.
16/192,333
Granted
Aug 9, 2022
Kind
B2
Abstract

In an audio signal, one or more processing circuits recognize spoken content in a user's own speech signal using speech recognition and natural language understanding. The spoken content describes a listening difficulty of the user. The one or more processing circuits generate, based on the spoken content, one or more actions for hearing devices and feedback for the user. The one or more actions attempt to resolve the listening difficulty. Additionally, the one or more processing circuits convert the user feedback to verbal feedback using speech synthesis and transmit the one or more actions and the verbal feedback to the hearing devices via a body-worn device. The hearing devices are configured to perform the one or more actions and play back the verbal feedback to the user.

Claims (60)

1. A sound processing system comprising:

a pair of non-implantable hearing devices;

a body-worn device, and a cloud-based computing system, wherein:

the cloud-based computing system comprises one or more processing circuits and is configured to connect with multiple body-worn devices wirelessly at the same time,

the body-worn device comprises one or more processing circuits and is configured to connect to either or both of the hearing devices wirelessly and is configured to connect with the cloud-based computing system wirelessly,

each of the hearing devices comprises at least one microphone and a receiver,

for at least one hearing device of the pair of hearing devices, the hearing device is configured to use the at least one microphone of the hearing device to detect a user's own speech signal,

the body-worn device is configured to wirelessly receive the user's own speech signal from one or more of the hearing devices and to wirelessly transmit the user's own speech signal to the cloud-based computing system,

the one or more processing circuits of the cloud-based computing system are configured to recognize spoken content in the user's own speech signal using speech recognition and natural language understanding, the spoken content describing a listening difficulty of the user;

the one or more processing circuits of the cloud-based computing system perform activities of a cognitive engine that is configured to generate, based on the spoken content describing the listening difficulty of the user, one or more actions for the hearing devices and feedback for the user, wherein the one or more actions attempt to resolve the listening difficulty by making adjustments to hearing device states of one or more of the hearing devices and the feedback is responsive to the spoken content describing the listening difficulty of the user;

the one or more processing circuits of the cloud-based computing system are further configured to convert the feedback to verbal feedback using speech synthesis;

the cloud-based computing system is configured to transmit the one or more actions and the verbal feedback to the hearing devices via the body-worn device,

the hearing devices are configured to perform the one or more actions and play back the verbal feedback to the user using the receivers in a binaural way,

the body-worn device or the cloud-based computing system is configured to ask the user whether to conference in a hearing professional in response to receiving an indication from the user that the one or more actions were unsuccessful in resolving the listening difficulty of the user;

in response to an indication from the user to conference in the hearing professional, the body-worn device or the cloud-based computing system is configured to conference in the hearing professional over a network connection to resolve the listening difficulty; and

the cognitive engine is configured to learn from what the hearing professional does to resolve the listening difficulty.

2. The sound processing system of claim 1 , wherein the cognitive engine is further configured to learn to generate the one or more actions based in part on what other users do in situations similar to that of the user.

3. The sound processing system of claim 1 , wherein one of the hearing devices is a master hearing device, and the master hearing device is configured to enhance the user's own speech signal and to reduce background noise and interference using microphones from both the hearing devices.

4. The sound processing system of claim 1 , wherein one of the hearing devices is a master hearing device, and the master hearing device is configured to enhance the user's own speech signal by beamforming all microphones on the hearing devices followed by postfiltering.

5. The sound processing system of claim 1 , wherein one of the hearing devices is a master hearing device, and the master hearing device is configured to use an activation trigger to activate wireless transmission of the user's own speech signal to the body-worn device.

6. The sound processing system of claim 1 , wherein the one or more processing circuits of the cloud-based computing system perform activities of a cognitive engine that is configured to:

generate the one or more actions and the feedback based on at least one of: the spoken content and what the cognitive engine learns from a current acoustic or geographic environment, a current hearing device state, a hearing profile of the user, preferences of the user, experiences of the user, or patterns of the user.

7. The sound processing system of claim 1 , wherein the actions include activation of binaural noise reduction.

8. The sound processing system of claim 1 , wherein the one or more actions include adjusting one or more compressor parameters.

9. The system of claim 1 , wherein the one or more processing circuits of the cloud-based computing system are further configured to:

receive a name for the hearing device states from the user;

associate the hearing device states with the name;

receive a spoken request from the user to change current hearing device states to the hearing device states associated with the name; and

in response to the spoken request, change the current hearing devices states to the hearing device states associated with the name.

10. A method for making adjustments to hearing devices of a sound processing system, the method comprising:

receiving, by a cloud-based computing system of the sound processing system, a user's own speech signal from a body-worn device of the sound processing system, wherein the user's own speech signal is detected by at least one microphone of the hearing devices, wherein at least one of the hearing devices wirelessly transmits the user's own speech signal to the body-worn device;

recognizing, by the cloud-based computing system, spoken content in the user's own speech signal using speech recognition and natural language understanding, the spoken content describing a listening difficulty of the user;

generating, by a cognitive engine implemented by the cloud-based computing system, based on the spoken content describing the listening difficulty of the user, one or more actions for the hearing devices and feedback for the user, wherein the one or more actions attempt to resolve the listening difficulty by making adjustments to hearing device states of one or more of the hearing devices and the feedback is responsive to the spoken content describing the listening difficulty of the user;

converting, by the cloud-based computing system, the feedback to verbal feedback using speech synthesis;

transmitting, by the cloud-based computing system, the one or more actions and the verbal feedback to the hearing devices via the body-worn device;

asking, by the body-worn device or the cloud-based computing system, the user whether to conference in a hearing professional in response to receiving an indication from the user that the one or more actions were unsuccessful in resolving the listening difficulty of the user;

in response to an indication from the user to conference in the hearing professional, conferencing in, by the body-worn device or the cloud-based computing system, the hearing professional over a network connection to resolve the listening difficulty; and

learning, by the cognitive engine, from what the hearing professional does to resolve the listening difficulty.

11. The method of claim 10 , wherein the cognitive engine is further configured to learn to generate the actions based in part on what other users do in situations similar to that of the user.

12. The method of claim 10 , wherein one of the hearing devices is a master hearing device, and the master hearing device is configured to enhance the own speech signal and to reduce background noise and interference using microphones from both the hearing devices.

13. The method of claim 10 , wherein one of the hearing devices is a master hearing device, and the master hearing device is configured to enhance the user's own speech signal by beamforming of all microphones of the hearing devices followed by postfiltering.

14. The method of claim 10 , wherein one of the hearing devices is a master hearing device, and the master hearing device is configured to use an activation trigger to activate wireless transmission of the user's own speech signal to the body-worn device.

15. The method of claim 10 , wherein the one or more processing circuits of the cloud-based computing system perform activities of a cognitive engine that is configured to:

generate the one or more actions and the feedback based on at least one of: the spoken content and what the cognitive engine learns from a current acoustic or geographic environment, a current hearing device state, a hearing profile of the user, or preferences of the user, experiences of the user, or patterns of the user.

16. The method of claim 10 , wherein the one or more actions include adjusting one or more compressor parameters.

17. The method of claim 10 , further comprising:

receiving a name for the hearing device states from the user;

associating the hearing device states with the name;

receiving a spoken request from the user to change current hearing device states to the hearing device states associated with the name; and

in response to the spoken request, changing the current hearing devices states to the hearing device states associated with the name.

18. A non-transitory computer-readable data storage medium having instructions thereon that, when executed, cause a sound processing system to:

receive, by a cloud-based computing system of the sound processing system, a user's own speech signal from a body-worn device of the sound processing system, wherein the user's own speech signal is detected by at least one microphone of a pair of hearing devices, wherein at least one of the hearing devices wirelessly transmits the user's own speech signal to the body-worn device;

recognize, by the cloud-based computing system, spoken content in the user's own speech signal using speech recognition and natural language understanding, the spoken content describing a listening difficulty of the user;

generate, by a cognitive engine implemented by the cloud-based computing system, based on the spoken content describing the listening difficulty of the user, one or more actions for the hearing devices and feedback for the user, wherein the one or more actions attempt to resolve the listening difficulty by making adjustments to hearing device states of one or more of the hearing devices and the feedback is responsive to the spoken content describing the listening difficulty of the user;

convert, by the cloud-based computing system, the feedback to verbal feedback using speech synthesis;

transmit, by the cloud-based computing system, the one or more actions and the verbal feedback to the hearing devices via the body-worn device;

ask, by the body-worn device or the cloud-based computing system, the user whether to conference in a hearing professional in response to receiving an indication from the user that the one or more actions were unsuccessful in resolving the listening difficulty of the user;

in response to an indication from the user to conference in the hearing professional, conference in the hearing professional over a network connection to resolve the listening difficulty; and

configure the cognitive engine to learn from what the hearing professional does to resolve the listening difficulty.

19. The non-transitory computer-readable data storage medium of claim 18 , wherein execution of the instructions further causes the cognitive engine to learn to generate the actions based in part on what other users do in situations similar to that of the user.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 22, 2019
From: ZHANG, TAO; DURANT, ERIC; MEYER, DEAN G.; MCKINNEY, MARTIN; KLEFFNER, MATTHEW D.; PERZ, DOMINIC; RECKER, KARRIE
To: STARKEY LABORATORIES, INC.
Reel/Frame 048095/0732 →
Continuity (2)
Provisional Application 62586561 · Nov 15, 2017
Related Publication 20190149927A1 · May 16, 2019