IP Library Granted Patent US 11,622,197
Granted Patent B2
US 11,622,197 · App. 17/006,476 · Granted Apr 4, 2023

Audio enhancement for hearing impaired in a shared listening environment

Inventors: Mahyar Nejat (San Diego, CA); Brant Candelore (San Diego, CA); Peter Shintani (San Diego, CA)
Assignee: SONY GROUP CORPORATION
H04R5/04G06T7/0012H04L65/75H04R3/04H04R5/033H04R29/001H04S1/007H04W72/1236G06T2207/30196H04R2430/01
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,622,197
App. No.
17/006,476
Granted
Apr 4, 2023
Kind
B2
Abstract

An electronic apparatus and method for enhancement of audio for users with a hearing impairment in a shared listening environment is provided. The electronic apparatus receives first audio content from a media source and detects a first user with a hearing disability as a wearer of the personal listening device. The electronic apparatus modifies one or more features of the first audio content based on an audio enhancement profile associated with the detected first user. Thereafter, the electronic apparatus generates second audio content based on the modification and shares the generated second audio content with the personal listening device.

Claims (70)

1. An electronic apparatus, comprising:

circuitry communicatively coupled to a personal listening device, wherein the circuitry is configured to:

receive first audio content from a media source;

detect a first user with a hearing disability as a wearer of the personal listening device;

modify one or more features of the first audio content based on an audio enhancement profile associated with the detected first user;

generate second audio content based on the modification;

share the generated second audio content with the personal listening device;

share the received first audio content with an audio reproductive system;

control the audio reproductive system to play the first audio content concurrently with a playback of the second audio content on the personal listening device for a shared listening experience, wherein

the audio reproductive system is different from the personal listening device, and

the audio enhancement profile is an audiogram comprising a hearing curve of the first user for both a left ear and a right ear of the first user; and

select an audio portion as one of a left channel audio or a right channel audio of the first audio content based on a determination that the hearing curve for the first user indicates a hearing impairment condition for one of the left ear or the right ear of the first user.

2. The electronic apparatus according to claim 1 , wherein the audio reproductive system comprises one or more of an external wireless speaker, a set of internal speakers, an external wired speaker, a sub-woofer, a tweeter, a soundbar, or an optical audio device.

3. The electronic apparatus according to claim 1 , further comprising an image-capture device communicatively coupled to the circuitry, wherein the circuitry is further configured to:

control the image-capture device to capture an image of a listening environment comprising the first user and the personal listening device; and

detect the first user as the wearer of the personal listening device based on the captured image.

4. The electronic apparatus according to claim 1 , wherein the circuitry is further configured to: compare the hearing curve of the first user with a reference hearing curve of a second user with normal hearing levels; and determine the one or more features of the first audio content for the modification based on the comparison.

5. The electronic apparatus according to claim 1 , wherein the circuitry is further configured to: modify the one or more features of the selected first audio portion of the first audio content; and generate the second audio content based on the modification.

6. The electronic apparatus according to claim 1 , wherein the circuitry is further configured to:

control the personal listening device to sequentially play a set of test tones at a corresponding set of audible frequencies,

wherein loudness of each test tone in the set of test tones increases while the respective test tone is played on the personal listening device;

receive a set of user inputs while the set of test tones is played sequentially on the personal listening device, wherein each user input indicates a hearing threshold level for the respective test tone; and

generate the hearing curve on the audiogram as the audio enhancement profile for the first user based on the received set of user inputs.

7. The electronic apparatus according to claim 1 , further comprising a display device configured to display video content that is associated with the first audio content.

8. The electronic apparatus according to claim 7 , wherein the circuitry is further configured to:

compute a first processing delay associated with encoding of the second audio content for a wireless sharing with the personal listening device;

compute a second processing delay associated with decoding of the second audio content at the personal listening device; and

calibrate a time of the display of the video content to synchronize with the playback of the second audio content on the personal listening device based on the computed first processing delay and the second processing delay.

9. The electronic apparatus according to claim 1 , wherein the one or more features comprises an amplitude of a number of audio frames of the first audio content, amplitude levels of a band of audible frequencies of the first audio content, a dynamic range of the first audio content, amplitude levels of speech frames in the first audio content, or a regulation speed for adjustment in gain levels for the audio frames of the first audio content.

10. The electronic apparatus according to claim 1 , wherein the circuitry is further configured to:

control the personal listening device to capture a voice input from the wearer of the personal listening device; and

detect the first user as the wearer of the personal listening device based on the voice input.

11. The electronic apparatus according to claim 1 , wherein the circuitry is further configured to determine whether the detected first user is wearing the personal listening device.

12. A method, comprising:

receiving first audio content from a media source;

detecting a first user with a hearing disability as a wearer of a personal listening device;

modifying one or more features of the first audio content based on an audio enhancement profile associated with the detected first user;

generating second audio content based on the modification;

sharing the generated second audio content with the personal listening device;

sharing the received first audio content with an audio reproductive system;

controlling the audio reproductive system to play the first audio content concurrently with a playback of the second audio content on the personal listening device for a shared listening experience, wherein

the audio reproductive system is different from the personal listening device, and

the audio enhancement profile is an audiogram comprising a hearing curve of the first user for both a left ear and a right ear of the first user; and

selecting an audio portion as one of a left channel audio or a right channel audio of the first audio content based on a determination that the hearing curve for the first user indicates a hearing impairment condition for one of the left ear or the right ear of the first user.

13. The method according to claim 12 , further comprising:

controlling an image-capture device to capture an image of a listening environment comprising the first user and the personal listening device; and

detecting the first user as the wearer of the personal listening device based on the captured image.

14. The method according to claim 12 , further comprising: comparing the hearing curve of the first user with a reference hearing curve of a second user with normal hearing levels; and determining the one or more features of the first audio content for the modification based on the comparison.

15. The method according to claim 12 , further comprising: modifying the one or more features of the selected first audio portion of the first audio content; and generating the second audio content based on the modification.

16. The method according to claim 12 , further comprising:

controlling the personal listening device to sequentially play a set of test tones at a corresponding set of audible frequencies,

wherein loudness of each test tone in the set of test tones increases while the respective test tone is played on the personal listening device;

receiving a set of user inputs while the set of test tones is played sequentially on the personal listening device, wherein each user input indicates a hearing threshold level for the respective test tone; and

generating the hearing curve on the audiogram as the audio enhancement profile for the first user based on the received set of user inputs.

17. The method according to claim 12 , further comprising:

computing a first processing delay associated with encoding of the second audio content for a wireless sharing with the personal listening device;

computing a second processing delay associated with decoding of the second audio content at the personal listening device; and

calibrating a time of a display of a video content associated with the first audio content for synchronizing with the playback of the second audio content on the personal listening device based on the computed first processing delay and the second processing delay.

18. An electronic apparatus, comprising:

circuitry communicatively coupled to a personal listening device, wherein the circuitry is configured to:

retrieve an audio enhancement profile associated with a first user;

receive first audio content from a media source;

modify one or more features of the first audio content based on the retrieved audio enhancement profile;

generate second audio content based on the modification;

share the generated second audio content with the personal listening device;

share the received first audio content with an audio reproductive system;

control the audio reproductive system to play the first audio content concurrently with a playback of the second audio content on the personal listening device, wherein

the audio reproductive system is different from the personal listening device, and

the audio enhancement profile is an audiogram comprising a hearing curve of the first user for both a left ear and a right ear of the first user; and

select an audio portion as one of a left channel audio or a right channel audio of the first audio content based on a determination that the hearing curve for the first user indicates a hearing impairment condition for one of the left ear or the right ear of the first user.

Assignments (2)
CHANGE OF NAME Recorded Aug 20, 2021
From: SONY CORPORATION
To: SONY GROUP CORPORATION
Reel/Frame 057249/0634 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 11, 2020
From: NEJAT, MAHYAR; CANDELORE, BRANT; SHINTANI, PETER
To: SONY CORPORATION
Reel/Frame 054333/0856 →
Continuity (1)
Related Publication 20220070583A1 · Mar 3, 2022
Cited By (2)
US 12,573,407 US 12,602,197