IP Library Granted Patent US 11,432,094
Granted Patent B2
US 11,432,094 · App. 17/175,920 · Granted Aug 30, 2022

Three-dimensional (3D) audio notification for vehicle

Inventors: Ted Yamazaki (San Diego, CA); Tonni Larsen (San Diego, CA); Leigh Anderson (Novi, MI)
Assignee: SONY GROUP CORPORATION
H04S7/30B60R11/0217G06V20/58H04R3/12H04R5/04H04R2499/13
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,432,094
App. No.
17/175,920
Granted
Aug 30, 2022
Kind
B2
Abstract

A system including circuitry communicably coupled to a plurality of sensors and a plurality of audio output devices are provided. The circuitry receives a plurality of signals from the plurality of sensors. The circuitry further detects a type of a first object and a position of the first object with respect to a first vehicle based on the received plurality of signals. The circuitry further generates metadata associated with the first object based on the detected type of the first object and the position of the first object. The metadata comprises a first three-dimensional (3D) audio object. The circuitry further controls a first set of audio output devices of the plurality of audio output devices to output the first 3D audio object in a 3D space of the first vehicle at a first time instant.

Claims (84)

1. A system, comprising:

circuitry communicatively coupled to a plurality of sensors and a plurality of audio output devices, wherein the circuitry is configured to:

receive a plurality of signals from the plurality of sensors;

detect a type of a first object and a position of the first object with respect to a first vehicle based on the received plurality of signals;

generate metadata associated with the first object based on the detected type of the first object and the position of the first object, wherein

the metadata comprises a first three-dimensional (3D) audio object, and

the first 3D audio object is at least one of a sound alert or a verbal notification;

receive a user input from an electronic device associated with a driver of the first vehicle, wherein the user input corresponds to a selection of a type of the sound alert among a plurality of types of sound alerts for each type of the first object; and

control a first set of audio output devices of the plurality of audio output devices to output the type of the sound alert as the first 3D audio object in a 3D space of the first vehicle at a first time instant.

2. The system according to claim 1 , wherein the circuitry is further configured to:

generate a textual notification based on the detected type of the first object and the position of the first object with respect to the first vehicle, wherein the textual notification corresponds to the verbal notification; and

control a display device associated with one of the first vehicle or the electronic device associated with the driver of the first vehicle to display the generated textual notification at the first time instant.

3. The system according to claim 1 , wherein

the metadata further comprises information associated with the first object,

the information associated with the first object comprises at least one of the detected type of the first object, a direction of the position of the first object with respect to the first vehicle, a distance of the first object from the first vehicle, a size of the first object, or a side of the first vehicle on which the first object is located, and

the verbal notification indicates the information associated with the first object.

4. The system according to claim 1 , wherein

the sound alert corresponds to at least one of: an engine sound, an artificial rumble sound, a vehicle horn sound, or a nature sound, and

the verbal notification corresponds to a synthesized speech sound.

5. The system according to claim 1 , wherein the circuitry is further configured to:

control the first set of audio output devices to output the first 3D audio object as a virtual sound source in the 3D space; and

switch from the first set of audio output devices to a second set of audio output devices of the plurality of audio output devices to continuously change a position and volume of the virtual sound source in the 3D space, wherein the position and the volume of the virtual sound source is changed based on a change in the position of the first object with respect to the first vehicle.

6. The system according to claim 5 , wherein

the 3D space comprises at least one of an interior space of the first vehicle or a space between the first vehicle and the first object, and

the circuitry is further configured to control the position of the virtual sound source to correspond to one of the interior space of the first vehicle or the space between the first vehicle and the first object.

7. The system according to claim 1 , wherein the circuitry is further configured to:

detect presence of a second object in a vicinity of the first vehicle;

detect a type of the second object and a position of the second object with respect to the first vehicle;

generate the metadata based on the detection of the type of the second object and the position of the second object with respect to the first vehicle, wherein the metadata further comprises a second 3D audio object; and

control the plurality of output audio devices to sequentially output the first 3D audio object at the first time instant and the second 3D audio object at a second time instant, based on a time synchronization between the first 3D audio object and the second 3D audio object.

8. The system according to claim 7 , wherein the circuitry is further configured to:

determine a priority for the output of the first 3D audio object and the second 3D audio object based on at least one of: the detected type of the first object and the detected type of the second object, a proximity of each of the position of the first object and the position of the second object with respect to the first vehicle, a speed of movement of each of the first object and the second object, a size of each of the first object and the second object, or a side of the first vehicle on which each of the first object and the second object is located.

9. The system according to claim 7 , wherein the circuitry is further configured to:

determine one of an operation of the first vehicle or a gesture of the driver of the first vehicle; and

determine a priority for the output of the first 3D audio object and the second 3D audio object based on the one of the operation of the first vehicle or the gesture of the driver of the first vehicle.

10. The system according to claim 1 , wherein the circuitry is further configured to:

control a second set of output audio devices of the plurality of output audio devices to gradually reduce a volume of a currently playing audio, in case the output of the first 3D audio object by the first set of output audio devices is started; and

control the second set of output audio devices to gradually increase the volume of the currently playing audio, in case the output of the first 3D audio object by the first set of output audio devices is completed.

11. The system according to claim 1 , wherein

the plurality of sensors is positioned at different positions on the first vehicle, and

the plurality of sensors comprises at least one of: an image sensor, a light detection and ranging sensor, or an audio sensor.

12. The system according to claim 1 , wherein

the type of the first object is one of a stationary object or a moving object, and

the circuitry is further configured to control the plurality of output audio devices to:

output the first 3D audio object as the verbal notification in case the type of the first object is the stationary object; and

output the first 3D audio object as the sound alert in case the type of the first object is the moving object.

13. The system according to claim 12 , wherein, in case the type of the first object is the moving object, the circuitry is further configured to:

determine a speed of movement of the first object based on a change in the position of the first object with respect to the first vehicle; and

generate the metadata associated with the first object based on the determined speed.

14. The system according to claim 1 , wherein

the type of the first object comprises one of a second vehicle, a rumble strip, a median, a pedestrian, or an obstacle, and

the circuitry is further configured to control a sound of the first 3D audio object based on the detected type of the first object.

15. A method, comprising:

in a system comprising circuitry communicatively coupled to a plurality of sensors and a plurality of audio output devices:

receiving a plurality of signals from the plurality of sensors;

detecting a type of a first object and a position of the first object with respect to a first vehicle based on the received plurality of signals;

generating metadata associated with the first object based on the detected type of the first object and the position of the first object, wherein

the metadata comprises a first three-dimensional (3D) audio object, and

the first 3D audio object is at least one of a sound alert or a verbal notification;

receiving a user input from an electronic device associated with a driver of the first vehicle, wherein the user input corresponds to a selection of a type of the sound alert among a plurality of types of sound alerts for each type of the first object; and

controlling a first set of audio output devices of the plurality of audio output devices to output the type of the sound alert as the first 3D audio object in a 3D space of the first vehicle at a first time instant.

16. The method according to claim 15 , further comprising:

detecting presence of a second object in a vicinity of the first vehicle;

detecting a type of the second object and a position of the second object with respect to the first vehicle;

generating the metadata based on the detection of the type of the second object and the position of the second object with respect to the first vehicle, wherein the metadata further comprises a second 3D audio object; and

controlling the plurality of output audio devices to sequentially output the first 3D audio object at the first time instant and the second 3D audio object at a second time instant, based on a time synchronization between the first 3D audio object and the second 3D audio object.

17. The method according to claim 16 , further comprising

determining a priority for the output of the first 3D audio object and the second 3D audio object based on at least one of: the detected type of the first object and the detected type of the second object, a proximity of each of the position of the first object and the position of the second object with respect to the first vehicle, a speed of movement of each of the first object and the second object, a size of each of the first object and the second object, or a side of the first vehicle on which each of the first object and the second object is located.

18. A non-transitory computer-readable medium having stored thereon, computer-executable instructions that when executed by a system communicatively coupled to a plurality of sensors and a plurality of audio output devices, causes the system to execute operations, the operations comprising:

receiving a plurality of signals from the plurality of sensors;

detecting a type of a first object and a position of the first object with respect to a first vehicle based on the received plurality of signals;

generating metadata associated with the first object based on the detected type of the first object and the position of the first object, wherein

the metadata comprises a first three-dimensional (3D) audio object, and

the first 3D audio object is at least one of a sound alert or a verbal notification;

receiving a user input from an electronic device associated with a driver of the first vehicle, wherein the user input corresponds to a selection of a type of the sound alert among a plurality of types of sound alerts for each type of the first object; and

controlling a first set of audio output devices of the plurality of audio output devices to output the type of the sound alert as the first 3D audio object in a 3D space of the first vehicle at a first time instant.

19. A system, comprising:

circuitry communicatively coupled to a plurality of sensors and a plurality of audio output devices, wherein the circuitry is configured to:

receive a plurality of signals from the plurality of sensors;

detect a type of a first object and a position of the first object with respect to a first vehicle based on the received plurality of signals;

detect a type of a second object and a position of the second object with respect to the first vehicle based on the received plurality of signals;

generate first metadata associated with the first object based on the detected type of the first object and the position of the first object, wherein the first metadata comprises a first three-dimensional (3D) audio object;

generate second metadata associated with the second object based on the detected type of the second object and the position of the second object, wherein the second metadata comprises a second three-dimensional (3D) audio object; and

control a first set of audio output devices of a plurality of output devices to sequentially output the first 3D audio object in a 3D space of the first vehicle at a first time instant and control a second set of audio output devices of the plurality of output devices to sequentially output the second 3D audio object in the 3D space of the first vehicle at a second time instant based on a time synchronization between the first 3D audio object and the second 3D audio object.

Assignments (2)
CHANGE OF NAME Recorded Aug 10, 2021
From: SONY CORPORATION
To: SONY GROUP CORPORATION
Reel/Frame 057156/0652 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 21, 2021
From: YAMAZAKI, TED; LARSEN, TONNI; ANDERSON, LEIGH
To: SONY CORPORATION
Reel/Frame 055989/0763 →
Continuity (2)
Provisional Application 63089402 · Oct 8, 2020
Related Publication 20220116724A1 · Apr 14, 2022