IP Library Granted Patent US 9,674,628
Granted Patent B1
US 9,674,628 · App. 15/441,199 · Granted Jun 6, 2017

Providing binaural sound to localize at an image during a telephone call

Inventors: Glen A. Norris (Tokyo, JP); Philip Scott Lyren (Hong Kong, CN)
H04S1/002H04S7/303H04S7/305H04S2400/11H04S2420/01
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,674,628
App. No.
15/441,199
Granted
Jun 6, 2017
Kind
B1
Abstract

A method executed by one or more electronic devices in an electronic system provides binaural sound of a voice of a person to localize at an image during a telephone call. During the telephone call, a wearable electronic device displays an image of a calling party, and a voice of the calling party externally localizes as the binaural sound to the image of the calling party.

Claims (59)

1. A method executed by one or more electronic devices in an electronic system that provides a voice of a caller in binaural sound to a wearer of a wearable electronic device during a telephone call between the caller and the wearer, the method comprising:

displaying, to the wearer and with the wearable electronic device during the telephone call, an image that represents the caller at a location in a room where the wearer is located;

convolving, during the telephone call and with a processor, the voice of the caller with head related transfer functions (HRTFs) so the voice of the caller localizes to the wearer as the binaural sound at a sound localization point (SLP) that is located in empty space at the image that represents the caller at the location in the room where the wearer is located;

tracking, during the telephone call and with the wearable electronic device, a location and a head orientation of the wearer as the wearer moves from the room to another room and away from the image that represents the caller; and

changing, during the telephone call and while the wearer moves from the room to the another room, the HRTFs convolving the voice of the caller so the voice of the caller continues to localize to the wearer at the SLP at the location in the room even while the wearer is located in the another room.

2. The method of claim 1 , wherein the HRTFs change in real-time during the telephone call as the wearer moves from the room to the another room so the voice of the caller remains fixed at the SLP that is located in empty space at the image that represents the caller.

3. The method of claim 1 , wherein the image that represents the caller moves in real-time with movements of the caller during the telephone call.

4. The method of claim 1 , wherein the wearable electronic device is a head mounted display, and the room and the another room are virtual rooms that the wearer sees during the telephone call with the caller.

5. The method of claim 1 further comprising:

reducing, during the telephone call and by the processor, the voice of the caller that the wearer hears as the wearer moves from the room to the another room according to the following equation:

SPL(receiving room)=SPL(source room)−STL−10 log(SA/A),

wherein the SPL (receiving room) is a sound pressure level in a receiving room, SPL (source room) is a sound pressure level in a source room, STL is a sound transmission loss of a wall between the receiving room and the source room, SA is a sound absorption in the receiving room, and A is a surface area of the wall between the source room and the receiving room, and

wherein the room and the another room are virtual rooms and calculations for the reducing the voice of the caller are for the virtual rooms.

6. The method of claim 1 further comprising:

tracking, during the telephone call, head movements of the caller;

moving, during the telephone call and in response to the head movements of the caller, a head of the image that represents the caller such that the head of the image copies the head movements of the caller; and

changing, during the telephone call and during the head movements of the caller, the HRTFs convolving the voice of the caller so the voice of the caller continues to localize to the wearer at the SLP that is the image that represents the caller.

7. The method of claim 1 further comprising:

tracking, during the telephone call, rotation of a head of the caller twenty degrees (20°) azimuth;

rotating, during the telephone call and as seen through the wearable electronic device, a head of the image that represents the caller by twenty degrees (20°) azimuth to emulate the rotation of the head of the caller; and

adjusting, in order to compensate for the rotation of the head of the caller by twenty degrees (20°) azimuth, convolution of the voice of the caller so a direction of the voice of the caller changes to originate from the head of the image that was rotated by twenty degrees (20°) azimuth.

8. The method of claim 1 further comprising:

capturing, during the telephone call and with the wearable electronic device, a voice of the wearer;

adjusting the voice of the wearer to correspond to the voice of the wearer being captured at a virtual microphone point (VMP) that is located at a SLP of the caller; and

providing, during the telephone call, the voice of the wearer adjusted by the VMP to the caller.

9. A method executed by one or more electronic devices in an electronic system that improves an experience of a calling party and a called party talking to each other during a telephone call, the method comprising:

processing, during the telephone call and with a processor, a voice of the calling party with head related transfer functions (HRTFs) so the voice of the calling party localizes to the called party as binaural sound at a sound localization point (SLP) that is one meter to two meters away from the called party;

displaying, during the telephone call and to the called party with a wearable electronic device worn on a head of the called party, an image that represents the calling party that is seen by the called party as being at the SLP that is one meter to two meters away from the called party;

tracking, during the telephone call and with the wearable electronic device, head movements of the called party with respect to the image that represents the calling party and that is seen by the called party as being at the SLP; and

changing, during the telephone call and in response to the head movements of the called party, the HRTFs processing the voice of the calling party so the voice of the calling party continues to localize to the SLP that is seen by the called party as being the image that represents the calling party that is one meter to two meters away from the called party while the head of the called party moves with respect to the image that represents the calling party.

10. The method of claim 9 further comprising:

tracking, during the telephone call and with a wearable electronic device worn on a head of the calling party, head movements of the calling party;

moving, during the telephone call, the image that represents the calling party to copy the head movements of the calling party so the called party sees, with the display of the wearable electronic device worn on the head of the called party, the head movements of the calling party; and

changing, during the telephone call and in response to the head movements of the calling party, the HRTFs processing the voice of the calling party so the voice of the calling party continues to localize to the image that represents the calling party that is seen by the called party as being at the SLP that is one meter to two meters away from the called party while the image that represents the calling party moves.

11. The method of claim 9 further comprising:

tracking, during the telephone call and with a wearable electronic device worn on a head of the calling party, head movements of the calling party; and

changing, during the telephone call and in response to the head movements of the calling party, the voice of the calling party so the voice of the calling party emulates a sound that the called party would hear if the calling party were physically present at the SLP and talking to the called party.

12. The method of claim 9 further comprising:

displaying, during the telephone call and to the called party with the wearable electronic device worn on the head of the called party, the image that represents the calling party as a virtual image of the calling party.

13. The method of claim 9 further comprising:

displaying, during the telephone call and to the called party with the wearable electronic device worn on the head of the called party, the image that represents the calling party as an augmented reality image of the calling party.

14. The method of claim 9 further comprising:

moving, during the telephone call, the image that represents the calling party around a room where the called party is located; and

changing, during the telephone call and in response to movements in the room of the image that represents the calling party, the HRTFs processing the voice of the calling party so the voice of the calling party continues to localize to the image that represents the calling party even as the image that represents the calling party moves around the room.

15. The method of claim 9 , wherein the wearable electronic device is a smartphone that is attached to the called party.

16. A non-transitory computer readable storage medium storing instructions that one or more electronic devices in an electronic system execute as a method during a telephone call between a calling party and a called party, the method comprising:

displaying, during the telephone call and to the called party with a wearable electronic device worn on a head of the called party, an image of the calling party that appears to the called party as being at least one meter away from a head of the called party;

processing, during the telephone call, a voice of the calling party with head related transfer functions (HRTFs) so the voice of the calling party localizes to the called party as binaural sound at the image of the calling party that appears at least one meter away from the head of the called party;

tracking, during the telephone call, head movements of the called party; and

changing, during the telephone call and in response to the head movements of the called party, the HRTFs processing the voice of the calling party so the voice of the calling party continues to localize to the image of the calling party that appears at least one meter away from the head of the called party as the head of the called party moves and changes head orientations.

17. The non-transitory computer readable storage medium storing instructions of claim 16 further to cause the one or more electronic devices in the electronic system to execute the method comprising:

tracking, during the telephone call, head movements of the calling party; and

changing, during the telephone call and in response to the head movements of the calling party, the HRTFs processing the voice of the calling party so the voice of the calling party continues to localize to the image of the calling party that appears at least one meter away from the head of the called party as a head of the calling party moves.

18. The non-transitory computer readable storage medium storing instructions of claim 16 , wherein a location of the image of the calling party also functions as a virtual microphone point (VMP) for sound that is captured by the wearable electronic device worn on the head of the called party, and the sound captured by the wearable electronic device worn on the head of the called party is provided to the calling party.

19. The non-transitory computer readable storage medium storing instructions of claim 16 further to cause the one or more electronic devices in the electronic system to execute the method comprising:

changing, during the telephone call, the HRTFs processing the voice of the calling party so the voice of the calling party moves and follows the image of the calling party as the image of the calling party moves with respect to the called party.

20. The non-transitory computer readable storage medium storing instructions of claim 16 further to cause the one or more electronic devices in the electronic system to execute the method comprising:

providing the image of the calling party as being a virtual microphone point; and

reducing, during the telephone call, an amplitude of a voice of the called party that is provided to the calling party when the called party moves away from the virtual microphone point.

Assignments (2)
SECURITY INTEREST Recorded Apr 7, 2022
From: EIGHT KHZ, LLC
To: LIT-US CHISUM 22-A LLC
Reel/Frame 059536/0708 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 19, 2021
From: LYREN, PHILIP S; NORRIS, GLEN A
To: EIGHT KHZ, LLC
Reel/Frame 057839/0225 →
Continuity (7)
Continuation 15430549 · Feb 13, 2017
Continuation 15367186 · Dec 2, 2016
Continuation 15259072 · Sep 8, 2016
Continuation 15096352 · Apr 12, 2016
Continuation 15011563 · Jan 31, 2016
Continuation 14946806 · Nov 20, 2015
Continuation 14311532 · Jun 23, 2014