Augmented reality system with spatialized audio tied to user manipulated virtual object
View Patent ↗An augmented reality system includes left and right transparent eyepieces through which a user can view the real world and which serve to couple imagewise modulated light into the user's eyes in order to display virtual content components of a mixed reality experience. The system further includes left and right speakers and a facility for user spatial manipulation of virtual objects. The system produces spatialized audio that has a virtual sound source position fixed to one or more virtual objects that are spatially manipulated by the user. Thus the system provides more realistic visual and auditory presentation of virtual components of a mixed reality experience.
1. An augmented reality system comprising: a handheld controller; a wearable headgear unit including: a left eye display comprising a transmissive eyepiece; a right eye display comprising a transmissive eyepiece; a left side speaker; and a right side speaker; first tracking components; and circuitry coupled to the left eye display, the right eye display, the left side speaker, the right side speaker, and the first tracking components wherein the circuitry is configured to: identify, via the first tracking components, a position of the handheld controller relative to the wearable headgear unit; output, based on the identified position, one or more signals to the left eye display and the right eye display to display, via one or more of the transmissive eye piece of the left eye display and the transmissive eyepiece of the right eye display, a view of a virtual object and a view of the handheld controller, the view of the virtual object substantially fixed relative to the view of the handheld controller, wherein the virtual object includes at least one rotational degree of freedom; and output spatialized audio that includes a left channel audio signal that is output to the left side speaker and a right channel audio signal that is output to the right side speaker, wherein the spatialized audio is associated with a virtual sound source location that is substantially fixed relative to the handheld controller.
2. The augmented reality system according to claim 1 , wherein the circuitry is further configured to receive one or more signals indicative of an orientation of the handheld controller relative to the wearable headgear unit via the first tracking components.
3. The augmented reality system according to claim 1 , wherein the circuitry is configured to generate the left channel audio signal and the right channel audio signal by applying at least one head related transfer function to a mono audio signal.
4. The augmented reality system according to claim 1 , further comprising: second tracking components coupled to the circuitry, wherein the circuitry is further configured to receive, via the second tracking components, one or more signals indicative of a position and orientation of the wearable headgear unit relative to a real world environment.
5. The augmented reality system according to claim 1 , wherein the handheld controller is visible through the left eye display and right eye display.
6. The augmented reality system according to claim 1 , wherein the wearable headgear unit further comprises one or more of the first tracking components.
7. The augmented reality system according to claim 1 , wherein the handheld controller further comprises one or more of the first tracking components.
8. The augmented reality system according to claim 1 , wherein the virtual object comprises at least one coordinate fixed relative to the handheld controller.
9. An augmented reality system comprising: a wearable headgear unit including: a left eye display comprising a transmissive eyepiece; a right eye display comprising a transmissive eyepiece; a left side speaker; and a right side speaker; a hand tracking machine vision subsystem; and circuitry coupled to the left eye display, the right eye display, the left side speaker, the right side speaker, and the hand tracking machine vision subsystem, wherein the circuitry is configured to:
identify, via the hand tracking machine vision subsystem, a position of a hand of a user relative to the wearable headgear unit;
output, based on the identified position, one or more signals to the left eye display and the right eye display to display, via one or more of the transmissive eyepiece of the left eye display and the transmissive eyepiece of the right eye display, a view of a virtual object and a view of the hand, the view of the virtual object fixed relative to the view of the hand, wherein the virtual object includes at least one rotational degree of freedom; and
output spatialized audio that includes a left channel audio signal that is output to the left side speaker and a right channel audio signal that is output to the right side speaker, wherein the spatialized audio is associated with a virtual sound source location that is substantially fixed relative to the hand.
10. The augmented reality system according to claim 9 , wherein the circuitry is further configured to identify, via the hand tracking machine vision subsystem, a gesture of the hand and to adjust at least one kinematic parameter of the virtual object in response to identifying the gesture.
11. The augmented reality system according to claim 9 , wherein the circuitry is configured to generate the left channel audio signal and the right channel audio signal by applying at least one head related transfer function to a mono audio signal.
12. The augmented reality system according to claim 9 , wherein the wearable headgear unit further comprises one or more components of the hand tracking machine vision subsystem.
13. The augmented reality system according to claim 9 , further comprising a handheld controller, wherein the handheld controller further comprises one or more components of the hand tracking machine vision subsystem.
14. The augmented reality system according to claim 9 , wherein the virtual object comprises at least one coordinate fixed relative to the view of the hand.
15. An augmented reality system comprising: a wearable headgear unit including: a left eye display comprising a transmissive eyepiece; a right eye display comprising a transmissive eyepiece; a left side speaker; and a right side speaker; one or more receivers; and
circuitry coupled to the left eye display, the right eye display, the left side speaker, the right side speaker, and the one or more receivers, wherein the circuitry is configured to:
output signals to the left eye display and the right eye display to display, via one or more of the transmissive eyepiece of the left eye display and the transmissive eyepiece of the right eye display, a view of a virtual object, wherein the virtual object includes at least one rotational degree of freedom;
output spatialized audio that includes a left channel audio signal that is output to the left side speaker and a right channel audio signal that is output to the right side speaker, wherein the spatialized audio is associated with a virtual sound source location that is substantially fixed relative to the virtual object; and receive user input through the one or more receivers and in response to receiving the user input through the one or more receivers, change a location of the virtual object and change the at least one virtual sound source location.
16. The augmented reality system according to claim 15 , wherein the one or more receivers comprise a hand tracking machine vision subsystem.
17. The augmented reality system according to claim 16 , wherein the one or more receivers comprise a handheld controller.
18. The augmented reality system according to claim 15 , wherein the wearable headgear unit further comprises the one or more receivers.
19. The augmented reality system according to claim 15 , further comprising a handheld controller, wherein the handheld controller comprises the one or more receivers.
20. The augmented reality system according to claim 15 , further comprising a handheld controller, wherein the virtual object comprises at least one coordinate fixed relative to the handheld controller.