Granted Patent B2

US 12,190,016 · App. 18/301,890 · Granted Jan 7, 2025

Inventor: Ziqiang Huang (Parkland, FL)

Assignee: Magic Leap, Inc.

G06F3/165A63F13/00A63F13/424A63F13/5255A63F13/5372A63F13/54A63F13/65G02B27/0093G02B27/0172G06F3/04815G06F3/04842G06T19/006H04R1/32H04R1/403G02B2027/0138G02B2027/014G06F3/012G06F3/017G06F3/167G06T2200/24H04R1/1075H04R3/005H04R5/02H04S2400/11

Abstract

Claims (62)

1. A method of operating a virtual image generation system by an end user, comprising:

persistently selecting at least one object from a field of view of the end user that includes both virtual and real objects;

generating video data originating from the at least one selected object;

rendering a plurality of image frames in a three-dimensional scene from the generated video data;

displaying the image frames to the end user;

generating audio data originating from the at least one selected object; and

storing the audio data originating from the at least one selected object in synchronization with stored video data, within memory.

2. The method of claim 1 , further comprising transforming the audio data originating from the at least one selected object into sound for perception by the end user.

3. The method of claim 1 , wherein the persistently selecting the at least one object comprises moving a three-dimensional cursor in the field of view of the end user and selecting the at least one object with the three-dimensional cursor.

4. The method of claim 1 , wherein said persistently selecting at least one object comprises issuing one or more voice commands.

5. The method of claim 1 , wherein said persistently selecting at least one object comprises persistently selecting at least a real object.

6. The method of claim 1 , wherein the at least one object comprises a plurality of objects, and persistently selecting the plurality of objects comprises individually selecting the objects.

7. The method of claim 1 , wherein the at least one selected object comprises a virtual object and the method further comprising:

storing content data corresponding to sounds for a plurality of virtual objects;

acquiring the content data corresponding to the selected virtual object, wherein the audio data stored in the memory comprises the acquired content data; and

generating meta data corresponding to the selected virtual object, wherein the audio data stored in the memory comprises the acquired content data and the generated meta data.

8. The method of claim 7 , wherein the meta data includes position, orientation and volume data for the selected virtual object.

9. The method of claim 7 , further comprising:

tracking a head pose of the end user; and

storing absolute meta data for the plurality of virtual objects;

wherein generating the meta data comprises retrieving the absolute meta data corresponding to the selected virtual object, and localizing the absolute meta data to the end user based on the tracked head pose of the end user.

10. The method of claim 1 , further comprising retrieving the stored audio data, deriving audio from the retrieved audio data, and transforming the audio into sound for perception by the end user.

11. The method of claim 1 , wherein the stored audio data comprises content data and meta data, the method further comprising:

retrieving the stored content data and meta data from the memory;

rendering spatialized audio based on the retrieved content data and meta data; and

transforming the spatialized audio into sound for perception by the end user.

12. The method of claim 1 , wherein the persistently selecting the at least one object comprises the end user forming a circle or partial circle around the object using at least two fingers.

13. The method of claim 1 , further comprising storing audio content data and video content data originating from the at least one selected object when the at least one selected object includes a real object and a virtual object in an original spatial environment;

retrieving the audio content data and video content data; and

rendering audio and video from the retrieved audio content data and video content data in a new spatial environment different from the original spatial environment.

14. The method of claim 1 , wherein said rendering comprises a video processor including a graphics processing unit (GPU) acquiring the video data of the virtual objects from a three-dimensional database and rendering synthetic video frames therefrom, and further comprising a central processing unit (CPU) acquiring video frames of said real objects from a forward-facing camera.

15. The method of claim 1 , further comprising providing the field of view that includes both virtual and real objects by providing a partially transparent display screen through which real objects in the ambient environment are seen by the end user and onto which images of virtual objects are displayed.

16. A method of operating a virtual image generation system by an end user, comprising:

persistently selecting at least one object from a field of view of the end user that includes both virtual and real objects;

generating video data originating from the at least one selected object;

rendering a plurality of image frames in a three-dimensional scene from the generated video data;

displaying the image frames to the end user;

generating audio data originating from the at least one selected object; and

storing the audio data originating from the at least one selected object within memory, wherein the at least one object comprises a plurality of objects, and selecting the plurality of objects comprises globally selecting the objects, and

wherein globally selecting the objects comprises defining an angular range of the field of view of the end user, and selecting all of the objects in the defined angular range of the field of view of the end user.

17. The method of claim 1 , further comprising persistently deselecting the at least one previously selected object.

18. A method of operating a virtual image generation system by an end user, comprising:

persistently selecting at least one object from a field of view of the end user that includes both virtual and real objects;

generating video data originating from the at least one selected object;

rendering a plurality of image frames in a three-dimensional scene from the generated video data;

displaying the image frames to the end user;

generating audio data originating from the at least one selected object; and

storing the audio data originating from the at least one selected object within memory; and tracking a location of the at least one selected object relative to the field of view of the end user.

19. The method of claim 18 , further comprising ceasing to store the audio data originating from the at least one selected object in the memory when the tracked location of the at least one selected object moves out of the field of view of the end user.

20. The method of claim 18 , further comprising continuing to store the audio data originating from the at least one selected object in the memory when the tracked location of the at least one selected object moves out of the field of view of the end user.

21. A method of operating a virtual image generation system by an end user, comprising:

persistently selecting at least one object from a field of view of the end user that includes both virtual and real objects;

generating video data originating from the at least one selected object;

rendering a plurality of image frames in a three-dimensional scene from the generated video data;

displaying the image frames to the end user;

generating audio data originating from the at least one selected object;

storing the audio data originating from the at least one selected object within memory, wherein the at least one selected object comprises a real object; and

preferentially sensing sounds originating from the selected real object relative to sounds originating from other real objects, wherein the audio data is derived from the preferentially sensed sounds.

22. The method of claim 21 , further comprising:

capturing video data originating from the selected real object; and

storing the video data in synchronization with the audio data in the memory.

23. The method of claim 22 , further comprising transforming the captured video data into virtual content data, and storing the virtual content data in the memory.

Assignments (4)

Continuity (5)

References Cited (235)