Granted Patent B2

US 12,271,995 · App. 18/048,602 · Granted Apr 8, 2025

Inventors: Steven Paul Lansel (Apopka, FL); Guodong Rong (Saratoga, CA); Jian Zhang (Fremont, CA)

Assignee: META PLATFORMS TECHNOLOGIES, LLC

G06T15/503G06F3/012G06T1/60G06T3/18G06T5/80G06T15/20G06T19/006G06T19/20G06T2219/2016

Abstract

Claims (81)

1. A method comprising, by a computing system of an artificial reality device:

obtaining, from an application, (a) an image including an object at a first position and (b) a layer frame having a first pose in front of the image, wherein the image and the layer frame are distinct;

generating, for a first viewpoint associated with a first time, a first display frame by separately:

rendering the image including the object at the first position into a display buffer; and

rendering the layer frame having the first pose into the display buffer;

displaying, at an artificial reality lens, the first display frame at the first time;

determining an extrapolated pose for the layer frame based on the first pose of the layer frame and a second pose of a previously submitted layer frame;

determining an extrapolated position of the object in the image based on a motion vector of the object, wherein the motion vector of the object is based on a comparison between one or more pixels of the object at the first position in the image and one or more corresponding pixels of the object at a previous position in a previously submitted image;

warping the layer frame having the extrapolated pose into a warped layer frame having the extrapolated pose and warping the image including the object at the extrapolated position into a warped image including the object at the extrapolated position to account for geometric distortion caused by the artificial reality lens;

after warping the layer frame and warping the image to account for geometric distortion caused by the artificial reality lens, generating, for a second viewpoint associated with a second time, a second display frame by separately:

rendering the warped image including the object at the extrapolated position into the display buffer; and

rendering the warped layer frame having the extrapolated pose into the display buffer; and

displaying, at the artificial reality lens, the second display frame at the second time.

2. The method of claim 1 , further comprising:

applying, for the first viewpoint associated with the first time, Asynchronous TimeWarp (ATW) to the image including the object at the first position to adjust for head rotation before rendering the image including the object at the first position into the display buffer; and

warping, for the first viewpoint associated with the first time, the image including the object at the first position into a warped image including the object at the first position to account for geometric distortion caused by the artificial reality lens before rendering the warped image including the object at the first position into the display buffer.

3. The method of claim 1 , further comprising:

applying, for the first viewpoint associated with the first time, ATW to the layer frame having the first pose to adjust for head rotation before rendering the layer frame having the first pose into the display buffer; and

warping, for the first viewpoint associated with the first time, the layer frame having the first pose into a warped layer frame having the first pose to account for geometric distortion caused by the artificial reality lens before rendering the warped layer frame having the first pose into the display buffer.

4. The method of claim 1 , wherein the extrapolated pose of the layer frame is determined using a linear extrapolation algorithm based on the second time, the first pose and the first time of the layer frame, and the second pose and a corresponding time of a previously submitted layer frame.

5. The method of claim 1 , further comprising:

applying, for the second viewpoint associated with the second time, ATW to the image including the object at the extrapolated position to adjust for head rotation before warping the image including the object at the extrapolated position.

6. The method of claim 1 , wherein determining the extrapolated position of the object in the image based on the motion vector of the object includes moving the object in the image linearly according to the motion vector to account for object movements.

7. The method of claim 1 , further comprising:

applying, for the second viewpoint associated with the second time, ATW to the layer frame having the extrapolated pose to adjust for head rotation before warping the layer frame having the extrapolated pose.

8. The method of claim 1 , wherein the first time is rendered at half of a desired frame rate and the second time is an extrapolated in-between time frame.

9. The method of claim 1 , wherein:

the first and second display frames are displayed on a quad layer, and

the first and second display frames are displayed on a cylindrical layer.

10. The method of claim 1 , wherein:

the extrapolated pose is determined by a position and an orientation of a first space in a second space,

the position determines the translation between the first and second spaces, and

the orientation determines the rotation between the first and second spaces.

11. One or more computer-readable non-transitory storage media including executable instructions that, when executed by one or more processors, cause the one or more processors to:

obtain, from an application, (a) an image including an object at a first position and (b) a layer frame having a first pose in front of the image, wherein the image and the layer frame are distinct;

cause generation of, for a first viewpoint associated with a first time, a first display frame by separately:

rendering the image including the object at the first position into a display buffer; and

rendering the layer frame having the first pose into the display buffer;

cause display, at an artificial reality lens, of the first display frame at the first time;

determine an extrapolated pose for the layer frame based on the first pose of the layer frame and a second pose of a previously submitted layer frame;

determine an extrapolated position of the object in the image based on a motion vector of the object, wherein the motion vector of the object is based on a comparison between one or more pixels of the object at the first position in the image and one or more corresponding pixels of the object at a previous position in a previously submitted image;

cause warping of the layer frame having the extrapolated pose into a warped layer frame having the extrapolated pose and warping of the image including the object at the extrapolated position into a warped image including the object at the extrapolated position to account for geometric distortion caused by the artificial reality lens;

after warping the layer frame and warping the image to account for geometric distortion caused by the artificial reality lens, cause generation of, for a second viewpoint associated with a second time, a second display frame by separately:

rendering the warped image including the object at the extrapolated position into the display buffer; and

rendering the warped layer frame having the extrapolated pose into the display buffer; and

cause display, at the artificial reality lens, of the second display frame at the second time.

12. The one or more computer-readable non-transitory storage media of claim 11 , wherein the executable instructions that further cause the one or more processors to:

cause application of, for the first viewpoint associated with the first time, Asynchronous TimeWarp (ATW) to the image including the object at the first position to adjust for head rotation before rendering the image including the object at the first position into the display buffer; and

cause warping of, for the first viewpoint associated with the first time, the image including the object at the first position into a warped image including the object at the first position to account for geometric distortion caused by the artificial reality lens before rendering the warped image including the object at the first position into the display buffer.

13. The one or more computer-readable non-transitory storage media of claim 11 , wherein the executable instructions that further cause the one or more processors to:

cause application of, for the first viewpoint associated with the first time, ATW to the layer frame having the first pose to adjust for head rotation before rendering the layer frame having the first pose into the display buffer; and

cause warping of, for the first viewpoint associated with the first time, the layer frame having the first pose into a warped layer frame having the first pose to account for geometric distortion caused by the artificial reality lens before rendering the warped layer frame having the first pose into the display buffer.

14. The one or more computer-readable non-transitory storage media of claim 11 , wherein the extrapolated pose of the layer frame is determined using a linear extrapolation algorithm based on the second time, the first pose and the first time of the layer frame, and the second pose and corresponding time of a previously submitted layer frame.

15. The one or more computer-readable non-transitory storage media of claim 11 , wherein the executable instructions that further cause the one or more processors to:

cause application of, for the second viewpoint associated with the second time, ATW to the image including the object at the extrapolated position to adjust for head rotation before warping the image including the object at the extrapolated position.

16. The one or more computer-readable non-transitory storage media of claim 11 , wherein determining the extrapolated position of the object in the image based on the motion vector of the object includes moving the object in the image linearly according to the motion vector to account for object movements.

17. The one or more computer-readable non-transitory storage media of claim 11 , wherein the executable instructions that further cause the one or more processors to:

cause application of, for the second viewpoint associated with the second time, ATW to the layer frame having the extrapolated pose to adjust for head rotation before warping the layer frame having the extrapolated pose.

18. The one or more computer-readable non-transitory storage media of claim 11 , wherein:

the first time is rendered at half of a desired frame rate and the second time is an extrapolated in-between time frame,

the first and second display frames are displayed on a quad layer, and

the first and second display frames are displayed on a cylindrical layer.

19. The one or more computer-readable non-transitory storage media of claim 11 , wherein:

the extrapolated pose is determined by a position and an orientation of a first space in a second space,

the position determines the translation between the first and second spaces, and

the orientation determines the rotation between the first and second spaces.

20. A system comprising:

one or more processors; and

one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to:

obtain, from an application, (a) an image including an object at a first position and (b) a layer frame having a first pose in front of the image, wherein the image and the layer frame are distinct;

generate, for a first viewpoint associated with a first time, a first display frame by separately:

rendering the image including the object at the first position into a display buffer; and

rendering the layer frame having the first pose into the display buffer;

display, at an artificial reality lens, the first display frame at the first time;

determine an extrapolated pose for the layer frame based on the first pose of the layer frame and a second pose of a previously submitted layer frame;

determine an extrapolated position of the object in the image based on a motion vector of the object, wherein the motion vector of the object is based on a comparison between one or more pixels of the object at the first position in the image and one or more corresponding pixels of the object at a previous position in a previously submitted image;

warp the layer frame having the extrapolated pose into a warped layer frame having the extrapolated pose and warp the image including the object at the extrapolated position into a warped image including the object at the extrapolated position to account for geometric distortion caused by the artificial reality lens;

after warping the layer frame and warping the image to account for geometric distortion caused by the artificial reality lens, generate, for a second viewpoint associated with a second time, a second display frame by separately:

rendering the warped image including the object at the extrapolated position into the display buffer; and

rendering the warped layer frame having the extrapolated pose into the display buffer; and

display, at the artificial reality lens, the second display frame at the second time.

Assignments (1)

Continuity (2)

References Cited (17)