Methods and systems for reprojection in augmented-reality displays
Methods and systems are provided for a reprojection engine for augmented-reality devices. The augmented-reality device projects virtual content within a real-world environment. The augmented-reality device tracks a six degrees of freedom headpose of the augmented-reality device, depth information of the virtual content, motion vectors that correspond to movement of the virtual content, and a color buffer for a reprojection engine. The reprojection engine generates a reprojection of the virtual content defined by an extrapolation of a first frame using the headpose, the depth information, motion vectors, and the color surface data structure. The reprojected virtual content continues to appear as if positioned with the real-world environment regardless of changes in the headpose of the augmented-reality device or motion of the virtual content.
1. A method for projection in an augmented-reality device, the method comprising:
receiving a first headpose at a first time, the first time coinciding with rendering of a first frame on a partially transparent eyepiece displaying the first frame and portions of a real-world environment;
determining a second headpose using one or more sensors of the augmented-reality device, the second headpose corresponding to a second time coinciding with a second frame to be rendered, the second frame to include virtual content that is projected onto the real-world environment;
defining a headpose difference using the first headpose and the second headpose;
determining depth information for the virtual content;
receiving, from an augmented-reality application executing on the augmented-reality device, one or more motion vectors associated with the virtual content;
receiving a color surface data structure that represents color values of the virtual content;
generating a reprojected color surface for the virtual content that includes an updated color surface data structure defined by an extrapolation of the first frame using the headpose difference, the depth information, the one or more motion vectors, and the color surface data structure; and
displaying the virtual content that includes the updated color surface data structure on the augmented-reality device.
2. The method of claim 1 wherein the depth information includes a depth buffer, and wherein the depth buffer is generated by:
receiving, for each pixel of a set of pixels, a first depth value from the augmented-reality application;
aggregating the first depth value of each pixel of the set of pixels into the depth buffer;
deriving a second depth value for each pixel of the set of pixels using stereo disparity between two versions of a same frame;
aggregating the second depth value of each pixel of the set of pixels into a temporary depth buffer;
determining, for each pixel of the set of pixels, whether the second depth value is smaller than the first depth value; and
replacing, for each pixel of the set of pixels, the first depth value with the second depth value in response to determining that the second depth value is smaller than the first depth value.
3. The method of claim 1 wherein the depth information includes an augmented depth buffer generated by:
initializing the augmented depth buffer using a predetermined value for each pixel of a set of pixels;
calculating, for each pixel of the set of pixels, an approximate depth value using stereo disparity between two versions of a same frame; and
selecting, from the predetermined value and the approximate depth value, a value for each pixel of the set of pixels that is smaller.
4. The method of claim 1 wherein the one or more motion vectors comprises one or more two-dimensional motion vectors.
5. The method of claim 1 wherein the depth information comprises a stabilization depth value.
6. The method of claim 1 wherein the one or more motion vectors comprise one or more three-dimensional motion vectors.
7. The method of claim 1 wherein the headpose difference is represented using six-degrees of freedom.
8. The method of claim 1 wherein the depth information comprises a depth value for each pixel of the virtual content, wherein the depth value indicates an approximate depth of each pixel of the virtual content with respect to a real-world environment.
9. The method of claim 1 wherein the reprojected color surface is generated for each of two or more color layers.
10. The method of claim 1 wherein the depth information includes a depth buffer, and wherein the depth buffer is generated by:
receiving, for each pixel of a set of pixels, a first depth value from the augmented-reality application;
deriving a second depth value for each pixel of the set of pixels using stereo disparity between two versions of a same frame;
determining, for each pixel of the set of pixels, whether the second depth value is smaller than the first depth value; and
replacing, for each pixel of the set of pixels, the first depth value with the second depth value in response to determining that the second depth value is smaller than the first depth value.