Frame extrapolation with application generated motion vector and depth

1. A method comprising, by a computing system associated with a wearable device:

receiving a rendered image, motion vector data, and a depth map corresponding to a current frame of a video stream generated by an application;

processing the received motion vector data and the depth map such that regions corresponding to foreground of the rendered image are expanded;

calculating, for an object presented in the rendered image, a current three-dimensional position of the object corresponding to the current frame using the depth map;

calculating a past three-dimensional position of the object corresponding to a past frame using the motion vector data and the depth map;

estimating a future three-dimensional position of the object corresponding to a future frame based on the past three-dimensional position and the current three-dimensional position of the object; and

generating an extrapolated image corresponding to the future frame by reprojecting the object presented in the rendered image to a future viewpoint associated with the future frame using the future three-dimensional position of the object.

2. The method of claim 1 , wherein the motion vector data and the depth map are generated based on three-dimensional objects rendered by the application.

3. The method of claim 2 , wherein motion vectors in the motion vector data are three-dimensional.

4. The method of claim 1 , wherein calculating the current three-dimensional position of the object comprises inversely projecting the depth map onto a three-dimensional space from a current viewpoint associated with the current frame.

5. The method of claim 4 , wherein the current viewpoint is associated with a position and an orientation of the wearable device at a time instance when the current frame is rendered.

6. The method of claim 1 , wherein calculating the past three-dimensional position of the object comprises:

generating an estimated depth map corresponding to the past frame by subtracting the motion vectors from the depth map; and

inversely projecting the estimated depth map onto a three-dimensional space from a past viewpoint associated with the past frame.

7. The method of claim 1 , wherein estimating the future three-dimensional position of the object is performed based on an assumption that the object moves at a constant speed from a time instance corresponding to the past frame to a time instance corresponding to the future frame.

8. The method of claim 7 , further comprising generating a distortion mesh by projecting the estimated future three-dimensional position of the object onto the future viewpoint.

9. The method of claim 8 , wherein generating the extrapolated image corresponding to the future frame comprises applying the distortion mesh to the rendered image.

10. One or more computer-readable non-transitory storage media embodying software that is operable when executed, by a computing system associated with a wearable device, to:

receive a rendered image, motion vector data, and a depth map corresponding to a current frame of a video stream generated by an application;

process the received motion vector data and the depth map such that regions corresponding to foreground of the rendered image are expanded;

calculate, for an object presented in the rendered image, a current three-dimensional position of the object corresponding to the current frame using the depth map;

calculate a past three-dimensional position of the object corresponding to a past frame using the motion vector data and the depth map;

estimate a future three-dimensional position of the object corresponding to a future frame based on the past three-dimensional position and the current three-dimensional position of the object; and

generate an extrapolated image corresponding to the future frame by reprojecting the object presented in the rendered image to a future viewpoint associated with the future frame using the future three-dimensional position of the object.

11. The media of claim 10 , wherein the motion vector data and the depth map are generated based on three-dimensional objects rendered by the application.

12. The media of claim 11 , wherein motion vectors in the motion vector data are three-dimensional.

13. The media of claim 10 , wherein calculating the current three-dimensional position of the object comprises inversely projecting the depth map onto a three-dimensional space from a current viewpoint associated with the current frame.

14. The media of claim 13 , wherein the current viewpoint is associated with a position and an orientation of the wearable device at a time instance when the current frame is rendered.

15. The media of claim 10 , wherein calculating the past three-dimensional position of the object comprises:

generating an estimated depth map corresponding to the past frame by subtracting the motion vectors from the depth map; and

inversely projecting the estimated depth map onto a three-dimensional space from a past viewpoint associated with the past frame.

16. The media of claim 10 , wherein estimating the future three-dimensional position of the object is performed based on an assumption that the object moves at a constant speed from a time instance corresponding to the past frame to a time instance corresponding to the future frame.

17. The media of claim 16 , wherein the software is further operable when executed to: generate a distortion mesh by projecting the estimated future three-dimensional position of the object onto the future viewpoint.

18. A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:

receive a rendered image, motion vector data, and a depth map corresponding to a current frame of a video stream generated by an application;

process the received motion vector data and the depth map such that regions corresponding to foreground of the rendered image are expanded;

calculate, for an object presented in the rendered image, a current three-dimensional position of the object corresponding to the current frame using the depth map;

calculate a past three-dimensional position of the object corresponding to a past frame using the motion vector data and the depth map;

estimate a future three-dimensional position of the object corresponding to a future frame based on the past three-dimensional position and the current three-dimensional position of the object; and

generate an extrapolated image corresponding to the future frame by reprojecting the object presented in the rendered image to a future viewpoint associated with the future frame using the future three-dimensional position of the object.