Continuous time warp and binocular time warp for virtual and augmented reality display systems and methods
View Patent ↗Embodiments of the present disclosure relate to continuous and/or binocular time warping methods to account for head movement of the user without having to re-render a displayed image. Continuous time warping allows for transformation of an image from a first perspective to a second perspective of the viewer without having to re-render the image from the second perspective. Binocular time warp refers to the late-frame time warp used in connection with a display device including a left display unit for the left eye and a right display unit for the right eye where the late-frame time warp is performed separately for the left display unit and the right display unit. Warped images are sent to the left and the right display units where photons are generated and emitted toward respective eyes of the viewer, thereby displaying an image on the left and the right display units at the same time.
1. A method for time warping an image frame based on an updated position of a viewer, the method comprising performing, using two or more graphics processing units including a first graphics processing unit and a second graphics processing unit separate and distinct from the first graphics processing unit, steps of:
rendering, at a first time, a first image frame for a first display of a binocular near-eye display device, wherein the first image frame is associated with a first position of the viewer;
rendering a second image frame for a second display of the binocular near-eye display device, wherein the first image frame is one of a right image frame for a right display of the binocular near-eye display device or a left image frame for a left display of the binocular near-eye display device, and the second image frame is other one of the right image frame or the left image frame;
receiving, at a second time later than the first time, data associated with a second position of the viewer;
time warping, by the first graphics processing unit, at least a portion of the first image frame using the data associated with the second position of the viewer;
generating an updated first image frame for the first display of the binocular near-eye display device based on time warped portion of the first image frame;
receiving, at a third time later than the second time, data associated with a third position of the viewer;
time warping, by the second graphics processing unit, at least a portion of the second image frame using the data associated with the third position of the viewer;
generating an updated second image frame for the second display of the binocular near-eye display device based on time warped portion of the second image frame, wherein the updated second image frame is generated separately from the updated first image frame;
transmitting, by the first graphics processing unit, the updated first image frame for display by the first display; and
transmitting, by the second graphics processing unit, the updated second image frame for display by the second display.
2. The method of claim 1 , wherein the first image frame and the second image frame are time warped substantially at a same time.
3. The method of claim 1 , further comprising:
displaying the updated first image frame on the first display of the binocular near-eye display device, and
displaying the updated second image frame on the second display of the binocular near-eye display device,
wherein the updated first image frame and the updated second image frame are displayed substantially at a same time.
4. The method of claim 1 , further comprising:
displaying the updated first image frame on the first display of the binocular near-eye display device, and
displaying the updated second image frame on the second display of the binocular near-eye display device,
wherein the updated first image frame and the updated second image frame are displayed sequentially.
5. The method of claim 1 , further comprising:
interpolating the data associated with the third position of the viewer into the second image frame to generate the updated second image frame.
6. The method of claim 1 , wherein the updated first image frame is transmitted to the first display of the binocular near-eye display device before the updated second image frame is transmitted to the second display of the binocular near-eye display device.
7. The method of claim 1 , further comprising:
displaying the updated first image frame on the first display of the binocular near-eye display device, and
displaying the updated second image frame on the second display of the binocular near-eye display device,
wherein the updated first image frame is displayed before the updated second image frame.
8. The method of claim 1 , wherein the first image frame and the second image frame are rendered staggered in time such that the first image frame is rendered before the second image frame.
9. The method of claim 1 , wherein the first image frame and the second image frame are time warped sequentially.
10. A system comprising:
a first graphics processing unit comprising:
a first processor; and
a first non-transitory computer-readable medium coupled to the first processor, the first non-transitory computer-readable medium storing instructions that, when executed by the first processor, cause the first processor to:
render, at a first time, a first image frame for a first display of a binocular near-eye display device, wherein the first image frame is associated with a first position of a viewer,
receive, at a second time later than the first time, data associated with a second position of the viewer,
time warp at least a portion of the first image frame using the data associated with the second position of the viewer;
generate an updated first image frame for the first display of the binocular near-eye display device based on time warped portion of the first image frame, and
transmit the updated first image frame for display by the first display;
a second graphics processing unit separate and distinct from the first graphics processing unit, the second graphics processing unit comprising:
a second processor; and
a second non-transitory computer-readable medium coupled to the second processor, the second non-transitory computer-readable medium storing instructions that, when executed by the second processor, cause the second processor to:
render, at a third time later than the first time, a second image frame for a second display of the binocular near-eye display device, wherein the first image frame is one of a right image frame for a right display of the binocular near-eye display device or a left image frame for a left display of the binocular near-eye display device, and the second image frame is other one of the right image frame or the left image frame,
receive, at a fourth time later than the second time, data associated with a third position of the viewer,
time warp at least a portion of the second image frame using the data associated with the third position of the viewer,
generate an updated second image frame for the second display of the binocular near-eye display device based on time warped portion of the second image frame, and
transmit the updated second image frame for display by the second display.
11. The system of claim 10 , wherein the first image frame and the second image frame are time warped substantially at a same time.
12. The system of claim 10 , wherein the second non-transitory computer-readable medium further stores instructions that, when executed by the second processor, cause the second processor to:
interpolate the data associated with the third position of the viewer into the second image frame to generate the updated second image frame.
13. The system of claim 10 , further comprising:
the first display of the binocular near-eye display device configured to display the updated first image frame, and
the second display of the binocular near-eye display device configured to display the updated second image frame,
wherein the updated first image frame is displayed before the updated second image frame.
14. The system of claim 10 , wherein the first image frame and the second image frame are rendered staggered in time such that the first image frame is rendered before the second image frame.
15. The system of claim 10 , wherein the first image frame and the second image frame are time warped sequentially.