Optical engine time warp for augmented or mixed reality environment
A device, system, and method for applying time warp in an augmented or mixed reality environment. An image may be rendered corresponding to an estimated head position calculated based on head position and orientation measured at the initial time. Time warp may be independently applied to each sequential one of the plurality of segmented visual units of an image (e.g., color-frames or pixel-groups) to generate a plurality of respective time warp visual units corresponding to a plurality of different respective estimated head position and orientations calculated based on head position and orientations measured at a plurality of different respective sequential times after the initial time. The plurality of time warp visual units may be projected in the sequential order at the augmented or mixed reality device.
1. A method of applying time warp in an augmented or mixed reality environment, the method comprising:
receiving head position and orientation data from sensors operably connected to an augmented or mixed reality device for measuring head position and orientation at an initial time;
rendering an image corresponding to an estimated head position calculated based on the head position and orientation measured at the initial time;
independently applying time warp to each sequential one of a plurality of segmented visual units of an image to generate a plurality of respective time warp visual units corresponding to a plurality of different respective estimated head position and orientations calculated based on head position and orientations measured at a plurality of different respective sequential times after the initial time; and
projecting the plurality of time warp visual units in the sequential order at the augmented or mixed reality device.
2. The method of claim 1 , wherein the plurality of segmented visual units are a plurality of monochromatic color-frames of the rendered image, each monochromatic color-frame depicting a single color or color spectrum portion of the rendered image.
3. The method of claim 1 , wherein the plurality of segmented visual units are a plurality of pixel-groups of the rendered image, each pixel-group depicting a different spatial region of the rendered image.
4. The method of claim 1 comprising applying time warp in multiple phases including:
in an initial phase, storing the rendered image comprising the plurality of segmented visual units; and
in a subsequent phase, independently applying time warp to each sequential one of the plurality of segmented visual units.
5. The method of claim 1 comprising:
in an active buffer, outputting the plurality of segmented visual units to be time warped for projecting in a current image iteration; and
in a standby buffer, inputting a new rendered image comprising the plurality of visual units to be projected in a subsequent image iteration.
6. The method of claim 5 comprising, after the completion of an image iteration, switching the roles of buffers between active and standby to output for projection the plurality of time warp visual units loaded in the previous image iteration and to input a new plurality of visual units of a new image.
7. The method of claim 1 , wherein the image is rendered with an additional border area outside of the field of view corresponding to the head position and orientation at the initial time to fill in image data in a region initially outside the display field of view in the rendered image and revealed by applying time warp.
8. The method of claim 7 , wherein the size of the additional border is based on a maximum allowed head position and orientation change from the initial time to one or more of the sequential times or an average thereof.
9. The method of claim 7 , wherein the size of the additional border is based on a head speed and a time between the initial time and a final one of the sequential times at which the visual units are projected.
10. The method of claim 1 , wherein the image is rendered to have a higher resolution than a resolution of a screen onto which the plurality of time warp visual units are projected.
11. The method of claim 1 , wherein the head position and orientation for each time warp visual unit is estimated based on the associated sequential measurement time that occurs a predetermined time warp calculation duration prior to the projection time for the time warp visual unit.
12. The method of claim 1 , wherein head position and orientation is measured using sensors selected from the group consisting of: cameras, depth sensors, magnetometers, gyroscopes, accelerometers, and inertial measurement units (IMUs).
13. A system for applying time warp in an augmented or mixed reality environment, the system comprising:
one or more processors configured to:
receive head position and orientation data from sensors operably connected to an augmented or mixed reality device for measuring head position and orientation at an initial time,
render an image corresponding to an estimated head position calculated based on the head position and orientation measured at the initial time, and
independently apply time warp to each sequential one of a plurality of segmented visual units of an image to generate a plurality of respective time warp visual units corresponding to a plurality of different respective estimated head position and orientations calculated based on head position and orientations measured at a plurality of different respective sequential times after the initial time; and
a projector to display the plurality of time warp visual units in the sequential order at the augmented or mixed reality device.
14. The system of claim 13 comprising two or more buffers of which at least one buffer is designated as active for outputting the plurality of segmented visual units to be time warped for projecting in a current image iteration and at least one other buffer is designated as standby for inputting a new rendered image comprising the plurality of visual units to be projected in a subsequent image iteration.
15. The system of claim 14 , wherein after the completion of an image iteration, the one or more processors are configured to switch the designation of buffer roles between active and standby to output for projection the plurality of time warp visual units loaded in the previous image iteration and to input a new plurality of visual units of a new image.
16. The system of claim 13 comprising one or more sensors selected from the group consisting of: cameras, depth sensors, magnetometers, gyroscopes, accelerometers, and inertial measurement units (IMUs).
17. The system of claim 13 , wherein the one or more processors are configured to apply time warp in multiple phases including:
in an initial phase, storing the rendered image comprising the plurality of segmented visual units, and
in a subsequent phase, independently applying time warp to each sequential one of the plurality of segmented visual units.
18. The system of claim 13 , wherein the plurality of segmented visual units are a plurality of monochromatic color-frames of the rendered image, each monochromatic color-frame depicting a single color or color spectrum portion of the rendered image.
19. The system of claim 13 , wherein the plurality of segmented visual units are a plurality of pixel-groups of the rendered image, each pixel-group depicting a different spatial region of the rendered image.
20. The system of claim 13 , wherein the augmented or mixed reality device is selected from the group consisting of: glasses, headsets, contact lenses, holograms, screens, prisms, televisions, windows, kiosks, and projectors.