System and methods for performing electronic display stabilization via retained lightfield rendering

1. A method comprising:

maintaining a lightfield data structure representing at least a portion of a four-dimensional (4D) lightfield for a three-dimensional (3D) world in association with a first pose of a user-portable display device relative to the 3D world, by:

determining a set of potential poses based on the first pose, the set of potential poses representing at least some of the poses that the user-portable display device is capable of taking within a specified time span by:

determining a probability distribution of potential poses for the specified time span; and

determining the set of potential poses from the probability distribution; and

for each potential pose of the set of potential poses, rendering a lightfield frame representing a field of view of the 3D world from the potential pose and incorporating the lightfield frame into the lightfield data structure;

determining a second pose of the user-portable display device relative to the 3D world based on the set of potential poses, the second pose comprising an updated pose of the user-portable display device;

generating a display frame from the lightfield data structure based on the second pose, the display frame representing a field of view of the 3D world from the second pose; and

displaying the generated display frame at the user-portable display device.

2. The method of claim 1 , wherein: rendering wherein rendering the lightfield frame comprises:

rendering the lightfield frame at a first rate; and

rendering the display frame comprises rendering the display frame at a second rate greater than the first rate.

3. The method of claim 1 , wherein:

the user-portable display device comprises a head mounted display; and

determining the probability distribution of potential poses comprises determining the probability distribution of potential poses further based on potential poses of a user's eyes for the specified time span.

4. The method of claim 1 , wherein:

generating the display frame from the lightfield data structure based on the second pose comprises generating the display frame based at least in part on ray-tracing-based interpolation from at least one rendered lightfield frame represented in the lightfield data structure.

5. The method of claim 1 , wherein:

the user-portable display device comprises a head mounted display having a left-eye display and a right-eye display; and

generating the display frame from the lightfield data structure based on the second pose comprises:

generating a first display frame for display on the left-eye display based on the second pose and an orientation of a left eye of the user relative to the second pose; and

generating a second display frame for display on the right-eye display based on the second pose and an orientation of a right eye of the user relative to the second pose.

6. A user-portable display device comprising:

a display;

a storage component;

an inertial sensor;

an application processor coupled to the inertial sensor and the storage component, the application processor configured to:

maintain, in the storage component, a lightfield data structure representing at least a portion of a four-dimensional (4D) lightfield of a three-dimensional (3D) world in association with one or more poses of the user-portable display device relative to the 3D world;

determine a set of potential poses, the set of potential poses representing at least some of the poses that the user-portable display device is capable of taking within a specified time span by:

determining a probability distribution of potential poses for the specified time span; and

determining the set of potential poses from the probability distribution; and

for each potential pose of the set of poses, generate a lightfield frame representing a field of view of the 3D world from the respective potential pose and incorporating the lightfield frame into the lightfield data structure; and

a compositor coupled to the storage component and the inertial sensor, the compositor configured to:

determine a current pose of the user-portable display device relative to the 3D world based on output from the inertial sensor, the current pose comprising an updated pose of the user-portable display device; and

generate a display frame from the lightfield data structure based on the current pose, the display frame representing a field of view of the 3D world from the current pose, wherein the user-portable display device is configured to display the generated display frame.

7. The user-portable display device of claim 6 , wherein the application processor is further configured to:

generate lightfield frames at a first rate; and

render display frames at a second rate different than the first rate.

8. The user-portable display device of claim 6 , wherein:

the user-portable display device comprises a head mounted display; and

the application processor is further configured to determine the probability distribution of potential poses further based on potential poses of a user's eyes for the specified time span.

9. The user-portable display device of claim 6 , wherein:

the application processor is further configured to generate the display frame based at least in part on ray-tracing-based interpolation from at least one rendered lightfield frame represented in the lightfield data structure.

10. The user-portable display device of claim 6 , wherein:

the user-portable display device comprises a head mounted display having a left-eye display and a right-eye display; and

the application processor is further configured to:

generate a first display frame for display on the left-eye display based on the current pose and an orientation of a left eye of the user relative to the current pose; and

generate a second display frame for display on the right-eye display based on the current pose and an orientation of a right eye of the user relative to the current pose.

11. A head mounted display (HMD) system comprising:

a left-eye display and a right-eye display;

an inertial sensor; and

a processing system coupled to the left-eye display, right-eye display, and the inertial sensor, the processing system to:

determine a probability distribution of potential future poses of the HMD system based on a current pose of the HMD system;

within a specified time span, select a set of potential poses from the probability distribution;

render a set of lightfield frames based on the set of potential poses, each lightfield frame representing a view of a 3D world from a corresponding potential pose of the set;

determine an updated pose of the HMD system;

generate a first display frame for display at the left-eye display based on the set of lightfield frames and the updated pose; and

generate a second display frame for display at the right-eye display based on the set of lightfield frames and the updated pose.

12. The HMD system of claim 11 , wherein:

the processing system is to generate the first display frame further based on an orientation of a left eye of a user; and

the processing system is to generate the second display frame further based on an orientation of a right eye of the user.