Variable resolution virtual reality display system
A virtual reality display system that renders images at different resolutions in different parts of a display. Reduces rendering latency by rendering at a lower resolution in selected regions, for example on the sides of a display where human vision has lower resolution than in the center. Pixels in low resolution regions are combined into grid elements, and rendering may generate grid element values rather than individual pixel values. Rendering may use ray casting, rasterization, or both. Variable resolution rendering may be combined with variable level of detail geometry models to further reduce rendering time. Selected objects may be designed as high resolution objects that are rendered at a high resolution even in low resolution display regions.
1. A variable resolution virtual reality display system comprising
at least one display viewable by a user and comprising a plurality of display pixels, wherein said plurality of display pixels is partitioned into at least two display regions;
wherein each display region of said at least two display regions is partitioned into a grid of grid elements, each grid element of said grid of grid elements comprising one or more display pixels; and,
at least two of said at least two display regions have different ratios of display pixels per grid element;
a 3D model of a scene;
a scene renderer coupled to said at least one display, to said at least two display regions, and to said 3D model, wherein said scene renderer
calculates one or more 2D projections of said 3D model onto said grid of grid elements of each of said at least two display regions, to form a grid element value for each grid element; and,
for each grid element, assigns a pixel value to said one or more display pixels associated with said grid element based on said grid element value of said grid element;
at least one sensor that generates sensor data that measures one or more aspects of a pose of one or more body parts of said user;
a pose analyzer coupled to said at least one sensor, that calculates pose data of said pose of one or more body parts of said user, based on said sensor data generated by said at least one sensor;
wherein said scene renderer is coupled to said pose analyzer, and said scene renderer receives said pose data from said pose analyzer;
calculates one or more rendering virtual camera poses, based on said pose data; and,
calculates said one or more 2D projections of said 3D model onto said grid of grid elements of each of said at least two display regions based on said one or more rendering virtual camera poses; and,
an image warper coupled to said at least one display, said scene renderer, and said pose analyzer, wherein said image warper
receives said one or more rendering virtual camera poses from said scene renderer;
receives said pose data from said pose analyzer;
calculates a change in pose between said one or more virtual camera poses and said pose data;
generates a rerendering approximation of said one or more 2D projections on said at least one display based on said change in pose;
modifies one or more pixels of said at least one display based on said rerendering approximation;
compares said change in pose to a threshold value; and,
generates said rerendering approximation if said change in pose is below said threshold value.
2. The system of claim 1 , wherein
said grid element value comprises one or more of a color and an intensity;
said pixel value comprises one or more of a color and an intensity; and,
said assigns a pixel value to said one or more display pixels associated with said grid element comprises
copy said grid element value to said pixel value of each of said one or more display pixels associated with said grid element.
3. The system of claim 1 , wherein
said at least two display regions comprise
a high resolution center region located in a center of a field of view of said user;
a low resolution left region located in a left portion of said field of view; and,
a low resolution right region located in a right portion of said field of view;
a ratio of display pixels per grid element for said low resolution left region is greater than said ratio of display pixels per grid element for said high resolution center region; and,
a ratio of display pixels per grid element for said low resolution right region is greater than said ratio of display pixels per grid element for said high resolution center region.
4. The system of claim 3 , wherein
said ratio of display pixels per grid element for said high resolution center region is 1:1.
5. The system of claim 3 , wherein
said ratio of display pixels per grid element for said low resolution left region is at least 4:1; and,
said ratio of display pixels per grid element for said low resolution right region equals said ratio of display pixels per grid element for said low resolution left region.
6. The system of claim 1 , wherein
at least one object in said 3D model comprises at least two geometry models at different levels of detail;
for each display region of said at least two display regions, said scene renderer selects a level of detail for said at least one object; and,
calculates a 2D projection of said at least one object onto said grid of grid elements using a geometry model for said level of detail associated with said at least one object; and,
said scene renderer selects at least two different levels of detail for said at least one object for at least two different display regions.
7. The system of claim 1 , wherein said calculates said one or more 2D projections of said 3D model onto said grid of grid elements for at least one of said at least two display regions comprises
uses ray casting to project a ray from a point within each grid element to a first intersected object in said 3D model; and,
sets said grid element value for each grid element based on one or more characteristics of said first intersected object in said 3D model at an intersection between said ray and said first intersected object.
8. The system of claim 1 , wherein said calculates said one or more 2D projections of said 3D model onto said grid of grid elements comprises projects one or more 3D geometric primitives from said 3D model onto an image plane corresponding to a display region, to form one or more projected primitives;
rasterizes said one or more projected primitives to form one or more grid element fragments, each associated with
a grid element; and,
one or more display pixels within said grid element; and,
blends said one or more grid element fragments associated with each grid element to form said grid element value for said grid element.
9. The system of claim 8 , wherein
said 3D model comprises at least one high resolution object and at least one low resolution object;
each fragment formed from projection and rasterization of said at least one high resolution object is associated with a single display pixel; and,
each fragment formed from projection and rasterization of said at least one low resolution object is associated with all display pixels within said grid element.
10. The system of claim 1 , wherein said rerendering approximation comprises
calculating a pixel translation vector; and,
translating one or more pixels of said one or more 2D projections by said pixel translation vector.
11. The system of claim 10 , wherein calculating said pixel translation vector comprises
approximating said change in pose as a rotation around a unit vector {circumflex over (ω)} comprising {circumflex over (ω)} y and {circumflex over (ω)} x by an angle Δθ;
calculating a spatial translation vector ({circumflex over (ω)} y Δθ, −{circumflex over (ω)} x Δθ);
calculating a scaling factor to convert spatial distances to pixels based on pixel dimensions and fields of view of said one or more 2D projections; and,
calculating said pixel translation vector by scaling said spatial translation vector by said scaling factor.
12. A variable resolution virtual reality display system comprising
at least one display viewable by a user and comprising a plurality of display pixels, wherein said plurality of display pixels is partitioned into at least two display regions;
wherein each display region of said at least two display regions is partitioned into a grid of grid elements, each grid element of said grid of grid elements comprising one or more display pixels; and,
at least two of said at least two display regions have different ratios of display pixels per grid element;
a 3D model of a scene;
a scene renderer coupled to said at least one display, to said at least two display regions, and to said 3D model, wherein said scene renderer
calculates one or more 2D projections of said 3D model onto said grid of grid elements of each of said at least two display regions, to form a grid element value for each grid element; and,
for each grid element, assigns a pixel value to said one or more display pixels associated with said grid element based on said grid element value of said grid element;
at least one sensor that generates sensor data that measures one or more aspects of a pose of one or more body parts of said user;
a pose analyzer coupled to said at least one sensor, that calculates pose data of said pose of one or more body parts of said user, based on said sensor data generated by said at least one sensor;
wherein said scene renderer is coupled to said pose analyzer, and said scene renderer receives said pose data from said pose analyzer;
calculates one or more rendering virtual camera poses, based on said pose data; and,
calculates said one or more 2D projections of said 3D model onto said grid of grid elements of each of said at least two display regions based on said one or more rendering virtual camera poses; and,
an image warper coupled to said at least one display, said scene renderer, and said pose analyzer, wherein said image warper
receives said one or more rendering virtual camera poses from said scene renderer;
receives said pose data from said pose analyzer;
calculates a change in pose between said one or more virtual camera poses and said pose data;
generates a rerendering approximation of said one or more 2D projections on said at least one display based on said change in pose;
modifies one or more pixels of said at least one display based on said rerendering approximation;
wherein said rerendering approximation comprises
calculating a pixel translation vector; and,
translating one or more pixels of said one or more 2D projections by said pixel translation vector; and,
wherein calculating said pixel translation vector comprises
approximating said change in pose as a rotation around a unit vector) comprising {circumflex over (ω)} y and {circumflex over (ω)} x by an angle Δθ;
calculating a spatial translation vector ({circumflex over (ω)} y Δθ, −{circumflex over (ω)} x Δθ);
calculating a scaling factor to convert spatial distances to pixels based on pixel dimensions and fields of view of said one or more 2D projections; and,
calculating said pixel translation vector by scaling said spatial translation vector by said scaling factor.
13. The system of claim 12 , wherein
said grid element value comprises one or more of a color and an intensity;
said pixel value comprises one or more of a color and an intensity; and,
said assigns a pixel value to said one or more display pixels associated with said grid element comprises
copy said grid element value to said pixel value of each of said one or more display pixels associated with said grid element.
14. The system of claim 12 , wherein
said at least two display regions comprise
a high resolution center region located in a center of a field of view of said user;
a low resolution left region located in a left portion of said field of view; and,
a low resolution right region located in a right portion of said field of view;
a ratio of display pixels per grid element for said low resolution left region is greater than said ratio of display pixels per grid element for said high resolution center region; and,
a ratio of display pixels per grid element for said low resolution right region is greater than said ratio of display pixels per grid element for said high resolution center region.
15. The system of claim 14 , wherein
said ratio of display pixels per grid element for said high resolution center region is 1:1.
16. The system of claim 14 , wherein
said ratio of display pixels per grid element for said low resolution left region is at least 4:1; and,
said ratio of display pixels per grid element for said low resolution right region equals said ratio of display pixels per grid element for said low resolution left region.
17. The system of claim 12 , wherein
at least one object in said 3D model comprises at least two geometry models at different levels of detail;
for each display region of said at least two display regions, said scene renderer
selects a level of detail for said at least one object; and,
calculates a 2D projection of said at least one object onto said grid of grid elements using a geometry model for said level of detail associated with said at least one object; and,
said scene renderer selects at least two different levels of detail for said at least one object for at least two different display regions.
18. The system of claim 12 , wherein said calculates one or more 2D projections of said 3D model onto said grid of grid elements for at least one of said at least two display regions comprises
uses ray casting to project a ray from a point within each grid element to a first intersected object in said 3D model; and,
sets said grid element value for each grid element based on one or more characteristics of said first intersected object in said 3D model at an intersection between said ray and said first intersected object.
19. The system of claim 12 , wherein said calculates one or more 2D projections of said 3D model onto said grid of grid elements comprises
projects one or more 3D geometric primitives from said 3D model onto an image plane corresponding to a display region, to form one or more projected primitives;
rasterizes said one or more projected primitives to form one or more grid element fragments, each associated with
a grid element; and,
one or more display pixels within said grid element; and,
blends said one or more grid element fragments associated with each grid element to form said grid element value for said grid element.
20. The system of claim 19 , wherein
said 3D model comprises at least one high resolution object and at least one low resolution object;
each fragment formed from projection and rasterization of said at least one high resolution object is associated with a single display pixel; and,
each fragment formed from projection and rasterization of said at least one low resolution object is associated with all display pixels within said grid element.
21. The system of claim 12 , wherein said image warper
compares said change in pose to a threshold value; and,
generates said rerendering approximation if said change in pose is below said threshold value.