Mixed reality recording of foveated display content systems and methods
A system may include a display for displaying an image frame that is divided into regions having respective resolutions based on display image data. The system may also include image processing circuitry to generate the display image data based on multi-resolution image data and generate record image data based on the multi-resolution image data. Generating the record image data may include obtaining boundary data indicative of locations of boundaries between the regions resampling the multi-resolution image data based on the boundary data. Resampling the multi-resolution image data may include performing a first resampling on a first portion of the multi-resolution image data corresponding to a first region and performing a second resampling, different from the first resampling, on a second portion of the multi-resolution image data corresponding to a second region.
1 . A system comprising:
an electronic display comprising a plurality of pixels and configured to display an image frame, the image frame having a plurality of resolutions based on display image data, wherein the image frame is divided into a plurality of regions having respective resolutions of the plurality of resolutions; and
image processing circuitry configured to:
generate the display image data based on multi-resolution image data of the image frame; and
generate record image data of a consistent resolution based on the multi-resolution image data of the image frame, wherein the record image data is different from the display image data, and wherein generating the record image data comprises:
determining boundary data indicative of locations of boundaries between the plurality of regions; and
resampling the multi-resolution image data of the display image data to the consistent resolution based on the boundary data, wherein resampling the multi-resolution image data comprises performing a first resampling on a first portion of the multi-resolution image data corresponding to a first region of the plurality of regions by a first factor and performing a second resampling, different from the first resampling, on a second portion of the multi-resolution image data corresponding to a second region of the plurality of regions by a second factor.
2 . The system of claim 1 , wherein generating the record image data comprises cropping the multi-resolution image data to a crop window, and wherein the crop window comprises less than an entirety of the image frame.
3 . The system of claim 2 , wherein the crop window is centered about a focal point of a user eye gaze on the electronic display, wherein the locations of the boundaries between the plurality of regions are adjustable and based on the focal point.
4 . The system of claim 2 , wherein the crop window excludes a third portion of the multi-resolution image data corresponding to a third region of the plurality of regions having a respective resolution, of the plurality of resolutions, less than a threshold resolution.
5 . The system of claim 1 , wherein performing the first resampling comprises resampling the first portion of the multi-resolution image data by the first factor in a first direction and downsampling the first portion of the multi-resolution image data by the second factor in a second direction, and wherein performing the second resampling comprises resampling the second portion of the multi-resolution image data by the first factor in the first direction and not downsampling the second portion of the multi-resolution image data by the second factor in the second direction.
6 . The system of claim 5 , wherein performing the first resampling comprises upsampling the first portion of the multi-resolution image data by the first factor in the first direction.
7 . The system of claim 1 , wherein the consistent resolution is the same as a respective resolution of the plurality of resolutions of a third region of the plurality of regions.
8 . The system of claim 1 , wherein generating the record image data comprises transforming a color space of the multi-resolution image data.
9 . The system of claim 8 , wherein generating the record image data comprises converting the color space of the multi-resolution image data to a chromatic color space.
10 . The system of claim 1 , wherein generating the record image data comprises gamma encoding the multi-resolution image data.
11 . Image processing circuitry comprising:
processing block circuitry configured to generate display image data based on an image frame of multi-resolution image data, wherein the image frame is divided into a plurality of regions having respective resolutions of a plurality of resolutions; and
record block circuitry configured to generate record image data having a consistent resolution based on the image frame of the multi-resolution image data based on:
determining a cropped region of the image frame of the multi-resolution image data corresponding to the record image data, wherein the cropped region excludes a portion of the multi-resolution image data corresponding to a region, of the plurality of regions, having less than a threshold resolution; and
resampling the multi-resolution image data of the cropped region to the consistent resolution to obtain the record image data, wherein the record image data comprises the resampled cropped region.
12 . The image processing circuitry of claim 11 , wherein resampling the multi-resolution image data comprises resampling the multi-resolution image data by performing a first resampling on a second portion of the multi-resolution image data corresponding to a second region of the plurality of regions and performing a second resampling, different from the first resampling, on a third portion of the multi-resolution image data corresponding to a third region of the plurality of regions.
13 . The image processing circuitry of claim 11 , wherein the consistent resolution comprises a lowest resolution of the cropped region.
14 . The image processing circuitry of claim 11 , wherein the record block circuitry is configured to generate the record image data by reducing a precision of the multi-resolution image data.
15 . The image processing circuitry of claim 14 , wherein the record block circuitry is configured to generate the record image data by downsampling and interleaving a first chromatic channel of the multi-resolution image data and a second chromatic channel of the multi-resolution image data and maintaining a luma channel of the multi-resolution image data.
16 . A non-transitory machine-readable medium comprising instructions, wherein, when executed by one or more processors, the instructions cause the one or more processors to perform operations or to control image processing circuitry to perform the operations, wherein the operations comprise:
determining boundary data indicative of locations of boundaries between a plurality of regions, wherein the plurality of regions define areas of different content resolutions of an image frame of multi-resolution image data;
generating record image data that excludes all regions of the image frame less than a threshold resolution; and
storing the record image data in a non-transitory memory, transmitting the record image data to an electronic display, or both.
17 . The non-transitory machine-readable medium of claim 16 , wherein the operations comprise generating display image data, based on the multi-resolution image data, to be displayed on a second electronic display, wherein the second electronic display comprises a foveated display, and wherein the locations of the boundaries of the plurality of regions are set based on a focal point of a viewer's gaze on the electronic display.
18 . The non-transitory machine-readable medium of claim 16 , wherein generating the record image data is based on resampling remaining regions of the image frame, wherein the resampling comprises resampling a first region of the plurality of regions by a first factor and resampling a second region of the plurality of regions by a second factor to a consistent resolution across a portion of the image frame that excludes all regions of the image frame less than the threshold resolution based on the boundary data.
19 . The non-transitory machine-readable medium of claim 18 , wherein resampling the multi-resolution image data comprises performing a first resampling on a first portion of the multi-resolution image data corresponding to the first region of the plurality of regions and performing a second resampling, different from the first resampling, on a second portion of the multi-resolution image data corresponding to the second region of the plurality of regions.
20 . The non-transitory machine-readable medium of claim 19 , wherein the first resampling comprises a downsampling in a first direction and no resampling in a second direction.