Foveated imaging based on regions of interest
Systems and techniques are described herein for displaying images. For instance, a method for displaying images is provided. The method may include obtaining image data for display at a display of a head-mounted device (HMD); determining a content-based region of interest (ROI) within the image data based on content of the image data; determining a gaze-based ROI of the image data based on gaze information; displaying, at the display, content-based-ROI image data of the content-based ROI at a first resolution; displaying, at the display, gaze-based ROI image data of the gaze-based ROI at a second resolution; and displaying, at the display, peripheral image data outside the content-based ROI and outside the ROI at a third resolution.
1 . An apparatus for displaying images, the apparatus comprising:
at least one memory; and
at least one processor coupled to the at least one memory and configured to:
obtain image data for display at a display of a head-mounted device (HMD);
identify pixels of the image data that represent content in the image data;
in response to determining that one or more characteristics of the pixels satisfy a threshold, determine a content-based region of interest (ROI) within the image data based on the pixels;
determine a gaze-based ROI of the image data based on gaze information;
display, at the display, content-based-ROI image data of the content-based ROI at a first resolution;
display, at the display, gaze-based ROI image data of the gaze-based ROI at a second resolution; and
display, at the display, peripheral image data outside the content-based ROI and outside the ROI at a third resolution.
2 . The apparatus of claim 1 , wherein the at least one processor is configured to determine the gaze information based on an image of an eye of a user captured by a user-facing camera of the HMD.
3 . The apparatus of claim 1 , wherein the at least one processor is configured to determine the first resolution based on a distance between the content-based ROI and the gaze-based ROI.
4 . The apparatus of claim 1 , wherein the image data is based on an image captured by a scene-facing camera of the HMD.
5 . The apparatus of claim 1 , wherein the image data comprises virtual content to be displayed at the display of the HMD.
6 . The apparatus of claim 1 , wherein, to determine that the one or more characteristics of the pixels satisfy the threshold, the at least one processor is configured to at least one of:
compare at least one color of the pixels to a color threshold;
compare a brightness of the pixels to a brightness threshold;
compare a visual complexity of the pixels to a visual-complexity threshold; or
compare movement of the content between the image data and prior image data to a movement threshold.
7 . The apparatus of claim 1 , wherein the at least one processor is configured to:
determine a classification of the content based on the image data; and
in response to determining that the classification of the content satisfies a criterion, determine the content-based ROI within the image data based on the pixels;
wherein the classification of the content comprises at least one of:
an association between a user of the HMD and the content;
a label determined by a classifier; or
an assigned label.
8 . The apparatus of claim 1 , wherein the at least one processor is configured to determine the first resolution based on at least one of:
at least one color of the pixels;
a brightness of the pixels;
a visual complexity of the pixels;
movement of the content; or
a classification of the content.
9 . The apparatus of claim 1 , wherein the at least one processor is configured to determine a level of processing for the content-based ROI based on at least one of:
at least one color of the pixels;
a brightness of the pixels;
a visual complexity of the pixels;
movement of the content; or
a classification of the content.
10 . The apparatus of claim 1 , wherein the content comprises at least one of:
a person;
an object;
a virtual object; or
a virtual entity.
11 . The apparatus of claim 1 , wherein the at least one processor is configured to:
determine, based on a plurality of factors, that the content satisfies a criterion; and
responsive to determining that the content satisfies the criterion, determine the content-based ROI within the image data based on the content.
12 . The apparatus of claim 11 , wherein the at least one processor is configured to:
determine a respective weight value for each factor of the plurality of factors; and
determine a score for the plurality of factors based on the respective weight value determined for each factor of the plurality of factors;
wherein, to determine that the content satisfies the criterion, the at least one processor is configured to determine that the score for the plurality of factors is greater than a score threshold.
13 . An apparatus for displaying images, the apparatus comprising:
at least one memory; and
at least one processor coupled to the at least one memory and configured to:
obtain image data for display at a display of a head-mounted device (HMD), wherein the image data is based on at least one image captured by a scene-facing camera of the HMD;
identify pixels of the image data that represent content in the image data;
determine, based on a plurality of factors, that the content satisfies a criterion based on one or more characteristics of the pixels;
responsive to determining that the content satisfies the criterion, determine a content-based region of interest (ROI) within the image data based on the pixels;
display, at the display, ROI image data of the content-based ROI at a first resolution; and
display, at the display, peripheral image data outside the content-based ROI at a second resolution, wherein the second resolution is lower than the first resolution.
14 . The apparatus of claim 13 , wherein the plurality of factors comprise at least two of:
at least one color of the pixels;
a brightness of the pixels;
a visual complexity of the pixels; or
movement of the content between the image data and prior image data.
15 . The apparatus of claim 13 , wherein the at least one processor is configured to:
determine a classification of the content based on the image data,
in response to determining that the classification of the content satisfies a criterion, determine the content-based ROI within the image data based on the pixels;
wherein the classification of the content comprises at least one of:
an association between a user of the HMD and the content; or
a label determined by a classifier.
16 . The apparatus of claim 15 , wherein the content comprises a person and the classification of the content comprises a relationship between a user of the HMD and the person.
17 . The apparatus of claim 13 , wherein the at least one processor is configured to:
determine a respective weight value for each factor of the plurality of factors; and
determine a score for the plurality of factors based on the respective weight value determined for each factor of the plurality of factors;
wherein, to determine that the content satisfies the criterion, the at least one processor is configured to determine that the score for the plurality of factors is greater than a score threshold.
18 . The apparatus of claim 13 , wherein the content comprises at least one of:
a person; or
an object.
19 . The apparatus of claim 13 , wherein, to identify the content, the at least one processor is configured to at least one of:
detect a person in the image data; or
detect an object in the image data.
20 . A method for displaying images, the method comprising:
obtaining image data for display at a display of a head-mounted device (HMD);
identify pixels of the image data that represent content in the image data;
in response to determining that one or more characteristics of the pixels satisfy a threshold, determining a content-based region of interest (ROI) within the image data based on the pixels;
determining a gaze-based ROI of the image data based on gaze information;
displaying, at the display, content-based-ROI image data of the content-based ROI at a first resolution;
displaying, at the display, ROI image data of the ROI at a second resolution; and
displaying, at the display, peripheral image data outside the content-based ROI and outside the ROI at a third resolution.