Generating foveated image data
Systems and techniques are described herein for generating foveated image data. For instance, an apparatus for generating foveated image data is provided. The apparatus may include a memory; and a first processor communicatively connected directly to an image sensor and communicatively connected to the memory, the first processor configured to: determine an orientation of at least one eye of a user; store an indication of the orientation in the memory; and provide the indication of the orientation to the image sensor wherein the image sensor is configured to capture a foveated image based on the indication of the orientation, the foveated image comprising a first portion and a second portion, wherein the first portion is based on the indication of the orientation and has a first resolution, and wherein the second portion has a second resolution that is lower than the first resolution.
1 . An apparatus for generating foveated image data, the apparatus comprising:
a first processor communicatively connected to an eye-facing image sensor, a scene-facing image sensor and a memory, wherein the memory is accessible to a second processor, wherein the first processor is configured to:
generate eye-orientation data based on an image of at least one eye of a user captured by the eye-facing image sensor, wherein the eye-orientation data comprises a position of the at least one eye relative to a coordinate system;
store the eye-orientation data in the memory for processing by the second processor; and
provide the eye-orientation data directly to the scene-facing image sensor, wherein the scene-facing image sensor is configured to capture a foveated image based on a region of interest (ROI) associated with the eye-orientation data, the foveated image comprising a first portion and a second portion, wherein the first portion is based on the ROI and has a first resolution, and wherein the second portion has a second resolution that is lower than the first resolution.
2 . The apparatus of claim 1 , further comprising the second processor, wherein the second processor is configured to:
read the eye-orientation data from the memory; and
process the eye-orientation data.
3 . The apparatus of claim 2 , wherein, to process the the eye-orientation data, the second processor is configured to at least one of:
render image data based on the eye-orientation data;
adjust image data based on the eye-orientation data; or
display image data based on the eye-orientation data.
4 . The apparatus of claim 1 , wherein the first processor is configured to store the image of the at least one eye of the user in the memory; wherein the apparatus further comprises the second processor, wherein the second processor is configured to:
read the image of the at least one eye of the user from the memory; and
process the image of the at least one eye of the user.
5 . The apparatus of claim 4 , wherein, to process the image of the at least one eye of the user, the second processor is configured to at least one of:
render image data based on the image of the at least one eye of the user;
adjust image data based on the image of the at least one eye of the user; or
display image data based on the image of the at least one eye of the user.
6 . The apparatus of claim 1 , wherein the eye-facing image sensor is configured to:
capture images of the at least one eye;
determine an orientation of the at least one eye of the user;
provide the eye-orientation data to the scene-facing image sensor; and
store the eye-orientation data in the memory.
7 . The apparatus of claim 1 , wherein the first processor comprises: an image signal processor configured to:
process the image of the at least one eye to determine an orientation of the at least one eye of the user;
provide the eye-orientation data to the scene-facing image sensor; and
store the eye-orientation data in the memory.
8 . The apparatus of claim 1 , further comprising an interface directly communicatively connecting the first processor to the scene-facing image sensor, wherein the first processor is configured to provide the eye-orientation data to the scene-facing image sensor using the interface.
9 . The apparatus of claim 1 , wherein the first processor is configured to predict a gaze and determine an orientation of at least one eye based on the predicted gaze.
10 . The apparatus of claim 1 , wherein:
the scene-facing image sensor is configured to determine the ROI based on the eye-orientation data; and
the first portion of the foveated image corresponds to the region of interest ROI.
11 . The apparatus of claim 10 , wherein ROI comprises pixel coordinates defining the first portion of the foveated image.
12 . The apparatus of claim 1 , wherein the coordinate system is associated with the apparatus.
13 . A method for generating foveated image data, the method comprising:
capturing an image of at least one eye of a user at an eye-facing image sensor;
generating, by a first processor communicatively connected directly to a scene-facing image sensor, eye-orientation data based on the image of at least one eye of a user;
storing, by the first processor, the eye-orientation data in a memory for processing by a second processor, wherein the memory is accessible to the second processor, wherein the eye-orientation data comprises a position of the at least one eye relative to a coordinate system; and
providing, from the first processor, the eye-orientation data directly to the scene-facing image sensor, wherein the scene-facing image sensor is configured to capture a foveated image based on the orientation a region of interest (ROI) associated with the eye-orientation data, the foveated image comprising a first portion and a second portion, wherein the first portion is based on the ROI and has a first resolution, and wherein the second portion has a second resolution that is lower than the first resolution.
14 . The method of claim 13 , further comprising:
reading, by the second processor the eye-orientation data from the memory; and
processing, by the second processor, the eye-orientation data.
15 . The method of claim 14 , wherein processing the eye-orientation data comprises:
rendering image data based on the eye-orientation data;
adjusting image data based on the eye-orientation data; or displaying image data based on the eye-orientation data.
16 . The method of claim 13 , further comprising:
storing, by the first processor, the image of the at least one eye of the user in the memory;
reading, by the second processor, the image of the at least one eye of the user from the memory; and
processing, by the second processor, the image of the at least one eye of the user.
17 . The method of claim 16 , wherein processing the image of the at least one eye of the user comprises:
rendering image data based on the image of the at least one eye of the user;
adjusting image data based on the image of the at least one eye of the user; or displaying image data based on the image of the at least one eye of the user.
18 . The method of claim 13 , wherein the first processor comprises the eye-facing image sensor, and wherein the eye-facing image sensor is configured to:
capture the image of the at least one eye;
determine an orientation of the at least one eye of the user;
provide the eye-orientation data to the scene-facing image sensor; and
store the eye-orientation data in the memory.
19 . The method of claim 13 , wherein the first processor comprises:
the eye-facing image sensor, wherein the eye-facing image sensor is configured to capture images of the at least one eye; and
an image signal processor configured to:
process the images of the at least one eye to determine an orientation of the at least one eye of the user;
provide the eye-orientation data to the scene-facing image sensor; and
store the eye-orientation data in the memory.
20 . The method of claim 13 , wherein the coordinate system is associated with the eye-facing image sensor.