Event-based camera with high-resolution frame output
A high-resolution image sensor suitable for use in an augmented reality (AR) system to provide low latency image analysis with low power consumption. The AR system can be compact, and may be small enough to be packaged within a wearable device such as a set of goggles or mounted on a frame resembling ordinary eyeglasses. The image sensor may receive information about a region of an imaging array associated with a movable object, selectively output imaging information for that region, and synchronously output high-resolution image frames. The region may be updated dynamically as the image sensor and/or the object moves. The image sensor may output the high-resolution image frames less frequently than the region being updated when the image sensor and/or the object moves. Such an image sensor provides a small amount of data from which object information used in rendering an AR scene can be developed.
1. An image sensor comprising an image array, an input configured to receive signals specifying a selected region of the imaging array, at least one output at which signals representative of at least the selected region of the imaging array are presented, and a second output at which signals representative of changes in a detected image in the selected region of the imaging array are presented, the image sensor comprising:
a plurality of pixel cells comprising the imaging array, each pixel cell of the plurality of pixel cells comprising a light-sensitive component;
at least one intensity readout circuit operatively coupled to the light-sensitive components of the plurality of pixel cells and configured to provide signals indicating sensed light intensity at at least a portion of the light-sensitive components of the plurality of pixel cells at an output of the at least one output of the image sensor;
at least one event detection circuit operatively coupled to the light-sensitive components of the plurality of pixel cells and configured to provide signals indicating a change in sensed light at at least a portion of the light-sensitive components of the plurality of pixel cells; and
an enable circuit, operatively coupled to the input specifying the selected region of the imaging array and operatively coupled to the at least one event detection circuit, to control a selected portion of the pixel cells for which a signal indicating a change in sensed light is coupled to an output of the at least one output of the image sensor.
2. The image sensor of claim 1 , wherein:
the imaging array has a resolution higher than 1 megapixel.
3. The image sensor of claim 1 , wherein:
the selected region of the imaging array constitutes no more than 20% of the imaging array.
4. The image sensor of claim 1 , wherein:
the at least one intensity readout circuit is configured to operate in a rolling shutter mode.
5. The image sensor of claim 4 , in combination with:
a support member, wherein the image sensor is a first image sensor mounted to the frame;
a second image sensor mounted to the support member; and
at least one processor, wherein the processor is configured to process image information from the first image sensor when operating in the rolling shutter mode and the second image sensor to produce stereoscopic image information.
6. The image sensor of claim 1 , wherein:
the at least one intensity readout circuit is configured to operate in a global shutter mode.
7. The image sensor of claim 6 , in combination with:
at least one transmissive diffraction mask in a light path to the image sensor; and
at least one processor, wherein the processor is configured to process image information from the image sensor when operating in the global shutter mode to extract passive depth information.
8. The image sensor of claim 6 , wherein:
the enable circuit is operatively coupled to the at least one intensity readout circuit to control a selected portion of the pixel cells for which a signal indicating the sensed light intensity is provided to an output of the at least one output.
9. The image sensor of claim 1 , wherein:
the at least one intensity readout circuit outputs the signals indicating the sensed light intensity at the at least a portion of the light-sensitive components of the plurality of pixel cells at fixed time intervals.
10. The image sensor of claim 1 , wherein:
the at least one intensity readout circuit outputs the signals indicating the sensed light intensity at the at least a portion of the light-sensitive components of the plurality of pixel cells at in response to a control input to the image sensor.
11. The image sensor of claim 1 , wherein:
the at least one intensity readout circuit outputs the signals indicating the sensed light intensity based on the signals indicating the change in sensed light from the at least one event readout circuit.
12. A device for use in a cross reality system, comprising:
a support member;
the image sensor of claim 1 mechanically coupled to the support member, wherein the image sensor is a first image sensor;
a second image sensor mechanically coupled to the support member and spaced apart from the first image sensor, the second image sensor comprising a second plurality of pixel cells generating image information for respective regions in a field of view of the second sensor;
a processor, operatively coupled to the first and second sensors and configured to generate depth information about an object based on the signals indicating the sensed light intensity based on correspondence between the signals indicating the change in sensed light from the at least one event readout circuit of the first image sensor and the image information from the second image sensor.
13. A method of operating a cross reality system comprising a sensor worn by a user and a processor configured to process image information, the sensor comprising a plurality of pixel cells generating image frames and event information for respective regions in a field of view of the sensor, the method comprising:
identifying an object in the vicinity of the user;
identifying a patch based at least in part on a portion of the identified object;
providing at a first data rate, from the sensor to the processor, the image frames from the plurality of pixel cells for the respective regions in the field of view of the sensor; and
selectively providing at a second data rate, from the sensor to the processor, the event information from a portion of the plurality of pixel cells based at least in part on correspondence between the patch and portion of the plurality of pixel cells, the second data rate being higher than the first data rate.
14. The method of claim 13 , wherein:
the portion of the plurality of pixel cells of the sensor is a first portion of the plurality of pixel cells, and
the method further comprises:
periodically estimating, at a third data rate, a trajectory for the patch based at least in part on the identified object; and
enabling a plurality of portions of the plurality of pixel cells of the sensor at different times based at least in part on the estimated trajectory of the patch, wherein the plurality of portions of the plurality of pixel cells comprise the first portion of the plurality of pixel cells.
15. The method of claim 13 , wherein:
the sensor is a first sensor,
the computing system comprises a second senor comprising a second plurality of pixel cells generating image frames for respective regions in a field of view of the second sensor, and
the method comprises:
providing, from the second sensor to the processor, the image frames from the second plurality of pixel cells of the second sensor for the respective regions in the field of view of the second sensor; and
computing depth information of an object within the field of views of both the first and second sensors based at least in part on correspondence between the image frames from the first sensor and the image frames from the second sensor.
16. An image sensor comprising a plurality of pixel cells, at least one pixel cell comprising:
a light-sensitive component;
an intensity readout circuit operatively coupled to the light-sensitive component, the intensity readout circuit configured to sample an intensity of a light incident sensed by the light-sensitive component;
an event readout circuit operatively coupled to the light-sensitive component, the event readout circuit configured to generate an output signal when the intensity of the light incident changes by at least a predetermined amount; and
a depth readout circuit operatively coupled to the light-sensitive component, the depth readout circuit configured to indicate an angle of the light incident.
17. The at least one pixel cell of claim 16 , wherein:
the intensity readout circuit is operatively coupled to the event readout circuit.
18. The at least one pixel cell of claim 17 , wherein:
the intensity readout circuit samples the intensity of the light incident when the event readout circuit generates the output signal.
19. The at least one pixel cell of claim 17 , wherein:
the event readout circuit is enabled or disabled based, at least in part, on an output of the intensity readout circuit.
20. The at least one pixel cell of claim 16 , wherein:
the intensity readout circuit is operatively coupled to the depth readout circuit, and
the depth readout circuit computes the angle of the light incident based, at least in part, on an output of the intensity readout circuit.