Event-based camera with high-resolution frame output
A high-resolution image sensor suitable for use in an augmented reality (AR) system to provide low latency image analysis with low power consumption. The AR system can be compact, and may be small enough to be packaged within a wearable device such as a set of goggles or mounted on a frame resembling ordinary eyeglasses. The image sensor may receive information about a region of an imaging array associated with a movable object, selectively output imaging information for that region, and synchronously output high-resolution image frames. The region may be updated dynamically as the image sensor and/or the object moves. The image sensor may output the high-resolution image frames less frequently than the region being updated when the image sensor and/or the object moves. Such an image sensor provides a small amount of data from which object information used in rendering an AR scene can be developed.
1. An augmented reality system, comprising:
at least one image sensor comprising:
a plurality of pixel cells, each pixel cell of the plurality of pixel cells comprising a light-sensitive component;
first circuitry coupled to light-sensitive components of pixel cells of the plurality of pixel cells and configurable to provide, as output signals from at least some pixel cells of the plurality of pixel cells, indications of light intensity; and
second circuitry coupled to light-sensitive components of pixel cells of the plurality of pixel cells and configurable to provide, as output signals from at least some pixel cells of the plurality of pixel cells, event-based indications of change in a light property based on the change in the light property exceeding a threshold; and
at least one processor configured to:
for a first image data processing technique, receive, from the first circuitry, indications of light intensity; and
for a second image data processing technique, receive, from the second circuitry, event-based indications of change in the light property.
2. The augmented reality system of claim 1 , wherein the at least one processor is further configured to:
perform the first image data processing technique, the first image data processing technique comprising generating or updating a world model using the indications of light intensity.
3. The augmented reality system of claim 1 , wherein the at least one processor is further configured to:
perform the first image data processing technique using indications of light intensity for a full image frame.
4. The augmented reality system of claim 1 , wherein the at least one processor is further configured to:
perform the first image data processing technique using indications of light intensity captured using a rolling shutter mode.
5. The augmented reality system of claim 1 , wherein the at least one processor is further configured to:
perform the first image data processing technique using indications of light intensity captured using a global shutter mode.
6. The augmented reality system of claim 1 , wherein:
the at least one image sensor comprises optical components disposed in light paths at least some pixel cells of the plurality of pixel cells;
the optical components are configured to convert an angle of incident light into light intensity; and
the at least one processor is further configured to perform the first image data processing technique, the first image data processing technique comprising generating passive depth information using the indications of light intensity.
7. The augmented reality system of claim 1 , wherein the augmented reality system comprises a wearable headset.
8. Non-transitory computer-readable medium comprising computer-executable instructions, configured for execution on at least one processor of an augmented reality system comprising at least one image sensor comprising a plurality of pixel cells, each pixel cell of the plurality of pixel cells comprising a light-sensitive component; first circuitry coupled to light-sensitive components of pixel cells of the plurality of pixel cells and configurable to provide, as output signals from at least some pixel cells of the plurality of pixel cells, indications of light intensity; and second circuitry coupled to light-sensitive components of pixel cells of the plurality of pixel cells and configurable to provide, as output signals from at least some pixel cells of the plurality of pixel cells, indications of change in a light property, the computer-executable instructions configured to, when executed, control the at least one processor to:
set an output mode of the at least one image sensor based on an image data processing technique to be performed, wherein:
the at least one image sensor is set to operate in a first mode in which the first circuitry provides, as output signals from at least some pixel cells of the plurality of pixel cells, indications of light intensity to perform a first image data processing technique;
the at least one image sensor is set to operate in a second mode in which the second circuitry provides, as output signals from at least some pixel cells of the plurality of pixel cells, indications of change in a light property to perform a second image data processing technique;
receive for processing, based on an image processing technique to be performed, indications of light intensity from the first circuitry or indications of change in the light property from the second circuitry, such that:
for the first image data processing technique, the at least one processor receives, from the first circuitry, indications of light intensity; and
for the second image data processing technique, the at least one processor receives, from the second circuitry, indications of change in the light property.
9. The non-transitory computer-readable medium of claim 8 , wherein the computer-executable instructions further comprise computer-executable instructions configured to, when executed control the at least one processor to:
perform the first image data processing technique, wherein the first image data processing technique comprises tracking an object or pose using the indications of change in the light property.
10. The non-transitory computer-readable medium of claim 8 , wherein the computer-executable instructions further comprise computer-executable instructions configured to, when executed control the at least one processor to:
perform the second image data processing technique using the indications of change in the light property for an image patch.
11. The non-transitory computer-readable medium of claim 8 , wherein the computer-executable instructions further comprise computer-executable instructions configured to, when executed control the at least one processor to:
perform the first image data processing technique using indications of light intensity received at a first data rate; and
perform the second image data processing technique using indications of change in the light property received at a second data rate.
12. The non-transitory computer-readable medium of claim 8 , wherein the computer-executable instructions further comprise computer-executable instructions configured to, when executed control the at least one processor to:
perform the first image data processing technique using indications of light intensity in response to an indication to operate in a first power consumption mode; and
perform the second image data processing technique using indications of change in the light property in response to an indication to operate in a second power consumption mode.
13. A method of operating an augmented reality system, comprising at least one image sensor comprising a plurality of pixel cells, each pixel cell of the plurality of pixel cells comprising a light-sensitive component, first circuitry coupled to light-sensitive components of the plurality of pixel cells and configurable to provide, as output signals from at least some pixel cells of the plurality of pixel cells, indications of light intensity; and second circuitry coupled to light-sensitive components of the plurality of pixel cells and configurable to provide, as output signals from at least some pixel cells of the plurality of pixel cells, indications of change in a light property, the method comprising:
controlling an output mode of the at least one image sensor such that the at least one image sensor operates in:
a first mode in which the first circuitry provides, as output signals from at least some pixel cells of the plurality of pixel cells, indications of light intensity; or
a second mode in which the second circuitry provides, as output signals from at least some pixel cells of the plurality of pixel cells, indications of change in a light property;
using at least one processor to, for a first image data processing technique, receive from the first circuitry, indications of light intensity; and
using the at least one processor to for a second image data processing technique, receive, from the second circuitry, indications of change in the light property.
14. The method of claim 13 , further comprising:
using the at least one processor to perform the first image data processing technique using indications of light intensity for a full image frame.
15. The method of claim 13 , further comprising:
using the at least one processor to perform the first image data processing technique using indications of light intensity captured using a rolling shutter mode.
16. The method of claim 13 , further comprising:
using the at least one processor to perform the first image data processing technique using indications of light intensity captured using a global shutter mode.
17. The method of claim 13 , further comprising:
using the at least one processor to perform the second image data processing technique, the first image data processing technique comprising tracking an object or pose using the indications of change in the light property.
18. The method of claim 13 , further comprising:
using the at least one processor to perform the second image data processing technique using the indications of change in the light property for an image patch.
19. The method of claim 13 , further comprising:
using the at least one processor to perform the first image data processing technique using indications of light intensity received at a first data rate; and
using the at least one processor to perform the second image data processing technique using indications of change in the light property received at a second data rate.
20. The method of claim 13 , further comprising:
using the at least one processor to perform the first image data processing technique using indications of light intensity in response to an indication to operate in a first power consumption mode; and
using the at least one processor to perform the second image data processing technique using indications of change in the light property in response to an indication to operate in a second power consumption mode.