Patch tracking image sensor
An image sensor suitable for use in an augmented reality system to provide low latency image analysis with low power consumption. The augmented reality system can be compact, and may be small enough to be packaged within a wearable device such as a set of goggles or mounted on a frame resembling ordinary eyeglasses. The image sensor may receive information about a region of an imaging array associated with a movable object and selectively output imaging information for that region. The region may be updated dynamically as the image sensor and/or the object moves. Such an image sensor provides a small amount of data from which object information used in rendering an augmented reality scene can be developed. The amount of data may be further reduced by configuring the image sensor to output indications of pixels for which the measured intensity of incident light changes.
1. A device comprising:
a support member;
a sensor mechanically coupled to the support member, the sensor comprising a plurality of pixel cells generating image information for respective regions in a field of view of the sensor;
at least one processor operatively coupled to the sensor and configured to:
process image information from the sensor;
dynamically modify a portion of the plurality of pixel cells included in a patch based on information indicating a position of the patch among the plurality of pixel cells at a first time and based on an estimated trajectory of the patch among the plurality of pixel cells subsequent to the first time; and
in a time period subsequent to the first time:
control the sensor to selectively provide for processing, by at least one of the at least one processor, image information from the portion of the plurality of pixel cells in the patch without providing image information from pixel cells of the plurality of pixel cells that are outside of the patch.
2. The device of claim 1 , wherein:
the sensor comprises a dynamic vision sensor (DVS).
3. The device of claim 1 , wherein:
the sensor comprises a transmissive diffraction mask (TDM).
4. The device of claim 1 , wherein:
the sensor is a first sensor,
the plurality of pixel cells is a first plurality of pixel cells, and
the computing device further comprises a second sensor coupled to the support member, the second sensor comprising a second plurality of pixel cells, and configured to output frames at fixed time intervals.
5. The device of claim 4 , wherein:
the at least one processor is further configured to selectively enable image information from the second plurality of pixel cells to be coupled to the at least one of the at least one processor at times subsequent to the first time.
6. The device of claim 1 , further comprising:
at least one memory, wherein the at least one processor is further configured to selectively enable image information from the at least one memory to be coupled to the at least one of the at least one processor at times subsequent to the first time.
7. The device of claim 1 , wherein controlling the sensor to selectively provide, to the at least one of the at least one processor, the image information from the portion of pixel cells in the patch without providing the image information from the pixel cells that are outside of the patch comprises:
providing signals to the portion of pixel cells in the patch to enable output of information from the portion of pixel cells without providing signals to the pixel cells that are outside of the patch.
8. The device of claim 7 , wherein each pixel cell of the plurality of pixel cells comprises a comparator, the comparator comprising an enable input.
9. The computing device of claim 8 , wherein providing the signals to the portion of pixel cells in the patch to enable output of information from the portion of pixel cells comprises providing the signals to enable inputs of comparators of the portion of pixel cells in the patch.
10. A system comprising:
a sensor worn by a user, the sensor comprising a plurality of pixel cells configured to generate image information for respective regions in a field of view of the sensor;
a processor configured process image information obtained by the sensor; and
circuitry configured to:
analyze one or more images depicting a vicinity of the user;
identify, based on analyzing the one or more images, an object in the vicinity of the user;
identify a portion of the plurality of pixel cells as a patch representing at least a portion of the identified object;
estimate a trajectory of the patch among the plurality of pixel cells;
dynamically update a location of the patch among the plurality of pixel cells based on the estimated trajectory of the patch by updating which of the plurality of pixel cells is included in the patch while the computing system is operating; and
control the sensor to selectively provide, to the processor, image information from the portion of the plurality of pixel cells in the patch without providing image information from pixel cells of the plurality of pixel cells that are outside of the patch.
11. The system of claim 10 , wherein:
the portion of the plurality of pixel cells of the sensor is a first portion of the plurality of pixel cells, and
the circuitry is further configured to:
estimate the trajectory of the patch based at least in part on objects represented by the portion of the plurality of pixel cells of the one or more images; and
enable a plurality of portions of the plurality of pixel cells of the sensor at different times based at least in part on the estimated trajectory of the patch, wherein the plurality of portions of the plurality of pixel cells comprise the first portion of the plurality of pixel cells.
12. The system of claim 10 , wherein the circuitry is configured to estimate the trajectory for the patch by performing:
predict one or more motion vectors for the objects and/or the user; and
compute the trajectory for the patch based at least in part on the predicted one or more motion vectors.
13. The system of claim 10 , wherein the circuitry is further configured to:
dynamically adjust a size of the patch based at least in part on the estimated trajectory.
14. The system of claim 10 , wherein obtaining the one or more images comprises obtaining the one or more images from another sensor and/or a storage memory.
15. The system of claim 10 , wherein the circuitry is configured to control the sensor to selective provide, to the processor, the image information from the portion of pixels in the patch without providing the image information from the pixel cells that are outside of the patch by performing:
provide signals to the portion of pixel cells in the patch to enable output of information from the portion of pixel cells without providing signals to the pixel cells that are outside of the patch.
16. The system of claim 15 , wherein:
each of the plurality of pixel cells comprises a comparator, the comparator comprising an enable input; and
the circuitry is configured to provide the signals to the portion of pixel cells in the patch to enable output of information from the portion of pixel cells by providing the signals to enable inputs of comparators in the portion of pixel cells in the patch.
17. A method of operating a system comprising a sensor worn by a user and a processor configured to process image information obtained by the sensor, the sensor comprising a plurality of pixel cells generating image information for respective regions in a field of view of the sensor, the method comprising:
analyzing one or more images depicting a vicinity of the user;
identifying, based on analyzing the one or more images, an object in the vicinity of the user;
identifying a portion of the plurality of pixel cells as a patch representing at least a portion of the identified object;
estimating a trajectory of the patch among the plurality of pixel cells;
dynamically updating a location of the patch among the plurality of pixel cells based on the estimated trajectory of the patch by updating which of the plurality of pixel cells is included in the patch while the computing system is operating; and
controlling the sensor to selectively provide, to the processor, image information from the portion of the plurality of pixel cells in the patch without providing image information from pixel cells of the plurality of pixels that are outside of the patch.
18. The method of claim 17 , wherein:
the portion of the plurality of pixel cells of the sensor is a first portion of the plurality of pixel cells, and
the method further comprises:
estimating the trajectory of the patch based at least in part on objects represented by the portion of the plurality of pixels of the one or more images; and
enabling a plurality of portions of the plurality of pixel cells of the sensor at different times based at least in part on the estimated trajectory of the patch, wherein the plurality of portions of the plurality of pixel cells comprise the first portion of the plurality of pixel cells.
19. The method of claim 18 , estimating the trajectory for the patch comprises:
predicting one or more motion vectors for the objects and/or the user; and
computing the trajectory for the patch based at least in part on the predicted one or more motion vectors.
20. The method of claim 19 , wherein:
estimating the trajectory for the patch further comprises dynamically adjusting a size of the patch based at least in part on the estimated trajectory.