Event based computer vision computation
Certain techniques are provided for triggering events based on detecting changes in CV features, such as edges, corners etc., by generating computed results based on sensor readings. Yet other techniques also provided for detecting CV features once an event is detecting based on a change in a sensor reading for a sensor element. In certain aspects, the event detection logic and the feature calculation CV vision operations may be performed, individually, or in combination in circuitry on a sensor or in software/firmware on a computing device.
1. A method, comprising:
comparing a previous localized computer vision (CV) feature associated with a subject sensor element of a sensor element array computed using CV feature computation hardware and a current localized CV feature associated with the subject sensor element computed using the CV feature computation hardware, wherein, for both the previous localized CV feature and the current localized CV feature, the CV feature computation hardware computes the respective CV feature using a respective reading from the subject sensor element and a respective reading from a neighboring sensor element to the subject sensor element, and wherein:
the sensor element array comprises a plurality of sensor elements, the plurality of sensor elements arranged along at least a first dimension and a second dimension of the sensor element array,
a pixel comprises the subject sensor element and in-pixel circuitry, and
the CV feature computation hardware is implemented as part of the in-pixel circuitry;
generating, based on the comparing, an event associated with the subject sensor element, the event comprising a location of the subject sensor element and a feature payload comprising a plurality of bits representing the computed current localized CV feature associated with the subject sensor element; and
sending the event to a processor.
2. The method of claim 1 , wherein the localized CV feature comprises a local binary pattern (LBP) label or a variation of the LBP label.
3. The method of claim 1 , wherein the localized CV feature comprises a histogram of signed gradient (HSG) label.
4. The method of claim 1 , wherein the sensor element array is a vision sensor and each of the plurality of sensor elements of the sensor element array comprises at least one photodiode.
5. The method of claim 1 , wherein the event is generated using dynamic vision sensor (DVS) circuitry.
6. A vision sensor comprising:
a sensor element array comprising a plurality of sensor elements, the plurality of sensor elements arranged along at least a first dimension and a second dimension of the sensor element array, each of the plurality of sensor elements capable of generating sensor readings based on light incident upon the sensor elements;
a computer vision (CV) feature computation hardware configured to compute a previous localized CV feature associated with a subject sensor element of the sensor element array and a current localized CV feature associated with the subject sensor element, wherein, for both the previous localized CV feature and the current localized CV feature, the CV feature computation hardware computes the respective CV feature using a respective reading from the subject sensor element and a respective reading from a neighboring sensor element to the subject sensor element, and wherein:
a pixel comprises the subject sensor element and in-pixel circuitry, and
the CV feature computation hardware is implemented as part of the in-pixel circuitry;
a comparator configured to compare the previous localized CV feature associated with the subject sensor element and the current localized CV feature associated with the subject sensor element; and
the CV feature computation hardware further configured to generate, based on the comparison by the comparator, an event associated with the subject sensor element, the event comprising a location of the subject sensor element and a feature payload comprising a plurality of bits representing the computed current localized CV feature associated with the subject sensor element, and send the event to a processor.
7. The vision sensor of claim 6 , wherein the localized CV feature comprises a local binary pattern (LBP) label or a variation of the LBP label.
8. The vision sensor of claim 6 , wherein the localized CV feature comprises a histogram of signed gradient (HSG) label.
9. The vision sensor of claim 6 , wherein each of the plurality of sensor elements of the sensor element array comprises at least one photodiode.
10. The vision sensor of claim 6 , wherein the event is generated using dynamic vision sensor (DVS) circuitry.
11. An apparatus, comprising:
means for comparing a previous localized computer vision (CV) feature associated with a subject sensor element of a sensor element array computed using CV feature computation hardware and a current localized CV feature associated with the subject sensor element computed using the CV feature computation hardware, wherein, for both the previous localized CV feature and the current localized CV feature, the CV feature computation hardware computes the respective CV feature using a respective reading from the subject sensor element and a respective reading from a neighboring sensor element to the subject sensor element, and wherein:
the sensor element array comprises a plurality of sensor elements, the plurality of sensor elements arranged along at least a first dimension and a second dimension of the sensor element array,
a pixel comprises the subject sensor element and in-pixel circuitry, and
the CV feature computation hardware is implemented as part of the in-pixel circuitry;
means for generating, based on the comparing, an event associated with the subject sensor element, the event comprising a location of the subject sensor element and a feature payload comprising a plurality of bits representing the computed current localized CV feature associated with the subject sensor element; and
means for sending the event to a processor.
12. The apparatus of claim 11 , wherein the localized CV feature comprises a local binary pattern (LBP) label or a variation of the LBP label.
13. The apparatus of claim 11 , wherein the localized CV feature comprises a histogram of signed gradient (HSG) label.
14. The apparatus of claim 11 , wherein the sensor element array is a vision sensor and each of the plurality of sensor elements of the sensor element array comprises at least one photodiode.
15. A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises instructions executable by a dedicated computer vision (CV) processing module for:
comparing a previous localized CV feature associated with a subject sensor element of a sensor element array computed using CV feature computation hardware and a current localized CV feature associated with the subject sensor element computed using the CV feature computation hardware, wherein, for both the previous localized CV feature and the current localized CV feature, the CV feature computation hardware computes the respective CV feature using a respective reading from the subject sensor element and a respective reading from a neighboring sensor element to the subject sensor element, and wherein:
the sensor element array comprises a plurality of sensor elements, the plurality of sensor elements arranged along at least a first dimension and a second dimension of the sensor element array,
a pixel comprises the subject sensor element and in-pixel circuitry, and
the CV feature computation hardware is implemented as part of the in-pixel circuitry;
generating, based on the comparing, an event associated with the subject sensor element, the event comprising a location of the subject sensor element and a feature payload comprising a plurality of bits representing the computed current localized CV feature associated with the subject sensor element; and
sending the event to a processor.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the localized CV feature comprises a local binary pattern (LBP) label or a variation of the LBP label.
17. The non-transitory computer-readable storage medium of claim 15 , wherein the localized CV feature comprises a histogram of signed gradient (HSG) label.
18. The non-transitory computer-readable storage medium of claim 15 , wherein the sensor element array is a vision sensor and each of the plurality of sensor elements of the sensor element array comprises at least one photodiode.