Neuromorphic compressive sensing in low-light environment
A method is provided for reconstructing images or video from output of neuromorphic vision (NMV) sensors in a low-light environment. The method includes passively sensing light by an array of NMV sensors in the low-light environment, integrating the sensed light by each of the NMV sensors, outputting a time-stamped event signal per sensor of the array of NMV sensors upon a value of the integration exceeding a threshold value, resetting each NMV sensor after outputting an event signal for new integration of sensed light, combining the event signals, and reconstructing an image and/or video based on the combined event signals.
1. A method of imaging comprising:
passively sensing light by a multi-dimensional array of neuromorphic vision (NMV) sensors in a low-light environment, wherein different NMV sensors have different spectral ranges and are distributed in a random or semi-random manner within the multi-dimensional array of NMV sensors;
integrating the sensed light by each of the NMV sensors;
outputting a time-stamped event signal per NMV sensor of the multi-dimensional array of NMV sensors upon a value of the integration exceeding a threshold value;
resetting each NMV sensor after outputting the time-stamped event signal for new integration of sensed light;
combining the time-stamped event signals; and
reconstructing at least one of an image or a video based on the combined time-stamped event signals using compressive sensing.
2. The method of claim 1 , wherein combining the time-stamped event signals includes converting the time-stamped event signals output by the NMV sensors into a format that is compatible for an algorithm used to perform the compressive sensing.
3. The method of claim 2 , wherein converting the time-stamped event signals output by the NMV sensors into the format includes formatting the time-stamped event signals output by the NMV sensors into a linear equation system having a matrix of linear equations constructed using the time-stamped event signals and identification of corresponding NMV sensors that output the respective time-stamped event signals and locations of the NMV sensors of the multi-dimensional array.
4. The method of claim 1 , further comprising:
performing online health monitoring of the NMV sensors of the multi-dimensional array using output from the compressive sensing independent of physical presence of the NMV sensors.
5. The method of claim 1 , wherein combining the time-stamped event signals uses spectral information about the spectral range of the NMV sensor that outputs each corresponding time-stamped event signal.
6. The method of claim 1 , wherein:
at least some of the NMV sensors include different types of polarization filters;
each type of polarization filter is configured to filter light at different respective angles and/or at least a portion of the multi-dimensional array of NMV sensors is oriented to observe a scene from different respective angles; and
the method further includes distributing in a random or semi-random manner the NMV sensors having the different types of polarization filters and/or being oriented at the different respective angles prior to passive sensing by the NMV sensors.
7. The method of claim 1 , further comprising:
outputting at least one of the reconstructed image or video.
8. The method of claim 6 , wherein, for each NMV sensor having a polarization filter, the threshold value is adjusted to compensate for a light loss from the polarization filter.
9. The method of claim 7 , wherein, when at least one NMV sensor of the multi-dimensional array of NMV sensors becomes inoperative, the reconstructed image or video does not include a blind spot at a location corresponding to the at least one inoperative NMV sensor.
10. A method of imaging comprising:
passively sensing light by an array of neuromorphic vision (NMV) sensors in a low-light environment;
integrating the sensed light by each of the NMV sensors;
outputting a time-stamped event signal per NMV sensor of the array of NMV sensors upon a value of the integration exceeding a threshold value;
resetting each NMV sensor after outputting the time-stamped event signal for new integration of sensed light;
combining the time-stamped event signals;
reconstructing at least one of an image or a video based on the combined time-stamped event signals; and
performing optimization of NMV array parameters using output from compressive sensing, wherein the NMV array parameters include firing thresholds for enhancement of image quality or adaptation to changing light conditions and/or a particular application.
11. A neuromorphic vision (NMV) system comprising:
a multi-dimensional array of NMV sensors configured to passively sense light, wherein the multi-dimensional array of NMV sensors is configured to be used in a low-light environment, wherein each of the NMV sensors is configured to integrate light sensed by the NMV sensor, output a time-stamped event signal upon a value of the integration exceeding a threshold value, and be reset for new integration of sensed light after outputting the time-stamped event signal, wherein different NMV sensors have different spectral ranges and are distributed in a random or semi-random manner within the multi-dimensional array of NMV sensors; and
a compressive sensing and reconstruction (CSR) engine configured to reconstruct at least one of an image or video from output of the multi-dimensional array of NMV sensors, the CSR engine comprising:
at least one memory configured to store instructions; and
at least one processor disposed in communication with the at least one memory, wherein the at least one processor, upon execution of the instructions, is configured to:
receive the time-stamped event signals from the NMV sensors of the multi-dimensional array;
combine the time-stamped event signals received; and
reconstruct at least one of the image or video based on the combined time-stamped event signals using compressive sensing.
12. The NMV system of claim 11 , wherein, to combine the time-stamped event signals, the at least one processor, upon execution of the instructions, is configured to convert the time-stamped event signals output by the NMV sensors into a format that is compatible for an algorithm used to perform the compressive sensing.
13. The NMV system of claim 12 , wherein, to convert the time-stamped event signals output by the NMV sensors into the format, the at least one processor, upon execution of the instructions, is configured to format the time-stamped event signals output by the NMV sensors into a linear equation system having a matrix of linear equations constructed using the time-stamped event signals and identification of corresponding NMV sensors that output the respective time-stamped event signals and locations of the NMV sensors of the multi-dimensional array.
14. The NMV system of claim 11 , wherein the at least one processor, upon execution of the instructions, is further configured to perform online health monitoring of the NMV sensors of the multi-dimensional array using output from the compressive sensing independent of physical presence of the NMV sensors.
15. The NMV system of claim 11 , wherein, to combine the time-stamped event signals, the at least one processor, upon execution of the instructions, is configured to use spectral information about the spectral range of the NMV sensor that outputs each corresponding time-stamped event signal.
16. The NMV system of claim 11 , wherein:
at least some of the NMV sensors include different types of polarization filters;
each type of polarization filter is configured to filter light at different respective angles and/or at least a portion of the multi-dimensional array of NMV sensors is oriented to observe a scene from different respective angles; and
the NMV sensors having the different types of polarization filters and/or being oriented at the different respective angles are distributed in a random or semi-random manner prior to passive sensing by the NMV sensors.
17. The NMV system of claim 16 , wherein the at least one processor, upon execution of the instructions, is further configured to adjust, for each NMV sensor having a polarization filter, the threshold value to compensate for a light loss from the polarization filter.
18. The NMV system of claim 11 , wherein, when at least one NMV sensor of the multi-dimensional array of NMV sensors becomes inoperative, the reconstructed image or video does not include a blind spot at a location corresponding to the at least one inoperative NMV sensor.
19. A compressive sensing and reconstruction (CSR) engine for reconstructing at least one of images or video from output of a multi-dimensional array of neuromorphic vision (NMV) sensors passively sensing light in a low-light environment, the CSR engine comprising:
at least one memory configured to store instructions;
at least one processor disposed in communication with the at least one memory, wherein the at least one processor, upon execution of the instructions, is configured to:
receive time-stamped event signals output by the NMV sensors of the multi-dimensional array, wherein the NMV sensors are configured to passively sense light, integrate the sensed light by each of the NMV sensors, output a time-stamped event signal upon a value of the integration exceeding a threshold value, and be reset for new integration of sensed light, and wherein different NMV sensors have different spectral ranges and are distributed in a random or semi-random manner within the multi-dimensional array of NMV sensors;
convert the time-stamped event signals output by the NMV sensors into a format that is compatible for an algorithm used by compressive sensing; and
reconstruct at least one of an image or a video using compressive sensing based on the converted time-stamped event signals.
20. The CSR engine of claim 19 , wherein, to convert the time-stamped event signals output by the NMV sensors into the format, the at least one processor, upon execution of the instructions, is configured to format the time-stamped event signals output by the NMV sensors into a linear equation system having a matrix of linear equations constructed using the time-stamped event signals and identification of corresponding NMV sensors that output the respective time-stamped event signals and locations of the NMV sensors of the multi-dimensional array.