Systems, methods, and media for eulerian single-photon computer vision
In accordance with some embodiments, systems, methods, and media for Eulerian single-photon computer vision are provided. In some embodiments, the system comprises: an image sensor comprising detectors configured to detect arrival of individual photons, and arranged in an array; a processor programmed to: cause the image sensor to generate a sequence of images representing a scene, each of the images comprising a plurality of pixels; perform, for each of a plurality of three dimensional filters, a convolution between the three dimensional filter and a plurality of frames, wherein each of the plurality of frames is based on one or more of the images of the sequence of images; generate, for each of the plurality of frames, a plurality of filter bank responses each corresponding to a three dimensional filter of the plurality of three dimensional filters; and perform a computer vision process based on the plurality of filter responses.
1. A system for facilitating single-photon computer vision tasks, comprising:
an image sensor comprising a plurality of detectors configured to detect arrival of individual photons, the plurality of detectors arranged in an array;
at least one processor that is programmed to:
cause the image sensor to generate a sequence of images representing a scene, each of the images comprising a plurality of pixels;
perform, for each of a plurality of three dimensional filters, a convolution between the three dimensional filter and a plurality of frames,
wherein each of the plurality of frames is based on one or more of the images of the sequence of images;
generate, for each of the plurality of frames, a plurality of filter bank responses each corresponding to a three dimensional filter of the plurality of three dimensional filters; and
perform a computer vision process based on the plurality of filter responses.
2. The system of claim 1 , wherein each of the plurality of detectors comprises a single photon avalanche diode (SPAD).
3. The system of claim 1 , wherein each image of the sequence of images comprises a binary image that represents photons detected by the image sensor during an exposure time T.
4. The system of claim 1 , wherein each of the plurality of three dimensional filters comprises a velocity tuned filter, and
wherein a first subset of the plurality of three dimensional filters are tuned to a three dimensional frequency k:=(k x , k y , k t ), where k x and k y represent spatial patterns, and k t represents a pattern in time, each of the three dimensional filters of the first subset having a different scale.
5. The system of claim 1 , wherein the at least one processor that is further programmed to:
determine a z-score for each of the plurality of filter bank responses;
map each z-score to a weight associated with the respective filter bank for which the z-score was determined; and
utilize the weighted filter bank responses to perform the computer vision process based on the plurality of filter responses.
6. The system of claim 5 , wherein the at least one processor that is further programmed to:
estimate a variance value Var(R k [p]) using a relationship V k [p]:=V(c[p])Σ q |h k [q]| 2 ,
where R k [p] is a filter bank response generated by applying a filter h k [q] to the plurality of frames B[p], V(ĉ[p]) is a variance of an estimated local average flux over B[p], and Σ q |h k [q]| 2 is a sum of the filter h k [q] over q<p; and
determine the z-score using a relationship
z
k
[
p
]
:=
❘
"\[LeftBracketingBar]"
R
k
[
p
]
❘
"\[RightBracketingBar]"
V
k
[
p
]
.
7. The system of claim 6 , wherein the at least one processor that is further programmed to:
map each z-score to a weight associated with the respective filter bank for which the z-score was determined using a relationship w(z):=1−exp(−max(0, z−z 0 )), where z 0 comprises a threshold z-score.
8. The system of claim 1 , wherein the computer vision process is an edge detection process, and
wherein at least one processor that is further programmed to:
perform the computer vision process based on a phase congruency associated with each of the plurality of filter responses.
9. The system of claim 8 , wherein at least one processor that is further programmed to:
detect one or more corners based on the phase congruency associated with each of the plurality of filter responses.
10. The system of claim 1 , wherein the computer vision process is a motion estimation process, and
wherein at least one processor that is further programmed to:
perform the computer vision process for each of the plurality of pixels based on a phase constancy relationship v x cos {tilde over (θ)}+v y sin {tilde over (θ)}=cot {tilde over (ϕ)},
where (v x , v y ) is a velocity at the respective pixel, v n =cot {tilde over (ϕ)} is a component velocity of a respective three dimensional filter {tilde over (k)} in a spatial direction {tilde over (θ)}, and {tilde over (k)}:={tilde over (s)}·(sin {tilde over (ϕ)} cos {tilde over (θ)}, sin {tilde over (ϕ)} sin {tilde over (θ)}, cos {tilde over (ϕ)}),
where ({tilde over (θ)}, {tilde over (ϕ)}) is a spatio-temporal direction of {tilde over (k)}.
11. A method for facilitating single-photon computer vision tasks, the method comprising:
causing an image sensor to generate a sequence of images representing a scene, each of the images comprising a plurality of pixels,
wherein the image sensor comprises a plurality of detectors configured to detect arrival of individual photons, the plurality of detectors arranged in an array;
performing, for each of a plurality of three dimensional filters, a convolution between the three dimensional filter and a plurality of frames,
wherein each of the plurality of frames is based on one or more of the images of the sequence of images;
generating, for each of the plurality of frames, a plurality of filter bank responses each corresponding to a three dimensional filter of the plurality of three dimensional filters; and
performing a computer vision process based on the plurality of filter responses.
12. The method of claim 11 , wherein each of the plurality of detectors comprises a single photon avalanche diode (SPAD).
13. The method of claim 11 , wherein each image of the sequence of images comprises a binary image that represents photons detected by the image sensor during an exposure time T.
14. The method of claim 11 , wherein each of the plurality of three dimensional filters comprises a velocity tuned filter, and
wherein a first subset of the plurality of three dimensional filters are tuned to a three dimensional frequency k:=(k x , k y , k t ), where k x and k y represent spatial patterns, and k t represents a pattern in time, each of the three dimensional filters of the first subset having a different scale.
15. The method of claim 11 , further comprising:
determining a z-score for each of the plurality of filter bank responses; mapping each z-score to a weight associated with the respective filter bank for which the z-score was determined; and
utilize the weighted filter bank responses to perform the computer vision process based on the plurality of filter responses.
16. The method of claim 15 , further comprising:
estimating a variance value Var(R k [p]) using a relationship V k [p]:=V(ĉ[p])Σ q |h k [q]| 2 ,
where R k [p] is a filter bank response generated by applying a filter h k [q] to the plurality of frames B[p], V(ĉ[p]) is a variance of an estimated local average flux over B[p], and Σ q |h k [q]| 2 is a sum of the filter h k [q] over q<p; and
determining the z-score using a relationship
z
k
[
p
]
:=
❘
"\[LeftBracketingBar]"
R
k
[
p
]
❘
"\[RightBracketingBar]"
V
k
[
p
]
.
17. The method of claim 11 , wherein the computer vision process is an edge detection process based on a phase congruency associated with each of the plurality of filter responses.
18. The method of claim 11 , wherein the computer vision process is a motion estimation process based on a phase constancy relationship.
19. A non-transitory computer readable medium containing computer executable instructions that, when executed by a processor, cause the processor to perform a method for facilitating single-photon computer vision tasks, the method comprising:
causing an image sensor to generate a sequence of images representing a scene, each of the images comprising a plurality of pixels,
wherein the image sensor comprises a plurality of detectors configured to detect arrival of individual photons, the plurality of detectors arranged in an array;
performing, for each of a plurality of three dimensional filters, a convolution between the three dimensional filter and a plurality of frames,
wherein each of the plurality of frames is based on one or more of the images of the sequence of images;
generating, for each of the plurality of frames, a plurality of filter bank responses each corresponding to a three dimensional filter of the plurality of three dimensional filters; and
performing a computer vision process based on the plurality of filter responses.
20. The non-transitory computer readable medium of claim 19 , wherein each of the plurality of detectors comprises a single photon avalanche diode (SPAD).
21. The non-transitory computer readable medium of claim 19 , wherein each image of the sequence of images comprises a binary image that represents photons detected by the image sensor during an exposure time T.