Systems, methods, and media for estimating a depth and orientation of a portion of a scene using a single-photon detector and diffuse light source

1 . A system for estimating at least one property in a scene, the system comprising:

a light source;

an image sensor comprising an array comprising a plurality of pixels including a pixel,

wherein each of the plurality of pixels comprises a single photon avalanche diode (SPAD),

wherein the pixel has a field of view of at least one degree, and

wherein the field of view of each of the plurality of pixels in the array overlaps with the field of view of each neighboring pixel;

at least one hardware processor that is programmed to:

cause the light source to emit a sequence of n defocused pulses toward the scene;

receive, from the pixel, information indicative of arrival times of light from the scene,

wherein the information indicative of arrival times of light from the scene comprises a plurality of timestamps each indicative of an arrival time of a single photon at the SPAD of the pixel;

generate a transient histogram using the information indicative of arrival times of light from the scene; and

estimate one or more properties of a portion of the scene within the field of view of the pixel based on the transient histogram, wherein the one or more properties of the portion of the scene includes at least a depth.

2 . The system of claim 1 , further comprising a diffuser disposed between the light source and the scene.

3 . The system of claim 1 , wherein the one or more properties of the portion of the scene includes a normal orientation θ n of the portion of the scene with respect to an optical axis of the pixel.

4 . The system of claim 3 , wherein the at least one hardware processor is further programmed to:

identify a leading edge of a peak in the transient histogram;

estimate a first distance D 1 based on the leading edge of the peak in the transient histogram;

generate a first estimate of the normal orientation θ n based on D 1 ;

identify a trailing edge of a peak in the transient histogram;

estimate a second distance D 2 based on the trailing edge of the peak in the transient histogram;

generate a second estimate of the normal orientation θ n based on D 2 ; and

estimate the normal orientation θ n based on the first estimate and the second estimate.

5 . The system of claim 3 , wherein the at least one hardware processor is further programmed to:

estimate a Fourier transform of the transient histogram, ({tilde over (φ)}), where is the Fourier transform and {tilde over (φ)} is the transient histogram;

estimate a Fourier transform of a forward rendering model R(θ n , Z 0 ), (R(θ n , Z 0 )), where Z 0 is a depth of the portion of the scene along the optical axis of the pixel; and

solve an optimization problem using gradient decent, wherein the optimization problem is expressed as:

minimize

⁢



ℱ

⁡

(

R

⁡

(

θ

n

,

Z

0

)

)

-

F

⁡

(

φ

¯

)



2

2

,

and initial values for θ n and Z 0 are estimated using a leading edge of a peak in the transient histogram and a trailing edge of the peak in the transient histogram,

wherein solving the optimization problem comprises calculating an 2 norm using k Fourier coefficients for ({tilde over (φ)}) and (R(θ n , Z 0 )), where k is less than all the Fourier coefficients.

6 . The system of claim 1 , wherein the at least one hardware processor is further programmed to:

receive, from each of the plurality of pixels, information indicative of arrival times of light from the scene;

generate a plurality of transient histograms, each of the plurality of transient histograms corresponding to a pixel of the plurality of pixels;

estimate a Fourier transform of each transient histogram of the plurality of transient histograms, yielding a plurality of Fourier coefficients for each transient histogram;

provide k Fourier coefficients of each of the plurality of Fourier coefficients as input to a trained machine learning model, wherein the trained machine learning model was trained to generate a depth map; and

receive, from the trained machine learning model, a depth map for the scene comprising the one or more properties of the portion of the scene.

7 . A method for estimating at least one property in a scene, the method comprising:

causing a light source to emit a sequence of n defocused pulses toward the scene;

receiving, from a pixel of an image sensor, information indicative of arrival times of light from the scene,

wherein the image sensor comprises an array comprising a plurality of pixels including the pixel,

wherein each of the plurality of pixels comprises a single photon avalanche diode (SPAD),

wherein the pixel has a field of video of at least one degree,

wherein the field of view of each of the plurality of pixels in the array overlaps with the field of view of each neighboring pixel, and

wherein the information indicative of arrival times of light from the scene comprises a plurality of timestamps each indicative of an arrival time of a single photon at the SPAD of the pixel;

generating a transient histogram using the information indicative of arrival times of light from the scene; and

estimating one or more properties of a portion of the scene within the field of view of the pixel based on the transient histogram, wherein the one or more properties of the portion of the scene includes at least a depth.

8 . The method of claim 7 , wherein a diffuser disposed between the light source and the scene causes the light emitted by the light source to be defocused.

9 . The method of claim 7 , wherein the one or more properties of the portion of the scene includes a normal orientation θ n of the portion of the scene with respect to an optical axis of the pixel.

10 . The method of claim 9 , further comprising:

identifying a leading edge of a peak in the transient histogram;

estimating a first distance D 1 based on the leading edge of the peak in the transient histogram;

generating a first estimate of the normal orientation θ n based on D 1 ;

identifying a trailing edge of a peak in the transient histogram;

estimating a second distance D 2 based on the trailing edge of the peak in the transient histogram;

generating a second estimate of the normal orientation θ n based on D 2 ; and

estimating the normal orientation θ n based on the first estimate and the second estimate.

11 . The method of claim 9 , further comprising:

estimating a Fourier transform of the transient histogram, ({tilde over (φ)}), where is the Fourier transform and {tilde over (φ)} is the transient histogram;

estimating a Fourier transform of a forward rendering model R(θ n , Z 0 ), (R(θ n , Z 0 )), where Z 0 is a depth of the portion of the scene along the optical axis of the pixel; and

solving an optimization problem using gradient decent, wherein the optimization problem is expressed as:

minimize

⁢



ℱ

(

R

(

θ

n

,

Z

0

)

)

-

F

(

φ

¯

)



2

2

,

and initial values for θ n and Z 0 are estimated using a leading edge of a peak in the transient histogram and a trailing edge of the peak in the transient histogram,

wherein solving the optimization problem comprises calculating an 2 norm using k Fourier coefficients for ({tilde over (φ)}) and (R(θ n , Z 0 )), where k is less than all the Fourier coefficients.

12 . The method of claim 7 , the method further comprising:

receiving, from each of the plurality of pixels, information indicative of arrival times of light from the scene;

generating a plurality of transient histograms, each of the plurality of transient histograms corresponding to a pixel of the plurality of pixels;

estimating a Fourier transform of each transient histogram of the plurality of transient histograms, yielding a plurality of Fourier coefficients for each transient histogram;

providing k Fourier coefficients of each of the plurality of Fourier coefficients as input to a trained machine learning model, wherein the trained machine learning model was trained to generate a depth map; and

receiving, from the trained machine learning model, a depth map for the scene comprising the one or more properties of the portion of the scene.

13 . A non-transitory computer readable medium containing computer executable instructions that, when executed by a processor, cause the processor to perform a method for estimating at least one property in a scene, the method comprising:

causing a light source to emit a sequence of n defocused pulses toward the scene;

receiving, from a pixel of an image sensor, information indicative of arrival times of light from the scene,

wherein the image sensor comprises an array comprising a plurality of pixels including the pixel,

wherein each of the plurality of pixels comprises a single photon avalanche diode (SPAD),

wherein the pixel has a field of video of at least one degree,

wherein the field of view of each of the plurality of pixels in the array overlaps with the field of view of each neighboring pixel, and

wherein the information indicative of arrival times of light from the scene comprises a plurality of timestamps each indicative of an arrival time of a single photon at the SPAD of the pixel;

generating a transient histogram using the information indicative of arrival times of light from the scene; and

estimating one or more properties of a portion of the scene within the field of view of the pixel based on the transient histogram, wherein the one or more properties of the portion of the scene includes at least a depth.

14 . The non-transitory computer readable medium of claim 13 , wherein the one or more properties of the portion of the scene includes a normal orientation θ n of the portion of the scene with respect to an optical axis of the pixel.

15 . The non-transitory computer readable medium of claim 14 , wherein the method further comprises:

identifying a leading edge of a peak in the transient histogram;

estimating a first distance D 1 based on the leading edge of the peak in the transient histogram;

generating a first estimate of the normal orientation θ n based on D 1 ;

identifying a trailing edge of a peak in the transient histogram;

estimating a second distance D 2 based on the trailing edge of the peak in the transient histogram;

generating a second estimate of the normal orientation θ n based on D 2 ; and

estimating the normal orientation θ n based on the first estimate and the second estimate.

16 . The non-transitory computer readable medium of claim 14 , further comprising:

estimating a Fourier transform of a forward rendering model R(θ n , Z 0 ), (R(θ n , Z 0 )), where Z 0 is a depth of the portion of the scene along the optical axis of the pixel; and

solving an optimization problem using gradient decent, wherein the optimization problem is expressed as:

minimize

⁢



ℱ

(

R

(

θ

n

,

Z

0

)

)

-

F

(

φ

¯

)



2

2

,

and initial values for θ n and Z 0 are estimated using a leading edge of a peak in the transient histogram and a trailing edge of the peak in the transient histogram,

wherein solving the optimization problem comprises calculating an 2 norm using k Fourier coefficients for ({tilde over (φ)}) and (R(θ n , Z 0 )), where k is less than all the Fourier coefficients.

17 . The non-transitory computer readable medium of claim 13 , wherein the method further comprises:

receiving, from each of the plurality of pixels, information indicative of arrival times of light from the scene;

generating a plurality of transient histograms, each of the plurality of transient histograms corresponding to a pixel of the plurality of pixels;

estimating a Fourier transform of each transient histogram of the plurality of transient histograms, yielding a plurality of Fourier coefficients for each transient histogram;

providing k Fourier coefficients of each of the plurality of Fourier coefficients as input to a trained machine learning model, wherein the trained machine learning model was trained to generate a depth map; and

receiving, from the trained machine learning model, a depth map for the scene comprising the one or more properties of the portion of the scene.