Methods and apparatus for multi-frequency camera
In exemplary implementations of this invention, a multi-frequency ToF camera mitigates the effect of multi-path interference (MPI), and can calculate an accurate depth map despite MPI. A light source in the multi-frequency camera emits light in a temporal sequence of different frequencies. For example, the light source can emit a sequence of ten equidistant frequencies f=10 MHz, 20 MHz, 30 MHz, . . . , 100 MHz. At each frequency, a lock-in sensor within the ToF camera captures 4 frames. From these 4 frames, one or more processors compute, for each pixel in the sensor, a single complex number. The processors stack all of such complex quantities (one such complex number per pixel per frequency) and solve for the depth and intensity, using a spectral estimation technique.
1. A method comprising, in combination:
(a) using a light source to emit light in a temporal sequence of frequencies, such that
(i) the light illuminates a scene,
(ii) reflected light from the scene reaches a light sensor, and
(iii) a first frequency is emitted during a first period of time in the sequence and not during other periods of time during the sequence, a second frequency is emitted during a second period of time in the sequence and not during other periods of time during the sequence, and a third frequency is emitted during a third period of time in the sequence and not during other periods of time during the sequence;
(b) using a light sensor to capture multiple frames for each respective frequency in the sequence; and
(c) using one or more processors
(i) to analyze data in the frames and
(ii) to calculate a set of complex numbers for each respective pixel in the light sensor, such that the set of complex numbers includes a complex number for each respective frequency in the sequence for each respective path length in a set of path lengths, wherein the set of path lengths consists of path lengths of paths of light that end at the respective pixel.
2. The method of claim 1 , wherein the method further comprises using the one or more processors to calculate a depth map of the scene.
3. The method of claim 1 , wherein the paths of light include a single-bounce path.
4. The method of claim 1 , wherein the paths of light include a multiple-bounce path.
5. The method of claim 1 , wherein the light sensor comprises a lock-in sensor.
6. The method of claim 1 , wherein the light sensor captures four frames for each respective frequency in the sequence of frequencies.
7. The method of claim 1 , wherein the one or more processors calculate multiple components of light incident on a single pixel of the light sensor, each respective component, out of the multiple components, comprising light that travels a particular path length, out of the set of path lengths.
8. The method of claim 1 , wherein the algorithm includes steps for computing frequency components of a signal.
9. The method of claim 1 , wherein the algorithm includes a matrix pencil computation.
10. The method of claim 1 , wherein the algorithm includes a computation that uses Prony's Method.
11. The method of claim 1 , wherein the algorithm includes a computation that uses a MUSIC method of spectral estimation.
12. The method of claim 1 , wherein the algorithm includes a computation that uses a maximum likelihood method of spectral estimation.
13. The method of claim 1 , wherein the algorithm includes a computation that uses an ESPRIT method of spectral estimation.
14. The method of claim 1 , wherein the algorithm includes a computation that uses a parametric method of spectral estimation.
15. A method of determining a depth map of a scene using a time-of-flight camera, which camera includes a light sensor,
THE IMPROVEMENT COMPRISING:
(a) using a light source to emit light in a temporal sequence of frequencies to illuminate the scene, such that a first frequency is emitted during a first period of time in the sequence and not during other periods of time during the sequence, a second frequency is emitted during a second period of time in the sequence and not during other periods of time during the sequence, and a third frequency is emitted during a third period of time in the sequence and not during other periods of time during the sequence; and
(b) using one or more processors to estimate a phase and coefficient of reflection for each respective pixel of the light sensor for each respective path length, out of a set of path lengths of light, each respective path length being for a path that starts at the light source and ends at the light sensor.
16. A time-of-flight camera, which camera includes a light sensor,
THE IMPROVEMENT COMPRISING:
(a) one or more light sources for emitting light in a temporal sequence of different frequencies, such that a first frequency is emitted during a first period of time in the sequence and not during other periods of time during the sequence, a second frequency is emitted during a second period of time in the sequence and not during other periods of time during the sequence, and a third frequency is emitted during a third period of time in the sequence and not during other periods of time during the sequence; and
(b) one or more processors programmed to estimate a phase and coefficient of reflection for each respective pixel of the light sensor for each respective path length, out of a set of path lengths of light, each respective path length being for a path that starts at the light source and ends at the light sensor.
17. The time-of-flight camera of claim 16 , wherein the one or more processors are further programmed to calculate a depth map of a scene.
18. The time-of-flight camera of claim 16 , wherein the one or more processors are further programmed to identify a direct component of light incident on a pixel of the sensor.
19. The time-of-flight camera of claim 16 , wherein the one or more processors are further programmed to identify a global component of light incident on a pixel of the sensor.