Depth sensing camera system
View Patent ↗A depth sensing camera system that comprises one or more fisheye lenses and infrared and/or near-infrared image sensors. In some examples, the image sensors may generate output signals based at least in part on receiving radiation via the fisheye lenses. A depth measurement may be calculated based at least in part on the output signals. For example, these output signals may be provided as input to a depth model, which may determine the depth measurement. In some examples, such a depth model may be integrated into an application-specific integrated circuit and/or may be operated by an application processor.
1. A system comprising:
an illuminator configured to emit a first type of radiation;
a fisheye lens characterized by a field of view (FOV);
a first image sensor configured to generate a first output signal based at least in part on a portion of first light received via the fisheye lens, the first light characterized by the first type of radiation, wherein the first image sensor is further configured to crop the FOV of the fisheye lens so that only a portion of the first light in a central portion of the FOV of the fisheye lens remains, wherein a curvature of one or more lenses of the fisheye lens and a spacing of the fisheye lens from the first image sensor is modified to adjust an effective sensor FOV of the fisheye lens;
a second image sensor configured to generate a second output signal based at least in part on receiving second light via a second lens, the second light comprising at least one of the first type of radiation or a second type of radiation; and
a controller operatively connected to the first image sensor and the second image sensor and configured to:
generate, based at least in part on the first output signal, a depth map of an environment visible in a field of view of the first image sensor and the second image sensor.
2. The system of claim 1 , wherein a first field of view of the first image sensor and a second field of view of the second image sensor overlap.
3. The system of claim 1 , wherein generating the depth map comprises:
providing, as input to a machine-learned model, at least one of the first output signal or a representation of the first output signal; and
receiving, from the machine-learned model, the depth map.
4. The system of claim 1 , wherein the first type of radiation includes infrared light or near-infrared light and the second type of radiation includes visible light.
5. The system of claim 1 , wherein the first image sensor comprises at least one of a monochrome infrared sensor or a monochrome near-infrared sensor.
6. The system of claim 1 , wherein:
the controller comprises a processor; and
generating the depth map is based at least in part on one or more processors executing instructions stored in a memory, the instructions stored in the memory comprising one or more weights of a neural network.
7. The system of claim 1 , wherein the controller comprises an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).
8. A device comprising:
a first fisheye lens characterized by a field of view (FOV);
a first image sensor configured to generate a first output signal responsive to a portion of first radiation via the first fisheye lens, wherein the first image sensor is further configured to crop the FOV of the fisheye lens so that only a portion of the first radiation in a central portion of the FOV of the fisheye lens remains, wherein a curvature of one or more lenses of the fisheye lens and a spacing of the fisheye lens from the first image sensor is modified to adjust an effective sensor FOV of the fisheye lens; and
a controller comprising a depth model, the controller configured to:
receive, as input, the first output signal; and
output, based at least in part on the first output signal, a depth measurement.
9. The device of claim 8 , wherein:
the controller comprises an application processor; and
the depth model is executed by the application processor.
10. The device of claim 8 , wherein:
the controller comprises an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA); and
the ASIC is configured to at least one of store or execute the depth model.
11. The device of claim 8 , wherein the first radiation is characterized by a first type of radiation comprising at least one of infrared light of near-infrared light.
12. The device of claim 8 , further comprising a second lens and a second sensor, wherein:
the second sensor is configured to generate a second output signal based at least in part on receiving second radiation via the second lens, the second radiation characterized by a second type of radiation different than a first type of radiation characterizing the first radiation.
13. The device of claim 8 , further comprising an illuminator configured to emit radiation of a first type, and wherein the first radiation comprises radiation that reflects from an object responsive to the radiation emitted by the illuminator.
14. The device of claim 8 , wherein a field of view associated with the first image sensor is less than or equal to 65 degrees.
15. A method comprising:
receiving, at a first image sensor, first radiation via a first fisheye lens characterized by a field of view (FOV);
receiving, at a second image sensor, second radiation via a second fisheye lens;
generating, by the first image sensor, a first output signal based at least in part on the first radiation, wherein the first image sensor crops the FOV of the fisheye lens so that only a portion of the first radiation in a central portion of the FOV of the fisheye lens remains, wherein a curvature of one or more lenses of the fisheye lens and a spacing of the fisheye lens from the first image sensor is modified to adjust an effective sensor FOV of the fisheye lens;
generating, by the second image sensor, a second output signal based at least in part on the second radiation; and
determining, based at least in part on the first output signal and the second output signal, a depth measurement.
16. The method of claim 15 , wherein determining the depth measurement is based at least in part on:
providing, as input to a depth model, the first output signal and the second signal or a representation thereof; and
receiving, from the depth model, the depth measurement.
17. The method of claim 15 , wherein determining the depth measurement is based at least in part on determining a disparity between the first output signal and the second output signal.
18. The method of claim 15 , wherein the first radiation and the second radiation comprise at least one of infrared light or near-infrared light.
19. The method of claim 15 , further comprising:
receiving, at a third image sensor, third radiation via a lens;
generating, by the third image sensor, a third output signal based at least in part on the third radiation; and
causing at least one of display or storage of an image, based at least in part on the third output signal.
20. The method of claim 15 , further comprising:
emitting, by an illuminator, radiation, wherein:
the first radiation and the second radiation are received based at least in part on reflection of the radiation by an object, and
the depth measurement includes a distance from the object to at least one of the first image sensor or the second image sensor.