Imager for detecting visual light and projected patterns
Methods and systems for depth sensing are provided. A system includes a first and second optical sensor each including a first plurality of photodetectors configured to capture visible light interspersed with a second plurality of photodetectors configured to capture infrared light within a particular infrared band. The system also includes a computing device configured to (i) identify first corresponding features of the environment between a first visible light image captured by the first optical sensor and a second visible light image captured by the second optical sensor; (ii) identify second corresponding features of the environment between a first infrared light image captured by the first optical sensor and a second infrared light image captured by the second optical sensor; and (iii) determine a depth estimate for at least one surface in the environment based on the first corresponding features and the second corresponding features.
1. An optical sensor comprising:
a planar array of photodetectors comprising a first plurality of photodetectors and a second plurality of photodetectors, wherein each photodetector is configured to generate a charge based on an intensity of light incident on the photodetector;
a light filter array coupled to the planar array of photodetectors comprising a first plurality of filters configured to pass visible light onto the first plurality of photodetectors and a second plurality of filters configured to pass infrared light within a particular infrared band onto the second plurality of photodetectors, wherein filters of the first plurality of filters are interspersed with filters of the second plurality of filters to form a plurality of mosaics, wherein each mosaic includes two filters of the first plurality of filters and two filters of the second plurality of filters arranged in a checkerboard configuration; and
a control circuit configured to:
generate a visible light image based on charges from the first plurality of photodetectors, wherein each pixel in the visible light image represents an average of charges generated by two photodetectors of the first plurality of photodetectors in a respective mosaic of the plurality of mosaics; and
generate an infrared light image based on charges from the second plurality of photodetectors, wherein each pixel in the infrared light image represents an average of charges generated by two photodetectors of the second plurality of photodetectors in a respective mosaic of the plurality of mosaics, and wherein the visible light image is separate from the infrared light image.
2. The optical sensor of claim 1 , wherein the control circuit is further configured to:
cause, over a first duration of time, the first plurality of photodetectors to capture visible light to generate the charges that the visible light image is based on; and
cause, over a second duration of time, the second plurality of photodetectors to capture infrared light to generate the charges that the infrared light image is based on,
wherein the second duration of time is greater than the first duration of time.
3. The optical sensor of claim 1 , wherein the control circuit is further configured to:
amplify the charges generated by the first plurality of photodetectors by a first gain value; and
amplify the charges generated by the second plurality of photodetectors by a second gain value, wherein the second gain value is greater than the first gain value.
4. The optical sensor of claim 1 , wherein the control circuit is further configured to transmit the visible light image and the infrared light image to one or more devices.
5. A system comprising:
a planar array of photodetectors comprising a first plurality of photodetectors configured to capture visible light and a second plurality of photodetectors configured to capture infrared light within a particular infrared band, wherein each photodetector is configured to generate a charge based on an intensity of light incident on the photodetector, and wherein the first plurality of photodetectors and the second plurality of photodetectors alternate along both rows and columns to form a checkerboard pattern;
a light filter array coupled to the planar array of photodetectors comprising a first plurality of filters configured to pass visible light onto the first plurality of photodetectors and a second plurality of filters configured to pass infrared light within a particular infrared band onto the second plurality of photodetectors, wherein the planar array of photodetectors and the light filter array are arranged to form a plurality of mosaics, wherein each mosaic includes two filters of the first plurality of filters and two filters of the second plurality of filters arranged according to the checkerboard pattern; and
a computing device configured to:
generate a visible light image based on charges from the first plurality of photodetectors, wherein each pixel in the visible light image represents an average of charges generated by two photodetectors of the first plurality of photodetectors in a respective mosaic of the plurality of mosaics; and
generate an infrared light image based on charges from the second plurality of photodetectors, wherein each pixel in the infrared light image represents an average of charges generated by two photodetectors of the second plurality of photodetectors in a respective mosaic of the plurality of mosaics, and wherein the visible light image is separate from the infrared light image.
6. The system of claim 5 , wherein the computing device is further configured to:
cause, over a first duration of time, the first plurality of photodetectors to capture visible light to generate the charges that the visible light image is based on; and
cause, over a second duration of time, the second plurality of photodetectors to capture infrared light to generate the charges that the infrared light image is based on,
wherein the second duration of time is greater than the first duration of time.
7. The system of claim 5 , wherein the computing device is further configured to:
amplify the charges generated by the first plurality of photodetectors by a first gain value; and
amplify the charges generated by the second plurality of photodetectors by a second gain value, wherein the second gain value is greater than the first gain value.
8. A method comprising:
receiving, at a computing system, a first set of charges generated based on respective intensities of light incident on a first plurality of photodetectors;
based on the first set of charges, generating a visible light image;
receiving, at the computing system, a second set of charges generated based on respective intensities of light incident on a second plurality of photodetectors, wherein the first plurality of photodetectors and the second plurality of photodetectors alternate along both rows and columns to form a checkerboard pattern arranged into a plurality of mosaics, wherein each mosaic includes two photodetectors of the first plurality of photodetectors and two photodetectors of the second plurality of photodetectors arranged according to the checkerboard pattern; and
based on the second set of charges, generating an infrared light image,
wherein each pixel in the visible light image represents an average of charges generated by two photodetectors of the first plurality of photodetectors in a respective mosaic of the plurality of mosaics,
wherein each pixel in the infrared light image represents an average of charges generated by two photodetectors of the second plurality of photodetectors in a respective mosaic of the plurality of mosaics, and wherein the visible light image is separate from the infrared light image.
9. The method of claim 8 , further comprising:
causing, over a first duration of time, the first plurality of photodetectors to capture visible light to generate the first set of charges; and
causing, over a second duration of time, the second plurality of photodetectors to capture infrared light to generate the second set of charges, wherein the second duration of time is greater than the first duration of time.
10. The method of claim 8 , further comprising:
amplifying the first set of charges generated by the first plurality of photodetectors by a first gain value; and
amplifying the second set of charges generated by the second plurality of photodetectors by a second gain value, wherein the second gain value is greater than the first gain value.
11. The method of claim 8 , wherein the first set of charges and the second set of charges are received simultaneously.
12. The method of claim 8 , wherein the first set of charges are received at a first time and the second set of charges are received at a second time, wherein the second time is subsequent to the first time.