Granted Patent B2

US 9,392,262 · App. 14/641,092 · Granted Jul 12, 2016

Inventors: Carlo Dal Mutto (Mountain View, CA); Abbas Rafii (Palo Alto, CA); David Demirdjian (Arlington, VA)

Assignee: Aquifi, Inc.

H04N13/0239G06K9/2036G06K9/46G06T7/0051G06T7/0073G06T7/0075H04N5/30H04N5/33H04N9/045G06K2009/4666G06K2209/401G06T2207/10048

Abstract

Claims (96)

1. A system for three-dimensional reconstruction of a scene, the system comprising:

a first camera comprising a first lens and a first multi-channel image sensor comprising a plurality of visible light detecting pixels and a plurality of invisible light detecting pixels;

a second camera comprising a second lens and a second multi-channel image sensor comprising a plurality of visible light detecting pixels and a plurality of invisible light detecting pixels;

a projection source;

a processor; and

a memory storing instructions that, when executed by the processor, cause the processor to:

control the projection source to emit invisible light toward the scene;

control the first multi-channel image sensor to capture a first image, the first image comprising a first invisible light channel and a first visible light channel;

control the second multi-channel image sensor to capture a second image substantially simultaneously as the capture of the first image by the first multi-channel image sensor, the second image comprising a second invisible light channel and a second visible light channel;

perform stereo analysis of the first image and the second image in accordance with the first and second invisible light channels and the first and second visible light channels to generate a depth map by:

performing stereo analysis of the first invisible light channel and the second invisible light channel to generate a coarse depth map; and

applying a cross-bilateral filter on the coarse depth map with weights from the first visible light channel and the second visible light channel to generate the depth map; and

output the depth map.

2. The system of claim 1 , wherein the memory further stores instructions that, when executed by the processor, cause the processor to perform stereo analysis of the first image and the second image by:

up-sampling the coarse depth map to a resolution of the first and second visible light channels.

3. The system of claim 1 , wherein the projection source is configured to emit infrared light, and

wherein the invisible light detecting pixels are configured to detect infrared light.

4. The system of claim 1 , wherein the first visible light channel comprises red, blue, and green color channels, and

wherein the second visible light channel comprises red, blue, and green color channels.

5. The system of claim 1 , wherein the projection source is configured to emit the invisible light in a pattern.

6. A system for three-dimensional reconstruction of a scene, the system comprising:

a first camera comprising a first lens and a first multi-channel image sensor comprising a plurality of visible light detecting pixels and a plurality of invisible light detecting pixels;

a second camera comprising a second lens and a second multi-channel image sensor comprising a plurality of visible light detecting pixels and a plurality of invisible light detecting pixels;

a projection source;

a processor; and

a memory storing instructions that, when executed by the processor, cause the processor to:

control the first multi-channel image sensor to capture an initial image before controlling the projection source to emit invisible light toward the scene;

identify a region of interest in the initial image;

control the projection source to emit invisible light toward the scene;

control the first multi-channel image sensor to capture a first image, the first image comprising a first invisible light channel and a first visible light channel;

control the second multi-channel image sensor to capture a second image substantially simultaneously as the capture of the first image by the first multi-channel image sensor, the second image comprising a second invisible light channel and a second visible light channel;

perform stereo analysis of the first image and the second image in accordance with the first and second invisible light channels and the first and second visible light channels to generate a depth map; and

output the depth map.

7. The system of claim 6 , wherein the memory further stores instructions that, when executed by the processor, cause the processor to:

detect a feature within the region of interest;

identify a minimum disparity bound in accordance with the feature; and

perform the stereo analysis in accordance with the minimum disparity bound and only within the region of interest.

8. The system of claim 7 , wherein the memory further stores instructions that, when executed by the processor, cause the processor to:

identify a maximum disparity bound in accordance with the feature; and

perform the stereo analysis in accordance with the maximum disparity bound.

9. The system of claim 7 , wherein the projection source comprises a scanner mirror and a light source configured to emit the invisible light, and

wherein the processor is configured to control the scanner mirror to direct the invisible light emitted by the light source toward only a portion of the scene corresponding to the region of interest.

10. A system for three-dimensional reconstruction of a scene, the system comprising:

a first camera comprising a first lens and a first multi-channel image sensor comprising a plurality of visible light detecting pixels and a plurality of invisible light detecting pixels;

a second camera comprising a second lens and a second multi-channel image sensor comprising a plurality of visible light detecting pixels and a plurality of invisible light detecting pixels;

a projection source;

a processor; and

a memory storing instructions that, when executed by the processor, cause the processor to:

control the projection source to emit invisible light toward the scene;

control the first multi-channel image sensor to capture a first image, the first image comprising a first invisible light channel and a first visible light channel;

control the second multi-channel image sensor to capture a second image substantially simultaneously as the capture of the first image by the first multi-channel image sensor, the second image comprising a second invisible light channel and a second visible light channel;

perform stereo analysis of the first image and the second image in accordance with the first and second invisible light channels and the first and second visible light channels to generate a depth map by:

performing stereo analysis of the first and second invisible light channels to generate an invisible light depth map;

performing stereo analysis of the first and second visible light channels to generate a visible light depth map; and

combining the invisible light depth map and the visible light depth map to generate the depth map; and

output the depth map.

11. A method for three-dimensional reconstruction of a scene, the method comprising:

controlling, by a processor, a projection source to emit invisible light toward the scene:

controlling a first multi-channel image sensor to capture a first image, the first multi-channel image sensor comprising a first camera comprising a first lens and a plurality of visible light detecting pixels and a plurality of invisible light detecting pixels, the first image comprising a first invisible light channel and a first visible light channel;

controlling a second multi-channel image sensor to capture a second image substantially simultaneously as the capture of the first image by the first multi-channel image sensor, the second multi-channel image sensor comprising a second camera comprising a second lens and a plurality of visible light detecting pixels and a plurality of invisible light detecting pixels, the second image comprising a second invisible light channel and a second visible light channel;

performing, by the processor, stereo analysis of the first image and the second image in accordance with the first and second invisible light channels and the first and second visible light channels to generate a depth map, the stereo analysis of the first image and the second image comprising:

performing, by the processor, stereo analysis of the first invisible light channel and the second invisible light channel to generate a coarse depth map; and

applying, by the processor, a cross-bilateral filter on the coarse depth map with weights from the first visible light channel and the second visible light channel to generate the depth map; and

outputting, by the processor, the depth map.

12. The method of claim 11 , wherein the performing stereo analysis of the first image and the second image further comprises up-sampling the coarse depth map to a resolution of the first and second visible light channels.

13. The method of claim 11 , wherein the projection source is configured to emit infrared light, and

wherein the invisible light detecting pixels are configured to detect infrared light.

14. The method of claim 11 , wherein the first visible light channel comprises red, blue, and green color channels, and

wherein the second visible light channel comprises red, blue, and green color channels.

15. The method of claim 11 , wherein the projection source is configured to emit the invisible light in a pattern.

16. A method for three-dimensional reconstruction of a scene, the method comprising:

controlling the first multi-channel image sensor to capture an initial image before controlling the projection source to emit invisible light toward the scene;

identifying a region of interest in the initial image;

controlling, by a processor, a projection source to emit invisible light toward the scene;

controlling a first multi-channel image sensor to capture a first image, the first multi-channel image sensor comprising a first camera comprising a first lens and a plurality of visible light detecting pixels and a plurality of invisible light detecting pixels, the first image comprising a first invisible light channel and a first visible light channel;

controlling a second multi-channel image sensor to capture a second image substantially simultaneously as the capture of the first image by the first multi-channel image sensor, the second multi-channel image sensor comprising a second camera comprising a second lens and a plurality of visible light detecting pixels and a plurality of invisible light detecting pixels, the second image comprising a second invisible light channel and a second visible light channel;

performing, by the processor, stereo analysis of the first image and the second image in accordance with the first and second invisible light channels and the first and second visible light channels to generate a depth map and

outputting, by the processor, the depth map.

17. The method of claim 16 , further comprising:

detecting, by the processor, a feature within the region of interest;

identifying, by the processor, a minimum disparity bound in accordance with the feature; and

performing the stereo analysis in accordance with the minimum disparity bound and only within the region of interest.

18. The method of claim 17 , further comprising:

identifying a maximum disparity bound in accordance with the feature; and

performing the stereo analysis in accordance with the maximum disparity bound.

19. The method of claim 17 , wherein the projection source comprises a scanner mirror and a light source configured to emit the invisible light, and

wherein the method further comprises controlling the scanner mirror to direct the invisible light emitted by the light source toward only a portion of the scene corresponding to the region of interest.

20. A method for three-dimensional reconstruction of a scene, the method comprising:

controlling, by a processor, a projection source to emit invisible light toward the scene;

controlling a first multi-channel image sensor to capture a first image, the first multi-channel image sensor comprising a first camera comprising a first lens and a plurality of visible light detecting pixels and a plurality of invisible light detecting pixels, the first image comprising a first invisible light channel and a first visible light channel;

controlling a second multi-channel image sensor to capture a second image substantially simultaneously as the capture of the first image by the first multi-channel image sensor, the second multi-channel image sensor comprising a second camera comprising a second lens and a plurality of visible light detecting pixels and a plurality of invisible light detecting pixels, the second image comprising a second invisible light channel and a second visible light channel;

performing, by the processor, stereo analysis of the first image and the second image in accordance with the first and second invisible light channels and the first and second visible light channels to generate a depth map, the stereo analysis of the first image and the second image comprising:

performing stereo analysis of the first and second invisible light channels to generate an invisible light depth map;

performing stereo analysis of the first and second visible light channels to generate a visible light depth map; and

combining the invisible light depth map and the visible light depth map to generate the depth map; and

outputting, by the processor, the depth map.

Assignments (8)

Continuity (3)