Selective processing or readout of data from one or more imaging sensors included in a depth camera assembly
A depth camera assembly (DCA) determines depth information within a local area. The DCA may selectively process a subset of data captured by an imaging sensor and obtained from the imaging sensor, such as pixels corresponding to a region of interest, for depth information. Alternatively, the DCA may limit retrieval of data from the imaging sensor to pixels corresponding to the region of interest from the imaging sensor for processing for depth information. The depth processing may include a semi-global match (SGM) algorithm, and the DCA adjusts a number of neighboring pixels used for determining depth information for a specific pixel based on one or more criteria. In some embodiments, the DCA performs the depth processing by analyzing images from different image sensors using left to right and right to left correspondence checks that are performed in parallel.
1. A depth camera assembly comprising:
a plurality of imaging sensors, each sensor configured to capture images of a local area and each imaging sensor comprising an array of pixels; and
a controller coupled to each of the plurality of image sensors, the controller configured to:
select a region of interest in an image captured by the plurality of image sensors based on prior information describing the local area,
identify a subset of pixels of an imaging sensor corresponding to the selected region of interest in the image, and
generate depth information for the local area by differently processing values obtained from the subset of pixels than from values obtained from other pixels of the imaging sensor that are not in the subset of pixels by:
retrieving data from each pixel of the sensor; and
applying a stereo imaging process to data retrieved from pixels in the subset of pixels and differently applying the stereo imaging process to data retrieved from pixels not in the subset of pixels.
2. The depth camera assembly of claim 1 , wherein applying the stereo imaging process to data retrieved from pixels in the subset of pixels and differently applying the stereo imaging process to data retrieved from pixels not in the subset of pixels comprises:
applying an outlier rejection process to depth information generated from data retrieved from pixels in the subset of pixels, while applying an alternative depth outlier rejection process to depth information generated from data retrieved from pixels not in the subset of pixels.
3. The depth camera assembly of claim 2 , wherein the outlier rejection process comprises both a left to right disparity search and a right to left disparity search, and the alternative outlier rejection process comprises one selected from a group consisting of: the left to right disparity search and the right to left disparity search.
4. The depth camera assembly of claim 1 , wherein select the region of interest in the image captured by the plurality of image sensors based on prior information describing the local area comprises:
receive information describing the local area from a mapping server; and
select the region of interest as a region of the image for which the information received from the mapping server has a specific characteristic.
5. The depth camera assembly of claim 1 , wherein generate depth information for the local area by differently processing values obtained from the subset of pixels than from values obtained from other pixels of the imaging sensor that are not in the subset of pixels further comprises:
propagate depth information from pixels neighboring a pixel of the subset of pixels to the pixel of the subset of pixels differently than propagating depth information from pixels neighboring an additional pixel that is not in the subset of pixels to the additional pixel.
6. The depth camera assembly of claim 5 , wherein propagate depth information from pixels neighboring the pixel of the subset of pixels to the pixel of the subset of pixels differently than propagating depth information from pixels neighboring the additional pixel that is not in the subset of pixels to the additional pixel comprises:
decrease a number of pixels neighboring the additional pixel for which depth information is propagated to the additional pixel relative to a number of pixels neighboring the pixel for which depth information is propagated to the pixel.
7. The depth camera assembly of claim 1 , wherein the controller is further configured to:
perform a plurality of disparity searches in parallel between disparity check between the image and an additional image.
8. The depth camera assembly of claim 7 , wherein perform the plurality of disparity searches in parallel between disparity check between the image and the additional image in parallel comprises:
perform a left to right disparity search between the image and the additional image; and
perform a right to left disparity search between the additional image and the image in parallel with the left to right disparity search.
9. The depth camera assembly of claim 1 , further comprising:
a light projector coupled to the controller, the light projector configured to project a structure light pattern including a plurality of structured light elements into the local area.
10. The depth camera assembly of claim 9 , wherein identify the subset of pixels of the imaging sensor corresponding to the selected region of interest in the image comprises:
identify the region of interest as a region within the local area that the controller determines has at least a threshold likelihood of being illuminated by a light projector.
11. A computer program product comprising a non-transitory computer-readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to:
capture images of a local area using a plurality of imaging sensors, each imaging sensor comprising an array of pixels;
select a region of interest in an image captured by the plurality of image sensors based on prior information describing the local area,
identify a subset of pixels of an imaging sensor corresponding to the selected region of interest in the image, and
generate depth information for the local area by differently processing values obtained from the subset of pixels than from values obtained from other pixels of the imaging sensor that are not in the subset of pixels by:
retrieving data from each pixel of the sensor; and
applying a stereo imaging process to data retrieved from pixels in the subset of pixels and differently applying the stereo imaging process to data retrieved from pixels not in the subset of pixels.
12. The computer program product of claim 11 , wherein select the region of interest in the image captured by the plurality of image sensors based on prior information describing the local area comprises:
receive information describing the local area from a mapping server; and
select the region of interest as a region of the image for which the information received from the mapping server has a specific characteristic.
13. The computer program product of claim 11 , wherein generate depth information for the local area by differently processing values obtained from the subset of pixels than from values obtained from other pixels of the imaging sensor that are not in the subset of pixels comprises:
propagate depth information from pixels neighboring a pixel of the subset of pixels to the pixel of the subset of pixels differently than propagating depth information from pixels neighboring an additional pixel that is not in the subset of pixels to the additional pixel.
14. The computer program product of claim 11 , wherein the non-transitory computer readable storage medium further has instructions encoded thereon that, when executed by the processor, cause the processor to:
perform a plurality of disparity searches in parallel between disparity check between the image and an additional image.