Ultrafast, robust and efficient depth estimation for structured-light based 3D camera system
A system and a method are disclosed for a structured-light system to estimate depth in an image. An image is received in which the image is of a scene onto which a reference light pattern has been projected. The projection of the reference light pattern includes a predetermined number of particular sub-patterns. A patch of the received image and a sub-pattern of the reference light pattern are matched based on either a hardcode template matching technique or a probability that the patch corresponds to the sub-pattern. If a lookup table is used, the table may be a probability matrix, may contain precomputed correlations scores or may contain precomputed class IDs. An estimate of depth of the patch is determined based on a disparity between the patch and the sub-pattern.
1. A method for a structured-light system to estimate depth in an image, the method comprising:
receiving the image, the image being of a scene onto which a reference light pattern has been projected, the image including a projection of the reference light pattern, and the reference light pattern comprising a predetermined number of particular sub-patterns;
matching a patch of the image and a sub-pattern of the reference light pattern based on a probability that the patch corresponds to the sub-pattern, the patch comprising a predetermined number of pixels, and the probability being contained in a lookup table that comprises a probability matrix, matching the patch and the sub-pattern further comprises:
binarizing the pixels comprising the patch;
forming a vector from the pixels; and
determining a class of the patch by multiplying the vector of the pixels by the probability matrix to form a histogram representing probabilities that the patch is a particular sub-pattern of the reference light pattern; and
determining an estimate of depth of the patch based on a disparity between the patch and the sub-pattern.
2. The method of claim 1 , wherein the predetermined number of pixels comprising the patch is 16.
3. The method of claim 1 , wherein the probability matrix is formed based on Bayes' rule, by linear optimization or by a neural network.
4. A method for a structured-light system to estimate depth in an image, the method comprising:
receiving the image, the image being of a scene onto which a reference light pattern has been projected, the image including a projection of the reference light pattern, and the reference light pattern comprising a predetermined number of particular sub-patterns;
binarizing at least one patch of the image, the patch comprising a predetermined number of pixels;
matching the binarized patch and a sub-pattern of the reference light pattern based on a probability that the binarized patch corresponds to the sub-pattern, the probability that the binarized patch corresponds to the sub-pattern being contained in a lookup table that comprises a probability matrix, matching the binarized patch and the sub-pattern further comprises:
forming a vector from the binarized patch; and
determining a class of the patch by multiplying the vector of the binarized patch by the probability matrix to form a histogram representing probabilities that the patch is a particular sub-pattern of the reference light pattern; and
determining an estimate of depth of the at least one patch based on a disparity between the patch and the sub-pattern.
5. The method of claim 4 , wherein the probability matrix is formed based on Bayes' rule, by linear optimization or by a neural network.
6. The method of claim 4 , wherein matching the binarized patch and the sub-pattern further comprises using the binarized patch to access an entry in the lookup table that represents the probability that the patch corresponds to the sub-pattern.
7. The method of claim 6 , wherein each entry of the lookup table represents a correlation score of the probability the patch corresponds to each of the respective sub-patterns.
8. The method of claim 6 , wherein each entry of the lookup table represents a class identification of each respective sub-pattern.
9. A method for a structured-light system to estimate depth in an image, the method comprising:
receiving the image, the image being of a scene onto which a reference light pattern has been projected, the image including a projection of the reference light pattern, and the reference light pattern comprising a predetermined number of particular sub-patterns;
dividing the image into a plurality of patches by a predetermined window that slides over the image to form each individual patch of the plurality of patches, each patch comprising a predetermined number of pixels;
binarizing pixels comprising each patch of the plurality of patches;
using the binarized piels of each patch to access an entry in a lookup table to obtain a sub-pattern identification for the patch;
generating a probability histogram for the image using a voting process based on the sub-pattern identifications obtained for the patches of the plurality of patches; and
determining an estimate of depth of the patch based on a disparity between the patch and a sub-pattern selected from the probability histogram generated by the voting process.