Methods and apparatus for enhancing depth maps with polarization cues
A 3D imaging system uses a depth sensor to produce a coarse depth map, and then uses the coarse depth map as a constraint in order to correct ambiguous surface normals computed from polarization cues. The imaging system outputs an enhanced depth map that has a greater depth resolution than the coarse depth map. The enhanced depth map is also much more accurate than could be obtained from the depth sensor alone. In many cases, the imaging system extracts the polarization cues from three polarized images. Thus, in many implementations, the system takes only three extra images—in addition to data used to generate the coarse depth map—in order to dramatically enhance the coarse depth map.
1. A method comprising:
(a) capturing a first set of images of an object, which first set of images comprises one or more images;
(b) capturing a second set of images of the object, which second set includes a first polarized image, a second polarized image and a third polarized image, in such a way that the first, second and third polarized images capture light that has been filtered at a first polarizer angle, a second polarizer angle and a third polarizer angle, respectively, the first, second and third polarizer angles being different from each other;
(c) calculating, based on the first set of images, a first depth map of a surface of the object;
(d) calculating, based on the second set of images, polarization surface normals of the surface; and
(e) performing a computation that (i) is based on the first depth map and the polarization surface normals, (ii) disregards gradients in a first region of the surface that has a lower degree of polarization than the rest of the surface, and (iii) calculates a second depth map of the surface, which second depth map is more accurate than the first depth map in part because the gradients in the first region of the surface are disregarded.
2. The method of claim 1 , wherein the calculating of the polarization surface normals takes into account refractive indices.
3. The method of claim 1 , wherein the calculating of the polarization surface normals takes into account behavior of light moving between media of different refractive indices.
4. The method of claim 1 , wherein the first and second set of images are not identical sets.
5. The method of claim 1 , wherein, in the calculating of the second depth map, the first depth map is a constraint on polarization normal data regarding the surface.
6. Apparatus comprising
(a) one or more cameras configured
(i) to capture a first set of images of an object, which first set of images comprises one or more images, and
(ii) to capture a second set of images of the object, which second set includes a first polarized image, a second polarized image and a third polarized image, such that the first, second and third polarized images capture light that has been filtered at a first polarizer angle, a second polarizer angle and a third polarizer angle, respectively, the first, second and third polarizer angles being different from each other; and
(b) one or more computers that are programmed
(i) to calculate, based on the first set of images, a first depth map of a surface of the object,
(ii) to calculate, based on the second set of images, polarization surface normals of the surface, and
(iii) to perform a computation that (A) is based on the first depth map and the polarization surface normals, (B) disregards gradients in a first region of the surface that has a lower degree of polarization than the rest of the surface, and (C) calculates a second depth map of the surface, which second depth map is more accurate than the first depth map in part because the gradients in the first region of the surface are disregarded.
7. The apparatus of claim 6 , wherein the one or more computers are programmed to calculate the polarization surface normals in a manner that takes into account refractive indices.
8. The apparatus of claim 6 , wherein the one or more computers are programmed to calculate the polarization surface normals in a manner that takes into account behavior of light moving between media of different refractive indices.
9. The apparatus of claim 6 , wherein the one or more computers are programmed to calculate the second depth map in such a way that the first depth map is a constraint on polarization normal data regarding the surface.