Producing a depth map from two-dimensional images
A system for producing a depth map can include a processor and a memory. The memory can store a candidate depth production module and a depth map production module. The candidate depth production module can include instructions that cause the processor to: (1) identify, in a first image, an epipolar line associated with a pixel in a second image and (2) sample, from a first image feature set, a set of candidate depths for pixels along the epipolar line. The depth map production module can include instructions that cause the processor to: (1) determine a similarity measure between a feature, from a second image feature set, and a member of the set and (2) produce, from the second image, the depth map with a depth for the pixel being a depth associated with a member, of the set, having a greatest similarity measure.
1 . A system, comprising:
a processor; and
a memory storing:
a candidate depth production module including instructions that, when executed by the processor, cause the processor to:
identify, in a first image, an epipolar line associated with a pixel in a second image; and
sample, from a first image feature set, a set of candidate depths for pixels along the epipolar line, a relationship between a first distance, of a first pair of adjacent members of the set, and a second distance, of a second pair of adjacent members of the set, being a logarithmic relationship; and
a depth map production module including instructions that, when executed by the processor, cause the processor to:
determine a similarity measure between a feature, from a second image feature set, and a member of the set; and
produce, from the second image, a depth map with a depth for the pixel being a depth associated with a member, of the set, having a greatest similarity measure.
2 . The system of claim 1 , wherein:
the second image is of a scene,
the pixel, in the second image, represents at least a portion of an object in the scene, and
the depth map production module further includes instructions to determine, using the depth for the pixel, a distance to the object.
3 . The system of claim 1 , further comprising a camera configured to:
produce the first image; and
produce the second image.
4 . The system of claim 3 , wherein the camera comprises a stereo camera.
5 . The system of claim 1 , wherein:
at least one of the first image or the second image is defined by:
a height measured by a first count of pixels,
a width measured by a second count of pixels, and
a count of color components represented by the pixel,
at least one of the first image feature set or the second image feature set is defined by:
a height measured by a third count of pixels,
a width measured by a fourth count of pixels, and
a count of features associated with the pixel, and
at least one of:
the third count of pixels is less than the first count of pixels, or
the fourth count of pixels is less than the second count of pixels.
6 . The system of claim 1 , wherein:
the set of candidate depths is defined by:
a range of candidate depths from a minimum candidate depth to a maximum candidate depth, and
a count of members of the set of candidate depths,
the members of the set of candidate depths include a first member, a second member, and a third member, the third member being directly adjacent to the second member, and the second member being directly adjacent to the first member, and
the relationship between the first distance and the second distance is the logarithmic relationship, the first distance being a distance between the first member and the second member, the second distance being a distance between the second member and the third member.
7 . The system of claim 1 , wherein the depth map production module further includes instructions to:
produce, from the first image, the first image feature set; and
produce, from the second image, the second image feature set.
8 . The system of claim 7 , wherein the instructions to determine the similarity measure include instructions to determine, using a first attention technique, a first set of similarity measures, the first set of similarity measures being between the feature, from the second image feature set, and members of the set of candidate depths.
9 . The system of claim 8 , wherein the instructions to determine the similarity measure further include instructions to determine the first set of similarity measures between features, from the second image feature set, and the members of the set of candidate depths.
10 . The system of claim 9 , wherein:
the instructions to determine the similarity measure further include instructions to divide the features into groups of features, and
the first set of similarity measures comprises, for each group of the groups of features, a corresponding similarity measure.
11 . The system of claim 10 , wherein:
the instructions to determine the similarity measure further include instructions to normalize the corresponding similarity measure to produce a matching probability, and
the greatest similarity measure comprises a greatest matching probability.
12 . The system of claim 8 , wherein instructions to determine the similarity measure further include instructions to determine, using a second attention technique, a second set of similarity measures, the second set of similarity measures being between one member, of the members of the set of candidate depths, and other members of the members of the set of candidate depths.
13 . The system of claim 12 , wherein:
the instructions to determine the similarity measure are performed in iterations, an iteration, of the iterations, comprising an operation of the instructions to determine, using the first attention technique, the first set of similarity measures followed by an operation of the instructions to determine, using the second attention technique, the second set of similarity measures, and
the instructions to determine the similarity measure further include an operation of a final iteration, the final iteration being an operation of the instructions to determine, using the first attention technique, the first set of similarity measures.
14 . The system of claim 8 , wherein the instructions to determine the similarity measure further include instructions to produce, from the first set of similarity measures, a cost volume.
15 . The system of claim 8 , wherein:
the first set of similarity measures corresponds to a set of matching probabilities, and
the instructions to determine the similarity measure further include instructions to:
determine, from the set of matching probabilities, a most probable member of the set of candidate depths, the most probable member being associated with a highest value of a matching probability among values of the matching probabilities in the set of matching probabilities; and
perform a normalization operation, with a window function centered on the most probable member, to determine the depth for the pixel.
16 . The system of claim 15 , wherein:
the system includes a neural network,
the neural network is associated with the instructions to identify the epipolar line, the instructions to produce the first image feature set, the instructions to produce the second image feature set, the instructions to sample the set of candidate depths, the instructions to determine the similarity measure, the instructions to determine the most probable member, the instructions to perform the normalization operation, and the instructions to produce the depth map,
the neural network is trainable in a self-supervised manner based on a difference between values of pixels in the second image and values of pixels in a third image,
the third image is produced by a projection of values of the first image onto a copy of the second image, and
the projection is based on predictions of depths associated with pixels in the copy of the second image.
17 . A method, comprising:
identifying, by a processor, in a first image, an epipolar line associated with a pixel in a second image;
sampling, by the processor and from a first image feature set, a set of candidate depths for pixels along the epipolar line, a relationship between a first distance, of a first pair of adjacent members of the set, and a second distance, of a second pair of adjacent members of the set, being a logarithmic relationship;
determining, by the processor, a similarity measure between a feature, from a second image feature set, and a member of the set; and
producing, by the processor and from the second image, a depth map with a depth for the pixel being a depth associated with a member, of the set, having a greatest similarity measure.
18 . The method of claim 17 , wherein:
the first image is of a scene, the second image is of the scene, and
a viewpoint of the second image is different from a viewpoint of the first image.
19 . The method of claim 17 , wherein:
the first image is a first frame in a sequence of frames,
the second image is a second frame in the sequence of frames, and
the second frame is separated from the first frame by a duration of time.
20 . A non-transitory computer-readable medium for producing a depth map, the non-transitory computer-readable medium including instructions that, when executed by one or more processors, cause the one or more processors to:
identify, in a first image, an epipolar line associated with a pixel in a second image;
sample, from a first image feature set, a set of candidate depths for pixels along the epipolar line, a relationship between a first distance, of a first pair of adjacent members of the set, and a second distance, of a second pair of adjacent members of the set, being a logarithmic relationship;
determine a similarity measure between a feature, from a second image feature set, and a member of the set; and
produce, from the second image, the depth map with a depth for the pixel being a depth associated with a member, of the set, having a greatest similarity measure.