Producing a depth map from a monocular two-dimensional image
A system for producing a depth map can include a processor and a memory. The memory can store a neural network. The neural network can include an encoding portion module, a multi-frame feature matching portion module, and a decoding portion module. The encoding portion module can include instructions that, when executed by the processor, cause the processor to encode an image to produce single-frame features. The multi-frame feature matching portion module can include instructions that, when executed by the processor, cause the processor to process the single-frame features to produce information. The decoding portion module can include instructions that, when executed by the processor, cause the processor to decode the information to produce the depth map. A first training dataset, used to train the multi-frame feature matching portion module, can be different from a second training dataset used to train the encoding portion module and the decoding portion module.
1. A system, comprising:
a first processor; and
a first memory storing:
a neural network, the neural network including:
a first encoding portion module including instructions that, when executed by the first processor, cause the first processor to encode an image to produce single-frame features;
a multi-frame feature matching portion module including instructions that, when executed by the first processor, cause the first processor to process the single-frame features to produce information; and
a decoding portion module including instructions that, when executed by the first processor, cause the first processor to decode the information to produce a depth map, wherein a first training dataset, used to train the multi-frame feature matching portion module, is different from a second training dataset used to train the encoding portion module and the decoding portion module.
2. The system of claim 1 , wherein:
the image is of a scene that includes an object, and
the first memory further stores a distance determination module including instructions that, when executed by the first processor, cause the first processor to determine, using the depth map, a distance to the object.
3. The system of claim 1 , wherein:
the information comprises an input information and an output information,
the instructions to process the single-frame features to produce the information include instructions to process the single-frame features to produce the input information,
the instructions to decode the information to produce the depth map include instructions to decode the output information to produce the depth map, and
the neural network further includes a second encoding portion module including instructions that, when executed by the first processor, cause the first processor to encode the input information to produce the output information.
4. The system of claim 1 , wherein the multi-frame feature matching portion module includes a cost volume of similarity measures between features of a first training image and features of a second training image, the first training image being a first frame in a sequence of frames, and the second training image being a second frame in the sequence of frames.
5. The system of claim 4 , wherein for a pixel of the first training image, a similarity measure, of the similarity measures, is determined using a first attention technique with respect to features, associated with a corresponding pixel of the second training image, and features associated with a set of candidate depths determined from the first training image.
6. The system of claim 5 , wherein the set of candidate depths is associated with pixels, in the first training image, along an epipolar line associated with the corresponding pixel in the second training image.
7. The system of claim 6 , wherein:
members of the set of candidate depths include a first member, a second member, and a third member, the third member being directly adjacent to the second member, and the second member being directly adjacent to the first member, and
a relationship between a first distance and a second distance is a logarithmic relationship, a first distance being a distance between the first member and the second member, the second distance being a distance between the second member and the third member.
8. The system of claim 5 , wherein for the pixel of the first training image, the similarity measure, of the similarity measures, is further determined using a second attention technique with respect to the features, associated with one member of the set of candidate depths, and the features associated with other members of the set of candidate depths.
9. The system of claim 1 , further comprising:
a second processor; and
a second memory storing a first training operation module including instructions that, when executed by the second processor, cause the second processor to cause a first geometry-based task training operation to be performed on the multi-frame feature matching portion module.
10. The system of claim 9 , wherein:
the second processor is the first processor; and
the second memory is the first memory.
11. The system of claim 9 , wherein the first memory further stores an installation module including instructions that, when executed by the first processor, cause the first processor to cause, after a completion of the first geometry-based task training operation, the multi-frame feature matching portion module to be included in the neural network.
12. The system of claim 11 , further comprising:
a third processor; and
a third memory storing a second training operation module including instructions that, when executed by the third processor, cause the third processor to cause, using the second training dataset, an image-based task training operation to be performed on the first encoding portion module and the decoding portion module.
13. The system of claim 12 , wherein:
the third processor is the first processor; and
the third memory is the first memory.
14. The system of claim 12 , wherein the instructions to cause the image-based task training operation to be performed on the first encoding portion module and the decoding portion module include instructions to cause, after the multi-frame feature matching portion module has been caused to be included in the neural network, the image-based task training operation to be performed on the first encoding portion module and the decoding portion module.
15. The system of claim 12 , wherein the instructions to cause the image-based task training operation to be performed on the first encoding portion module and the decoding portion module include instructions to cause, in a manner that causes weights associated with the multi-frame feature matching portion module to remain unchanged, the image-based task training operation to be performed on the first encoding portion module and the decoding portion module.
16. The system of claim 12 , wherein the second training operation module further includes instructions that, when executed by the third processor, cause the third processor to cause, using the second training dataset, a second geometry-based task training operation to be performed on the multi-frame feature matching portion module.
17. The system of claim 16 , wherein the second geometry-based task training operation is performed concurrently with the image-based task training operation.
18. A method, comprising:
encoding, by a processor operating an encoding portion of a neural network, an image to produce single-frame features;
processing, by the processor operating a multi-frame feature matching portion of the neural network, the single-frame features to produce information; and
decoding, by the processor operating a decoding portion of the neural network, the information to produce a depth map, wherein a first training dataset, used to train the multi-frame feature matching portion, is different from a second training dataset used to train the encoding portion and the decoding portion.
19. The method of claim 18 , wherein the depth map comprises a set of depth maps, a scale of a first depth map, in the set of depth maps, being different from a scale of a second depth map in the set of depth maps.
20. A non-transitory computer-readable medium for producing a depth map, the non-transitory computer-readable medium including instructions that, when executed by one or more processors, cause the one or more processors to:
encode, operating an encoding portion of a neural network, an image to produce single-frame features;
process, operating a multi-frame feature matching portion of the neural network, the single-frame features to produce information; and
decode, operating a decoding portion of the neural network, the information to produce the depth map, wherein a first training dataset, used to train the multi-frame feature matching portion, is different from a second training dataset used to train the encoding portion and the decoding portion.