Super-resolution depth map generation for multi-camera or other environments

1. A method comprising:

obtaining, using at least one processor, first and second input image frames, the first and second input image frames associated with first and second image planes, respectively;

obtaining, using the at least one processor, a depth map associated with the first input image frame; and

producing another version of the depth map by performing one or more times:

converting the depth map from a coordinate frame associated with the first image plane to a coordinate frame associated with the second image plane;

projecting, using the at least one processor, the first input image frame to the second image plane in order to produce a projected image frame using (i) the converted depth map and (ii) information identifying a conversion from the first image plane to the second image plane; and

adjusting, using the at least one processor, at least one of the depth map and the information identifying the conversion from the first image plane to the second image plane to reduce a difference between the projected image frame and the second input image frame.

2. The method of claim 1 , wherein the information identifying the conversion from the first image plane to the second image plane is used to convert the depth map from the coordinate frame associated with the first image plane to the coordinate frame associated with the second image plane.

3. The method of claim 1 , wherein adjusting at least one of the depth map and the information comprises:

minimizing a loss function that is based on the difference between the projected image frame and the second input image frame; and

adjusting at least one of the depth map and the information based on the minimization of the loss function.

4. The method of claim 3 , wherein:

the information identifying the conversion from the first image plane to the second image plane comprises rotation and translation values; and

the loss function is based on a combination of (i) a measure of photometric loss between the second input image frame and the projected image frame, (ii) a measure of total variance in the depth map, and (iii) a measure of errors between edges identified in the first input image frame and edges identified in the depth map.

5. The method of claim 1 , further comprising:

generating the depth map associated with the first input image frame, the depth map comprising a higher-resolution depth map generated by upsampling a lower-resolution depth map.

6. The method of claim 5 , wherein generating the depth map comprises:

applying multiple machine-learning convolutional layers to at least one of the first and second input image frames in order to produce the lower-resolution depth map; and

applying a machine-learning upsampling layer to the lower-resolution depth map in order to generate the higher-resolution depth map.

7. The method of claim 6 , wherein adjusting at least one of the depth map and the information comprises:

applying multiple machine-learning guided depth super-resolution layers to the higher-resolution depth map, each machine-learning guided depth super-resolution layer projecting the first input image frame to the second image plane and adjusting at least one of the depth map and the information, a last of the machine-learning guided depth super-resolution layers outputting the other version of the depth map.

8. An apparatus comprising:

at least one memory configured to store first and second input image frames, the first and second input image frames associated with first and second image planes, respectively; and

at least one processor configured to:

obtain a depth map associated with the first input image frame; and

produce another version of the depth map;

wherein, to produce the other version of the depth map, the at least one processor is configured to, one or more times:

convert the depth map from a coordinate frame associated with the first image plane to a coordinate frame associated with the second image plane;

project the first input image frame to the second image plane in order to produce a projected image frame using (i) the converted depth map and (ii) information identifying a conversion from the first image plane to the second image plane; and

adjust at least one of the depth map and the information identifying the conversion from the first image plane to the second image plane to reduce a difference between the projected image frame and the second input image frame.

9. The apparatus of claim 8 , wherein the information identifying the conversion from the first image plane to the second image plane is used to convert the depth map from the coordinate frame associated with the first image plane to the coordinate frame associated with the second image plane.

10. The apparatus of claim 8 , wherein, to adjust at least one of the depth map and the information, the at least one processor is configured to:

minimize a loss function that is based on the difference between the projected image frame and the second input image frame; and

adjust at least one of the depth map and the information based on the minimization of the loss function.

11. The apparatus of claim 10 , wherein:

the information identifying the conversion from the first image plane to the second image plane comprises rotation and translation values; and

the loss function is based on a combination of (i) a measure of photometric loss between the second input image frame and the projected image frame, (ii) a measure of total variance in the depth map, and (iii) a measure of errors between edges identified in the first input image frame and edges identified in the depth map.

12. The apparatus of claim 8 , wherein:

the at least one processor is further configured to generate the depth map associated with the first input image frame; and

to generate the depth map, the at least one processor is configured to upsample a lower-resolution depth map and produce a higher-resolution depth map.

13. The apparatus of claim 12 , wherein, to generate the depth map, the at least one processor is configured to:

apply multiple machine-learning convolutional layers to at least one of the first and second input image frames in order to produce the lower-resolution depth map; and

apply a machine-learning upsampling layer to the lower-resolution depth map in order to generate the higher-resolution depth map.

14. The apparatus of claim 13 , wherein:

to adjust at least one of the depth map and the information, the at least one processor is configured to apply multiple machine-learning guided depth super-resolution layers to the higher-resolution depth map;

each machine-learning guided depth super-resolution layer is configured to project the first input image frame to the second image plane and adjust at least one of the depth map and the information; and

a last of the machine-learning guided depth super-resolution layers is configured to output the other version of the depth map.

15. A non-transitory computer readable medium containing instructions that when executed cause at least one processor to:

obtain first and second input image frames, the first and second input image frames associated with first and second image planes, respectively;

obtain a depth map associated with the first input image frame; and

produce another version of the depth map;

wherein the instructions that when executed cause the at least one processor to produce the other version of the depth map comprise instructions that when executed cause the at least one processor to, one or more times:

convert the depth map from a coordinate frame associated with the first image plane to a coordinate frame associated with the second image plane;

project the first input image frame to the second image plane in order to produce a projected image frame using (i) the converted depth map and (ii) information identifying a conversion from the first image plane to the second image plane; and

adjust at least one of the depth map and the information identifying the conversion from the first image plane to the second image plane to reduce a difference between the projected image frame and the second input image frame.

16. The non-transitory computer readable medium of claim 15 , wherein the information identifying the conversion from the first image plane to the second image plane is used to convert the depth map from the coordinate frame associated with the first image plane to the coordinate frame associated with the second image plane.

17. The non-transitory computer readable medium of claim 15 , wherein the instructions that when executed cause the at least one processor to adjust at least one of the depth map and the information comprise:

instructions that when executed cause the at least one processor to:

minimize a loss function that is based on the difference between the projected image frame and the second input image frame; and

adjust at least one of the depth map and the information based on the minimization of the loss function.

18. The non-transitory computer readable medium of claim 15 , further containing instructions that when executed cause the at least one processor to:

generate the depth map associated with the first input image frame by upsampling a lower-resolution depth map to produce a higher-resolution depth map.

19. The non-transitory computer readable medium of claim 18 , wherein the instructions that when executed cause the at least one processor to generate the depth map comprise:

instructions that when executed cause the at least one processor to:

apply multiple machine-learning convolutional layers to at least one of the first and second input image frames in order to produce the lower-resolution depth map; and

apply a machine-learning upsampling layer to the lower-resolution depth map in order to generate the higher-resolution depth map.

20. The non-transitory computer readable medium of claim 19 , wherein:

the instructions that when executed cause the at least one processor to adjust at least one of the depth map and the information comprise:

instructions that when executed cause the at least one processor to apply multiple machine-learning guided depth super-resolution layers to the higher-resolution depth map;

each machine-learning guided depth super-resolution layer is configured to project the first input image frame to the second image plane and adjust at least one of the depth map and the information; and

a last of the machine-learning guided depth super-resolution layers is configured to output the other version of the depth map.