System and method for providing single image depth estimation based on deep neural network
View Patent ↗A method and system for determining depth information of an image are herein provided. According to one embodiment, the method includes receiving an image input, classifying the input image into a depth range of a plurality of depth ranges, and determining a depth map of the image by applying depth estimation based on the depth range into which the input image is classified.
1. A method for determining depth information of an image, comprising:
receiving an image input;
classifying, using a first network, the input image into a first depth range of a plurality of candidate depth ranges that comprises the first depth range and a second depth range different from the first depth range;
selecting, responsive to classifying the input image into the first depth range, from a plurality of candidate networks that includes (i) a second network optimized to obtain depth maps for the first depth range and (ii) a third network optimized to obtain depth maps for the second depth range, the second network; and
determining a depth map of the input image by applying the second network.
2. The method of claim 1 , wherein classifying the input image is performed based on coarse depth estimation.
3. The method of claim 2 , wherein determining the depth map of the image by applying the second network further comprises utilizing a coarse depth map generated by the course depth estimation as an input for a depth refinement single image depth estimation (SIDE) network.
4. The method of claim 3 , wherein determining the depth map of the image by applying the second network further comprises utilizing the coarse depth map and the input image as an RGB-depth (RGBD) input for an RGBD enhancement SIDE network.
5. The method of claim 2 , wherein the coarse depth estimation is performed by calculating a maximum depth of the input image and comparing the maximum depth with a depth range threshold.
6. The method of claim 1 , wherein classifying the input image includes classifying the input image into a predefined scene.
7. The method of claim 6 , wherein classifying the input image into a predefined scene is performed based on majority voting.
8. The method of claim 6 , wherein classifying the input image into a predefined scene is performed based on weighted voting.
9. The method of claim 1 , wherein determining the depth map of the image by applying the second network further comprises:
encoding, with a single image depth estimation (SIDE) network, the input image; and
decoding, with the SIDE network, the input image with a depth regression decoding branch.
10. The method of claim 1 , wherein determining the depth map of the image by applying the second network further comprises:
encoding, with a single image depth estimation (SIDE) network, the input image; and
decoding, with the SIDE network, the input image with a depth classification decoding branch.
11. A system for determining depth information of an image, comprising:
a memory; and
a processor configured to:
receive an image input;
classify, using a first network, the input image into a first depth range of a plurality of candidate depth ranges that comprises the first depth range and a second depth range different from the first depth range;
select, responsive to classifying the input image into the first depth range, from a plurality of candidate networks that includes (i) a second network optimized to obtain depth maps for the first depth range and (ii) a third network optimized to obtain depth maps for the second depth range, the second network; and
determine a depth map of the image by applying the second network.
12. The system of claim 11 , wherein the processor is configured to classify the input image based on coarse depth estimation.
13. The system of claim 12 , wherein the processor is configured to determine the depth map of the image by applying the second network by utilizing a coarse depth map generated by the course depth estimation as an input for a depth refinement single image depth estimation (SIDE) network.
14. The system of claim 13 , wherein the processor is further configured to determine the depth map of the image by applying the second network by utilizing the coarse depth map and the input image as an RGB-depth (RGBD) input for an RGBD enhancement SIDE network.
15. The system of claim 12 , wherein the coarse depth estimation is performed by calculating a maximum depth of the input image and comparing the maximum depth with a depth range threshold.
16. The system of claim 11 , wherein the processor is configured to classify the input image by classifying the input image into a predefined scene.
17. The system of claim 16 , wherein classifying the input image into a predefined scene is performed based on majority voting.
18. The system of claim 16 , wherein classifying the input image into a predefined scene is performed based on weighted voting.
19. The system of claim 11 , wherein the processor is configured to determine the depth map of the image by applying the second network by:
encoding, with a single image depth estimation (SIDE) network, the input image; and
decoding, with the SIDE network, the input image with a depth regression decoding branch.
20. The system of claim 11 , wherein the processor is configured to determine the depth map of the image by applying the second network by:
encoding, with a single image depth estimation (SIDE) network, the input image; and
decoding, with the SIDE network, the input image with a depth classification decoding branch.