Image processing method, apparatus, and device, path planning method, apparatus, and device, and storage medium
A method of depth map completion is described. A color map and a sparse depth map of a target scenario can be received. Resolutions of the color map and the sparse depth map are adjusted to generate n pairs of color maps and sparse depth maps of n different resolutions. The n pairs of color maps and the sparse depth maps can be processed to generate n prediction result maps using a cascade hourglass network including n levels of hourglass networks. Each of the n pair is input to a respective one of the n levels to generate the respective one of the n prediction result maps. The n prediction result maps each include a dense depth map of the same resolution as the corresponding pair. A final dense depth map of the target scenario can be generated according to the dense depth maps.
1. A method of depth map completion, comprising:
receiving, by processing circuitry of an information processing apparatus, a color map and a sparse depth map that are corresponding to a target scenario;
adjusting, by the processing circuitry of the information processing apparatus, resolutions of the color map and the sparse depth map to generate n pairs of color maps and sparse depth maps of n different resolutions, each pair including one color map and one sparse depth map of a respective same resolution of the n different resolutions, n being a positive integer greater than 1;
processing, by the processing circuitry of the information processing apparatus, the n pairs of the color maps and the sparse depth maps to generate n prediction result maps using a cascade hourglass network including n levels of hourglass networks cascaded together, each of the n pairs being an input to a respective one of the n levels of the hourglass networks to generate the respective one of the n prediction result maps, the n prediction result maps each including a dense depth map of the same resolution as the corresponding pair; and
generating, by the processing circuitry of the information processing apparatus, a final dense depth map corresponding to the target scenario according to the dense depth maps of the n prediction result maps, wherein
each input to a first one of the n levels of hourglass networks consists of a first pair of a first color map and a first sparse depth map,
the first pair is one of the n pairs of color maps and sparse depth maps, and
each input to the first one of the n levels of hourglass networks is of a first resolution.
2. The method according to claim 1 , wherein a current-level hourglass network in the cascade hourglass network is configured to process the prediction result map received from a previous-level hourglass network in the cascade hourglass network and the pair of the color map and the sparse depth map input to the current-level hourglass network, to obtain the prediction result map corresponding to the current-level hourglass network.
3. The method according to claim 1 , wherein the processing comprises:
invoking an i-th level of the hourglass network in the n levels of hourglass networks to process an i-th pair of the color map and the sparse depth map of an i-th resolution in the n pairs to obtain an i-th prediction result map of the i-th resolution in the n prediction result map, i being a positive integer smaller than n;
converting the i-th resolution of the i-th prediction result map to an (i+1)-th resolution to obtain a converted prediction result map; and
invoking an (i+1)-th hourglass network to process the converted prediction result map and an (i+1) the pair of the color map and the sparse depth map to obtain an (i+1)-th prediction result map.
4. The method according to claim 3 , wherein the (i+1)-th hourglass network comprises an initial layer, a downsampling layer, an upsampling layer, and a prediction layer, and
the invoking the (i+1)-th hourglass network comprises:
performing fusion processing on the converted prediction result map and the (i+1) the (i+1)-th pair of the color map and the sparse depth map by using the initial layer, to obtain a fused feature map;
performing downsampling on the fused feature map by using the downsampling layer to obtain a downsampled feature map;
performing upsampling on the downsampled feature map by using the upsampling layer to obtain a feature map of the (i+1)-th resolution; and
predicting by using the prediction layer the (i+1)-th prediction result map based on the feature map of the (i+1)-th resolution.
5. The method according to claim 4 , wherein one or more of the n prediction result maps each further comprise a feature guide map of the same resolution of the respective pair, the feature guide map being used for guiding filling of pixels in a sparse depth map; and
the performing fusion processing comprises:
combining a feature guide map in the converted prediction result map and the color map of the (i+1)-th pair to obtain a first feature map;
combining the dense depth map in the converted prediction result map and the sparse depth map of the (i+1)-th pair to obtain a second feature map; and
performing fusion processing on the first feature map and the second feature map by using the initial layer, to obtain the fused feature map, the fused feature map comprising a first feature map part and a second feature map part, and downsampling being separately performed on the first feature map part and the second feature map part in a case of passing through the downsampling layer.
6. The method according to claim 4 , wherein the downsampling layer comprises a first downsampling layer and a second downsampling layer, and the upsampling layer comprises a first upsampling layer and a second upsampling layer, and the method further comprises:
combining an output of the first downsampling layer and an output of the second upsampling layer by using a residual connection, to obtain an input to the prediction layer; and
combining an output of the second downsampling layer and an output of the first upsampling layer by using the residual connection, to obtain an input to the second upsampling layer.
7. The method according to claim 3 , wherein the converting the i-th resolution of the i-th prediction result map comprises:
performing upsampling on the i-th prediction result map to obtain the converted prediction result map.
8. The method according to claim 1 , wherein n is equal to 2, the n resolutions comprise the first resolution and a second resolution, the first resolution is different from the second resolution, and the cascade hourglass network comprises a level-1 hourglass network that is the first one of the n levels of hourglass networks and a level-2 hourglass network that are cascaded, and
the processing comprises:
invoking the level-1 hourglass network to process the first pair of the first color map and the first sparse depth map to obtain a first prediction result map in the first resolution;
converting the resolution of the first prediction result map into the second resolution to obtain a converted prediction result map in the second resolution; and
invoking the level-2 hourglass network to process the converted prediction result map and a second pair of the color map and the sparse depth map to obtain a second prediction result map in the second resolution.
9. The method according to claim 1 , wherein the generating comprises:
converting resolutions of ones of the n dense depth maps in the n prediction result maps into a preset resolution to obtain a set of dense depth maps in the preset resolution; and
combining the set of the dense depth maps in the preset resolution by using a residual connection, to obtain the final dense depth map corresponding to the target scenario.
10. The method according to claim 1 , wherein a training process of the cascade hourglass network comprises:
calculating, for a level-k hourglass network in the n levels of the hourglass networks, a predicted loss value of the level-k hourglass network according to a sparse depth map of a k-th resolution in the n different resolutions and a dense depth map of the k-th resolution, the predicted loss value being used for representing a prediction error of predicting the dense depth map of the k-th resolution by using the sparse depth map of the k-th resolution, and k being a positive integer smaller than or equal to n;
calculating a predicted loss value of the cascade hourglass network according to weight parameters and predicted loss values of each of the n hourglass networks, the weight parameters each corresponding to one of the predicted loss values of the n hourglass networks; and
adjusting parameters of the cascade hourglass network according to the predicted loss value of the cascade hourglass network.
11. An apparatus for information processing, comprising:
processing circuitry configured to
receive a color map and a sparse depth map that are corresponding to a target scenario;
adjust resolutions of the color map and the sparse depth map to generate n pairs of color maps and sparse depth maps of n different resolutions, each pair including one color map and one sparse depth map of a respective same resolution of the n different resolutions, n being a positive integer greater than 1;
process the n pairs of the color maps and the sparse depth maps to generate n prediction result maps using a cascade hourglass network including n levels of hourglass networks cascaded together, each of the n pairs being input to a respective one of the n levels of the hourglass networks to generate the respective one of the n prediction result maps, the n prediction result maps each including a dense depth map of the same resolution as the corresponding pair; and
generate a final dense depth map corresponding to the target scenario according to the dense depth maps of the n prediction result maps, wherein
each input to a first one of the n levels of hourglass networks consists of a first pair of a first color map and a first sparse depth map,
the first pair is one of the n pairs of color maps and sparse depth maps, and
each input to the first one of the n levels of hourglass networks is of a first resolution.
12. The apparatus according to claim 11 , wherein a current-level hourglass network in the cascade hourglass network is configured to process the prediction result map received from a previous-level hourglass network in the cascade hourglass network and the pair of the color map and the sparse depth map input to the current-level hourglass network, to obtain the prediction result map corresponding to the current-level hourglass network.
13. The apparatus according to claim 11 , wherein the processing circuitry is further configured to:
invoke an i-th level of the hourglass network in the n levels of hourglass networks to process an i-th pair of the color map and the sparse depth map of an i-th resolution in the n pairs to obtain an i-th prediction result map of the i-th resolution in the n prediction result map, I being a positive integer smaller than n;
convert the i-th resolution of the i-th prediction result map to an (i+1)-th resolution to obtain a converted prediction result map; and
invoke an (i+1)-th hourglass network to process the converted prediction result map and an (i+1)-th pair of the color map and the sparse depth map to obtain an (i+1)-th prediction result map.
14. The apparatus according to claim 13 , wherein the (i+1)-th hourglass network comprises an initial layer, a downsampling layer, an upsampling layer, and a prediction layer, and
the processing circuitry is further configured to:
perform fusion processing on the converted prediction result map and the (i+1)-th pair of the color map and the sparse depth map by using the initial layer, to obtain a fused feature map;
perform downsampling on the fused feature map by using the downsampling layer to obtain a downsampled feature map;
perform upsampling on the downsampled feature map by using the upsampling layer to obtain a feature map of the (i+1)-th resolution; and
predict by using the prediction layer the (i+1)-th prediction result map based on the feature map of the (i+1)-th resolution.
15. The apparatus according to claim 14 , wherein one or more of the n prediction result maps each further comprise a feature guide map of the same resolution of the respective pair, the feature guide map being used for guiding filling of pixels in a sparse depth map; and
the processing circuitry is further configured to:
combine a feature guide map in the converted prediction result map and the color map of the (i+1)-th pair to obtain a first feature map;
combine the dense depth map in the converted prediction result map and the sparse depth map of the (i+1)-th pair to obtain a second feature map; and
perform fusion processing on the first feature map and the second feature map by using the initial layer, to obtain the fused feature map, the fused feature map comprising a first feature map part and a second feature map part, and downsampling being separately performed on the first feature map part and the second feature map part in a case of passing through the downsampling layer.
16. The apparatus according to claim 14 , wherein the downsampling layer comprises a first downsampling layer and a second downsampling layer, and the upsampling layer comprises a first upsampling layer and a second upsampling layer, and the processing circuitry is further configured to:
combine an output of the first downsampling layer and an output of the second upsampling layer by using a residual connection, to obtain an input to the prediction layer; and
combine an output of the second downsampling layer and an output of the first upsampling layer by using the residual connection, to obtain an input to the second upsampling layer.
17. The apparatus according to claim 13 , wherein the processing circuitry is further configured to:
perform upsampling on the i-th prediction result map to obtain the converted prediction result map.
18. The apparatus according to claim 11 , wherein n is equal to 2, the n resolutions comprise the first resolution and a second resolution, the first resolution is different from the second resolution, and the cascade hourglass network comprises a level-1 hourglass network that is the first one of the n levels of hourglass networks and a level-2 hourglass network that are cascaded, and the processing circuitry is further configured to:
invoke the level-1 hourglass network to process the first pair of the first color map and the first sparse depth map to obtain a first prediction result map in the first resolution;
convert the resolution of the first prediction result map into the second resolution to obtain a converted prediction result map in the second resolution; and
invoke the level-2 hourglass network to process the converted prediction result map and a second pair of the color map and the sparse depth map to obtain a second prediction result map in the second resolution.
19. The apparatus according to claim 11 , wherein the processing circuitry is further configured to:
convert resolutions of ones of the n dense depth maps in the n prediction result maps into a preset resolution to obtain a set of dense depth maps in the preset resolution; and
combine the set of the dense depth maps in the preset resolution by using a residual connection, to obtain the final dense depth map corresponding to the target scenario.
20. A non-transitory computer-readable storage medium storing instructions, which when executed by at least one processor, cause the at least one processor to perform:
receiving a color map and a sparse depth map that are corresponding to a target scenario;
adjusting resolutions of the color map and the sparse depth map to generate n pairs of color maps and sparse depth maps of n different resolutions, each pair including one color map and one sparse depth map of a respective same resolution of the n different resolutions, n being a positive integer greater than 1;
processing the n pairs of the color maps and the sparse depth maps to generate n prediction result maps using a cascade hourglass network including n levels of hourglass networks cascaded together, each of the n pairs being input to a respective one of the n levels of the hourglass networks to generate the respective one of the n prediction result maps, the n prediction result maps each including a dense depth map of the same resolution as the corresponding pair; and
generating a final dense depth map corresponding to the target scenario according to the dense depth maps of the n prediction result maps, wherein
each input to a first one of the n levels of hourglass networks consists of a first pair of a first color map and a first sparse depth map,
the first pair is one of the n pairs of color maps and sparse depth maps, and
each input to the first one of the n levels of hourglass networks is of a first resolution.