Method and device for semantic segmentation of image

1. A method for semantic segmentation of an image, comprising:

receiving the image;

performing semantic segmentation on the image to obtain an initial semantic segmentation result; and

inputting image information containing the initial semantic segmentation result to a pre-trained convolutional neural network for semantic segmentation post-processing, so as to obtain a final semantic segmentation result,

wherein said inputting the image information containing the initial semantic segmentation result to the pre-trained convolutional neural network for semantic segmentation post-processing so as to obtain the final semantic segmentation result comprises:

inputting the image information to the convolutional neural network to obtain a modified semantic segmentation result;

determining whether an iteration condition is met; and

if so, replacing the initial semantic segmentation result in the image information with the modified semantic segmentation result and repeating the step of inputting the image information to the convolutional neural network, or

otherwise determining to stop an iteration and obtaining the final semantic segmentation result based on the modified semantic segmentation result.

2. The method of claim 1 , wherein the image information further contains at least one mode corresponding to the image and describing feature information of the image.

3. A method for semantic segmentation of an image, comprising:

receiving the image;

performing semantic segmentation on the image to obtain an initial semantic segmentation result; and

inputting image information containing the initial semantic segmentation result to a pre-trained convolutional neural network for semantic segmentation post-processing, so as to obtain a final semantic segmentation result,

wherein the convolutional neural network consists of at least two stages of convolutional neural sub-networks, and said inputting the image information containing the initial semantic segmentation result to the pre-trained convolutional neural network for semantic segmentation post-processing so as to obtain the final semantic segmentation result comprises:

for each of the at least two stages of convolutional neural sub-networks, in a cascaded order:

inputting the initial semantic segmentation result to the convolutional neural sub-network at that stage to obtain a modified semantic segmentation result; and

using the modified semantic segmentation result as an initial semantic segmentation result for the convolutional neural sub-network at the next stage; and

determining the final semantic segmentation result based on the modified semantic segmentation result outputted from the convolutional neural sub-network at the last stage.

4. The method of claim 3 , wherein said inputting the image information containing the initial semantic segmentation result to the pre-trained convolutional neural network for semantic segmentation post-processing so as to obtain the final semantic segmentation result comprises:

for each of the at least two stages of convolutional neural sub-networks, in a cascaded order:

determining whether an iteration condition is met; and

if so, replacing the initial semantic segmentation result for the convolutional neural sub-network at that stage with the modified semantic segmentation result; and repeating the step of inputting the image information to the convolutional neural sub-network at that stage, or

otherwise determining to stop an iteration;

wherein said using the modified semantic segmentation result as the initial semantic segmentation result for the convolutional neural sub-network at the next stage comprises:

in response to determining to stop the iteration, using the modified semantic segmentation result as the initial semantic segmentation result for the convolutional neural sub-network at the next stage.

5. The method of claim 3 , wherein the image information further contains at least one mode corresponding to the image and describing feature information of the image and inputting the initial semantic segmentation result to the convolutional neural sub-network at that stage to obtain the modified semantic segmentation result comprises:

inputting the initial semantic segmentation result; and a mode among the at least one mode that corresponds to the convolutional neural sub-network at that stage to the convolutional neural sub-network at that stage to obtain the modified semantic segmentation result.

6. The method of claim 5 , wherein said inputting the image information containing the initial semantic segmentation result to the pre-trained convolutional neural network for semantic segmentation post-processing so as to obtain the final semantic segmentation result further comprises:

for each of the at least two stages of convolutional neural sub-networks, in a cascaded order:

determining whether an iteration condition is met; and

if so, replacing the initial semantic segmentation result for the convolutional neural sub-network at that stage with the modified semantic segmentation result; and repeating the step of inputting the image information and the mode among the at least one mode that corresponds to the convolutional neural sub-network at that stage to the convolutional neural sub-network at that stage, or

otherwise determining to stop an iteration;

wherein said using the modified semantic segmentation results in the initial semantic segmentation result for the convolution neural sub-network at the next stage comprises:

in response to determining to stop the iteration, using the modified semantic segmentation result as the initial semantic segmentation result for the convolutional neural sub-network at the next stage.

7. The method of claim 1 , wherein the convolutional neural network consists of two stages of convolutional neural sub-networks, comprising a first stage of convolutional neural sub-network which is a convolutional neural network for global information optimization post-processing, and a second stage of convolutional neural sub-network which is a convolutional neural network for local edge optimization post-processing.

8. The method of claim 6 , wherein said determining whether the iteration condition is met comprises:

determining whether an accumulated number of iterations has reached a predetermined number threshold, and if so, determining that the iteration condition is not met, or otherwise determining that the iteration condition is met, or

determining whether a modified semantic segmentation result currently outputted from the convolutional neural sub-network at the stage and a semantic segmentation result previously outputted from the convolutional neural sub-network at the stage satisfy a convergence condition, and if so, determining that the iteration condition is not met, or otherwise determining that the iteration condition is met.

9. The method of claim 1 , wherein the semantic segmentation result is a confidence map or category labels to which respective pixels in the image belong.

10. The method of claim 2 , wherein the at least one mode corresponding to the image comprises one or more of: a visible image mode, a depth mode, a Computed Tomography (CT) mode, an infrared mode, a millimeter wave mode or an ultrasound mode.

11. An apparatus for semantic segmentation of an image, comprising a processor and at least one memory storing at least one machine executable instruction, the processor being operative to execute the at least one machine executable instruction to:

receive the image;

perform semantic segmentation on the image to obtain an initial semantic segmentation result; and

input image information containing the initial semantic segmentation result to a pre-trained convolutional neural network for semantic segmentation post-processing, so as to obtain a final semantic segmentation result;

wherein the image information further contains at least one mode corresponding to the image and describing feature information of the image, and further wherein the processor being operative to execute the at least one machine executable instruction to input the image information containing the initial semantic segmentation result to the pre-trained convolutional neural network for semantic segmentation post-processing so as to obtain the final semantic segmentation result comprises the processor being operative to execute the at least one machine executable instruction to:

input the image information to the convolutional neural network to obtain a modified semantic segmentation result;

determine whether an iteration condition is met; and

if so, replace the initial semantic segmentation result in the image information with the modified semantic segmentation result and repeat the step of inputting the image information to the convolutional neural network, or

otherwise determine to stop an iteration and obtain the final semantic segmentation result based on the modified semantic segmentation result.

12. An apparatus for semantic segmentation of an image, comprising a processor and at least one memory storing at least one machine executable instruction, the processor being operative to execute the at least one machine executable instruction to:

receive the image;

perform semantic segmentation on the image to obtain an initial semantic segmentation result; and

input image information containing the initial semantic segmentation result to a pre-trained convolutional neural network for semantic segmentation post-processing, so as to obtain a final semantic segmentation result;

wherein the convolutional neural network consists of at least two stages of convolutional neural sub-networks, and the processor being operative to execute the at least one machine executable instruction to input the image information containing the initial semantic segmentation result to the pre-trained convolutional neural network for semantic segmentation post-processing so as to obtain the final semantic segmentation result comprises the processor being operative to execute the at least one machine executable instruction to:

for each of the at least two stages of convolutional neural sub-networks, in a cascaded order:

input the initial semantic segmentation result to the convolutional neural sub-network at that stage to obtain a modified semantic segmentation result; and

use the modified semantic segmentation result as an initial semantic segmentation result for the convolutional neural sub-network at the next stage; and

determine the final semantic segmentation result based on the modified semantic segmentation result outputted from the convolutional neural sub-network at the last stage.

13. The apparatus of claim 12 , wherein the processor being operative to execute the at least one machine executable instruction to input the image information containing the initial semantic segmentation result to the pre-trained convolutional neural network for semantic segmentation post-processing so as to obtain the final semantic segmentation result further comprises the processor being operative to execute the at least one machine executable instruction to:

for each of the at least two stages of convolutional neural sub-networks, in a cascaded order:

determine whether an iteration condition is met; and

if so, replace the initial semantic segmentation result for the convolutional neural sub-network at that stage with the modified semantic segmentation result and repeat the step of inputting the image information to the convolutional neural sub-network at that stage, or

otherwise determine to stop an iteration;

the processor being operative to use the modified semantic segmentation result as the initial semantic segmentation result for the convolution neural sub-network at the next stare comprises the processor being operative to execute the at least one machine executable instruction to:

in response to determining to stop the iteration, use the modified semantic segmentation result as the initial semantic segmentation result for the convolutional neural sub-network at the next stage.

14. The apparatus of claim 12 , wherein the image information further contains at least one mode corresponding to the image and describing feature information of the image, and the processor being operative to input the initial semantic segmentation result to the convolution neural sub-network at that stage to obtain the modified semantic segmentation result comprises the processor being operative to execute the at least one machine executable instruction to:

input the initial semantic segmentation result; and a mode among the at least one mode that corresponds to the convolutional neural sub-network at that stage to the convolutional neural sub-network at that stage to obtain the modified semantic segmentation result.

15. The apparatus of claim 14 , wherein the processor being operative to execute the at least one machine executable instruction to input the image information containing the initial semantic segmentation result to the pre-trained convolutional neural network for semantic segmentation post-processing so as to obtain the final semantic segmentation result further comprises the processor being operative to execute the at least one machine executable instruction to:

for each of the at least two stages of convolutional neural sub-networks, in a cascaded order:

determine whether an iteration condition is met; and

if so, replace the initial semantic segmentation result for the convolutional neural sub-network at that stage with the modified semantic segmentation result and repeat the step of inputting the image information and the mode among the at least one mode that corresponds to the convolutional neural sub-network at that stage to the convolutional neural sub-network at that stage, or

otherwise determine to stop an iteration;

the processor being operative to use the modified semantic segmentation result as the initial semantic segmentation result for the convolution neural sub-network at the next stage comprises the processor being operative to execute the at least one machine executable instruction to:

in response to determining to stop the iteration, using the modified semantic segmentation result as the initial semantic segmentation result for the convolutional neural sub-network at the next stage.

16. The apparatus of claim 11 , wherein the convolutional neural network consists of two stages of convolutional neural sub-networks, comprising a first stage of convolutional neural sub-network which is a convolutional neural network for global information optimization post-processing, and a second stage of convolutional neural sub-network which is a convolutional neural network for local edge optimization post-processing.

17. The apparatus of claim 15 , wherein the processor being operative to execute the at least one machine executable instruction to determine whether the iteration condition is met comprises the processor being operative to execute the at least one machine executable instruction to:

determine whether an accumulated number of iterations has reached a predetermined number threshold, and if so, determine that the iteration condition is not met, or otherwise determine that the iteration condition is met, or

determine whether a modified semantic segmentation result currently outputted from the convolutional neural sub-network at the stage and a semantic segmentation result previously outputted from the convolutional neural sub-network at the stage satisfy a convergence condition, and if so, determine that the iteration condition is not met, or otherwise determine that the iteration condition is met.

18. The apparatus of claim 11 , wherein the semantic segmentation result is a confidence map or category labels to which respective pixels in the image belong.

19. The apparatus of claim 11 , wherein the at least one mode corresponding to the image comprises one or more of: a visible image mode, a depth mode, a Computed Tomography (CT) mode, an infrared mode, a millimeter wave mode or an ultrasound mode.

20. The method of claim 4 , wherein said determining whether the iteration condition is met comprises:

determining whether an accumulated number of iterations has reached a predetermined number threshold, and if so, determining that the iteration condition is not met, or otherwise determining that the iteration condition is met, or

determining whether a modified semantic segmentation result currently outputted from the convolutional neural sub-network at the stage and a semantic segmentation result previously outputted from the convolutional neural sub-network at the stage satisfy a convergence condition, and if so, determining that the iteration condition is not met, or otherwise determining that the iteration condition is met.

21. The apparatus of claim 13 , wherein the processor being operative to execute the at least one machine executable instruction to determine whether the iteration condition is met comprises the processor being operative to execute the at least one machine executable instruction to:

determine whether an accumulated number of iterations has reached a predetermined number threshold, and if so, determine that the iteration condition is not met, or otherwise determine that the iteration condition is met, or

determine whether a modified semantic segmentation result currently outputted from the convolutional neural sub-network at the stage and a semantic segmentation result previously outputted from the convolutional neural sub-network at the stage satisfy a convergence condition, and if so, determine that the iteration condition is not met, or otherwise determine that the iteration condition is met.