Method and device for semantic segmentation of image
The present disclosure provides a method and an apparatus for semantic segmentation of an image, capable of solving the problem in the related art associated with low speed and inefficiency in semantic segmentation of images. The method includes: receiving the image; performing semantic segmentation on the image to obtain an initial semantic segmentation result; and inputting image information containing the initial semantic segmentation result to a pre-trained convolutional neural network for semantic segmentation post-processing, so as to obtain a final semantic segmentation result. With the solutions of the present disclosure, the initial semantic segmentation result can be post-processed using the convolutional neural network, such that the speed and efficiency of the semantic segmentation of the image can be improved.
1. A method for semantic segmentation of an image, comprising:
receiving the image;
performing semantic segmentation on the image to obtain an initial semantic segmentation result, wherein the initial semantic segmentation result has, for each pixel in the image, each of predefined category labels that the pixel belongs to; and
obtaining a final semantic segmentation from the initial semantic segmentation result using a convolutional neural network in which a modified semantic segmentation result is iteratively determined until an iteration condition is met, comprising:
inputting the initial semantic segmentation result and at least one mode corresponding to the image to the convolutional neural network to output the modified semantic segmentation result,
wherein the modified semantic segmentation result has, for each pixel in the image, a determined category label that the pixel belongs to,
wherein the convolutional neural network is pre-trained with sample image information, and wherein the sample image information contains: a first mode, one of predefined category labels that each pixel in a first sample image corresponding to the first mode belongs to, a second mode different from the first mode, and one of predefined category labels that each pixel in a second sample image corresponding to the second mode belongs to.
2. The method of claim 1 , wherein the at least one mode corresponding to the image comprises one or more of the followings: a visible image mode, a depth mode, a Computed Tomography (CT) mode, an infrared mode, a millimeter wave mode and an ultrasound mode.
3. The method of claim 1 , wherein the determining whether the iteration condition is met comprises:
determining whether an accumulated number of iterations has reached a predetermined number threshold.
4. The method of claim 1 , wherein each of the first mode and the second mode is one of the followings: a visible image mode, a depth mode, a Computed Tomography (CT) mode, an infrared mode, a millimeter wave mode, and an ultrasound mode.
5. An apparatus for semantic segmentation of an image comprising a processor configured to:
receive the image;
perform semantic segmentation on the image to obtain an initial semantic segmentation result, wherein the initial semantic segmentation result has, for each pixel in the image, each of predefined category labels that the pixel belongs to;
obtain a final semantic segmentation from the initial semantic segmentation result using a convolutional neural network in which a modified semantic segmentation result is iteratively determined until an iteration condition is met;
input the initial semantic segmentation result and at least one mode corresponding to the image to the convolutional neural network to output the modified semantic segmentation result,
wherein the modified semantic segmentation result has, for each pixel in the image, a determined category label that the pixel belongs to, and
wherein the convolutional neural network is pre-trained with sample image information, and wherein the sample image information contains: a first mode, one of predefined category labels that each pixel in a first sample image corresponding to the first mode belongs to, a second mode different from the first mode, and one of predefined category labels that each pixel in a second sample image corresponding to the second mode belongs to.
6. The apparatus of claim 5 , wherein the at least one mode comprises a visible image mode.
7. The apparatus of claim 6 , wherein the visible image mode comprises an RGB mode or an HSV mode.
8. The apparatus of claim 5 , wherein the processor is configured to post-process the initial semantic segmentation without calculating correlations between respective pixels in the image.
9. The apparatus of claim 5 , wherein the iteration condition comprises whether an accumulated number of iterations has reached a predetermined number threshold.
10. The apparatus of claim 5 , wherein the iteration condition comprises whether a convergence condition between a modified segmentation result currently outputted and a semantic segmentation result previously outputted is satisfied.
11. A non-transitory computer storage medium, storing an executable program which, when executed by a processor, implements a method for semantic segmentation of an image, the method comprising:
receiving the image;
performing semantic segmentation on the image to obtain an initial semantic segmentation result, wherein the initial semantic segmentation result has, for each pixel in the image, each of predefined category labels that the pixel belongs to; and
obtaining a final semantic segmentation from the initial semantic segmentation result using a convolutional neural network in which a modified semantic segmentation result is iteratively determined until an iteration condition is met, comprising:
inputting the initial semantic segmentation result and at least one mode corresponding to the image to the convolutional neural network to output the modified semantic segmentation result,
wherein the modified semantic segmentation result has, for each pixel in the image, a determined category label that the pixel belongs to, and
wherein the convolutional neural network is pre-trained with sample image information, and wherein the sample image information contains: a first mode, one of predefined category labels that each pixel in a first sample image corresponding to the first mode belongs to, a second mode different from the first mode, and one of predefined category labels that each pixel in a second sample image corresponding to the second mode belongs to.
12. The non-transitory computer storage medium of claim 11 , wherein said obtaining the final semantic segmentation from the initial semantic segmentation result further comprises:
determining whether the iteration condition is met; and
in response to the iteration condition being met, replacing the initial semantic segmentation result with the modified semantic segmentation result and repeating the step of inputting the initial semantic segmentation result and at least one mode corresponding to the image to the convolutional neural network.
13. The non-transitory computer storage medium of claim 12 , wherein said determining whether the iteration condition is met comprises:
determining whether an accumulated number of iterations has reached a predetermined number threshold, and
in response to the accumulated number of iterations reaching the predetermined number threshold, determining that the iteration condition is not met.
14. The non-transitory computer storage medium of claim 12 , wherein said determining whether the iteration condition is met comprises:
determining whether the modified semantic segmentation result currently outputted from the convolutional neural network and a semantic segmentation result previously outputted from the convolutional neural network satisfy a convergence condition, and
in response to the modified semantic segmentation result currently outputted from the convolutional neural network and the semantic segmentation result previously outputted from the convolutional neural network satisfying the convergence condition, determining that the iteration condition is not met.
15. The non-transitory computer storage medium of claim 11 , wherein the convolutional neural network is a fully connected convolutional neural network.
16. The method of claim 3 , wherein the determining whether the accumulated number of iterations has reached the predetermined number threshold comprises using a counter to count the accumulated number of iterations, wherein the counter is incremented by 1 for each iteration.
17. The method of claim 1 , wherein the predefined category labels comprise at least one of the followings: bicycle, car, tricycle, pedestrian, road surface, fence, streetlight, tree, and traffic light.
18. The non-transitory computer storage medium of claim 11 , wherein said obtaining the final semantic segmentation from the initial semantic segmentation result further comprises:
determining whether the iteration condition is met; and
in response to the iteration condition being not met, determining to stop an iteration and obtaining the final semantic segmentation result based on the modified semantic segmentation result.