Integrated interactive image segmentation
Methods and systems are provided for optimal segmentation of an image based on multiple segmentations. In particular, multiple segmentation methods can be combined by taking into account previous segmentations. For instance, an optimal segmentation can be generated by iteratively integrating a previous segmentation (e.g., using an image segmentation method) with a current segmentation (e.g., using the same or different image segmentation method). To allow for optimal segmentation of an image based on multiple segmentations, one or more neural networks can be used. For instance, a convolutional RNN can be used to maintain information related to one or more previous segmentations when transitioning from one segmentation method to the next. The convolutional RNN can combine the previous segmentation(s) with the current segmentation without requiring any information about the image segmentation method(s) used to generate the segmentations.
1. A computer-implemented method, the method comprising:
receiving a first interaction with an image;
receiving a second interaction with the image subsequent to the first interaction;
based on the second interaction with the image, performing a segmentation of the image using a first segmentation method to generate a current probability distribution map;
combining a feature map of the image with the current probability distribution map and combining the feature map with a previous probability distribution map, the previous probability distribution map generated based on a previous segmentation of the image, the previous segmentation based on the first interaction with the image and performed using a second segmentation method different from the first segmentation method; and
generating a segmentation mask based on a concatenation of the feature map combined with the current probability distribution map and the feature map combined with the previous probability distribution map.
2. The computer-implemented method of claim 1 , further comprising:
generating the feature map for the image; and
incorporating removal information into the feature map, the removal information based on the second interaction being a removal interaction that indicates an object, feature, or portion of the image for exclusion from the segmentation mask.
3. The computer-implemented method of claim 1 , further comprising:
determining an image segmentation method to use for the segmentation, wherein the image segmentation method is based on the first interaction.
4. The computer-implemented method of claim 1 , wherein the feature map of the image is combined with the current probability distribution map and with the previous probability distribution map using a neural network to maintain a hidden state related to the previous probability distribution map;
wherein the neural network is trained to perform the combining of the feature map of the image with the current probability distribution map, and perform the combining the feature map with the previous probability distribution map, without knowledge of which segmentation method was used to generate the current probability distribution map and without knowledge of which segmentation method was used to generate the previous probability distribution map.
5. The computer-implemented method of claim 1 , wherein the segmentation mask is generated using a classification neural network to convert the feature map combined with the current probability distribution map and the previous probability distribution map into image-form.
6. The computer-implemented method of claim 5 , wherein the classification neural network is trained to intelligently concatenate the feature map combined with the current probability distribution map and the previous probability distribution map.
7. The computer-implemented method of claim 6 , wherein the neural network is trained to integrate various image segmentation methods, the training comprising:
receiving the first interaction with a first image;
based on the first interaction, performing a first segmentation of the first image to generate a first probability distribution map;
storing the first probability distribution map using a hidden state;
receiving the second interaction with the first image;
based on the second interaction, performing a second segmentation of the first image to generate a second probability distribution map;
combining an image feature map of the first image with the second probability distribution map and the first previous probability distribution map, first previous probability distribution map represented using the hidden state; and
generating an optimized segmentation mask based on the image feature map combined with the second probability distribution map and the first probability distribution map.
8. The computer-implemented method of claim 7 , the training further comprising:
comparing the optimized segmentation mask with a ground-truth segmentation mask to determine error; and
updating the neural network based on the determined error.
9. One or more non-transitory computer-readable media having a plurality of executable instructions embodied thereon, which, when executed by one or more processors, cause the one or more processors to perform a method, the method comprising:
receiving a first interaction with an image;
based on the first interaction with the image, performing a first segmentation of the image using a first segmentation method to generate a previous probability distribution map;
receiving a second interaction with the image subsequent to the first interaction;
based on the second interaction with the image, performing a second segmentation of the image using a second segmentation method different from the first segmentation method to generate a current probability distribution map;
generating, using a neural network, a feature map of the image;
combining, using the neural network, the feature map with the current probability distribution map to generate a first feature and the feature map with the previous probability distribution map to generate a second feature, wherein the previous probability distribution map is represented using a hidden state of the neural network, wherein the neural network is trained to perform the combining of the feature map of the image with the current probability distribution map, and perform the combining the feature map with the previous probability distribution map, without knowledge of which segmentation method was used to generate the current probability distribution map and the previous probability distribution map; and
generating a segmentation mask based on a concatenation of the first feature and the second feature.
10. The media of claim 9 , the method further comprising:
incorporating removal information into the feature map, the removal information based on the second interaction being a removal interaction that indicates an object, feature, or portion of the image for exclusion from the segmentation mask.
11. The media of claim 9 , the method further comprising:
determining the first image segmentation method to use for the first segmentation, wherein the first image segmentation method is based on the first interaction; and
determining the second image segmentation method to use for the segmentation, wherein the second image segmentation method is based on the second interaction.
12. The media of claim 9 , the method further comprising:
receiving an indication of an image segmentation method to use for the segmentation, wherein the indication is input by a user.
13. The media of claim 9 , the method further comprising:
receiving a further interaction with the image;
based on the further interaction, performing a subsequent segmentation of the image to generate a subsequent probability distribution map;
combining, using the neural network, the feature map with the subsequent probability distribution map to generate an updated first feature and the current probability distribution map to generate an updated second feature, wherein the current probability distribution map is represented using an updated hidden state of the neural network; and
generating an optimized segmentation mask based on a new concatenation of the updated first feature and the updated second feature.
14. The media of claim 9 , wherein the neural network system is trained by:
receiving the first interaction with a first image;
based on the first interaction, performing the first segmentation of the first image to generate a first probability distribution map;
storing the first probability distribution map using a first hidden state;
receiving the second interaction with the first image;
based on the second interaction, performing the second segmentation of the first image to generate a second probability distribution map;
combining an image feature map of the first image with the second probability distribution map and the first previous probability distribution map, first previous probability distribution map represented using the first hidden state; and
generating an optimized segmentation mask based on the image feature map combined with the second probability distribution map and the first probability distribution map.
15. The media of claim 14 , the training further comprising:
comparing the optimized segmentation mask with a ground-truth segmentation mask to determine error; and
updating the neural network based on the determined error.
16. The media of claim 9 , the method further comprising:
generating a unified library of image segmentation methods, wherein the neural network is trained using one or more of the image segmentation methods.
17. A computing system comprising:
means for receiving a first interaction with an image;
means for receiving a second interaction with the image subsequent to the first interaction;
means for performing segmentation of the image using a first segmentation method based on the second interaction with the image to generate a current probability distribution map;
means for combining a feature map of the image with the current probability distribution map to generate a first feature and for combining the feature map with a previous probability distribution map to generate a second feature, the previous probability distribution map generated based on a previous segmentation of the image, the previous segmentation based on the first interaction with the image and performed using a second segmentation method different from the first segmentation method; and
means for generating a segmentation mask based on a concatenation of the first feature and the second feature.
18. The system of claim 17 , further comprising:
means for incorporating removal information into the feature map, the removal information based on the second interaction being a removal interaction that indicates an object, feature, or portion of the image for exclusion from the segmentation mask.
19. The system of claim 17 , further comprising:
means for determining an image segmentation method to use for the segmentation, wherein the image segmentation method is based on the interaction.
20. The system of claim 17 , further comprising:
means for generating a unified library of image segmentation methods, wherein a neural network is trained using one or more of the image segmentation methods.