Semantic class localization digital environment
Semantic segmentation techniques and systems are described that overcome the challenges of limited availability of training data to describe the potentially millions of tags that may be used to describe semantic classes in digital images. In one example, the techniques are configured to train neural networks to leverage different types of training datasets using sequential neural networks and use of vector representations to represent the different semantic classes.
1. In a digital medium semantic class localization environment, a method implemented by a least one computing device, the method comprising:
converting, by the at least one computing device, a tag into a vector representation, the tag defining a semantic class to be located in a digital image;
generating, by the at least one computing device, an attention map by an embedding neural network based on the digital image and the vector representation, the attention map defining a location in the digital image that corresponds to the semantic class, the embedding neural network trained using image-level tags of respective semantic classes;
refining, by the at least one computing device, the location of the semantic class in the attention map by a refinement neural network, the refinement neural network trained using localized tags of respective semantic classes; and
indicating, by the at least one computing device, the refined location of the semantic class in the digital image using the refined attention map.
2. The method as described in claim 1 , wherein the converting of the vector representation uses an embedding neural network as part of machine learning.
3. The method as described in claim 1 , wherein the image-level tags indicate respective semantic classes that are associated with respective digital images as a whole that are used to train the embedding neural network.
4. The method as described in claim 1 , wherein the image-level tags are not localized to respective portions of digital images that are used to train the embedding neural network.
5. The method as described in claim 1 , wherein the localized tags of the semantic classes are localized to respective portions of digital images used to train the refinement neural network using respective bounding boxes.
6. The method as described in claim 1 , wherein the localized tags of the semantic classes are localized to respective portions of digital images used to train the refinement neural network using respective segmentation masks.
7. The method as described in claim 1 , wherein a number of semantic classes used to train the refinement neural network is less than a number of semantic classes used to train the embedding neural network.
8. The method as described in claim 1 , wherein the refining by the refinement neural network includes:
refining the location of the semantic class in the attention map to generate an initial refined location by an initial refinement neural network, the initial refinement neural network trained using localized tags that are localized using respective bounding boxes; and
refining the initial refined location of the semantic class to generate a subsequent refined location by a subsequent refinement neural network, the subsequent refinement neural network trained using localized tags that are localized using respective segmentation masks, and wherein the indicating is based on the subsequent refined location.
9. The method as described in claim 1 , wherein the tag defining the semantic class to be located in the digital image is not one of the image-level tags used to train the embedding neural network and is not one of the localized tags used to train the refinement neural network.
10. The method as described in claim 1 , wherein the converting is performed for first and second said tags and the generating, the refining, and the indicating are performed jointly based on the first and second tags.
11. In a digital medium semantic class localization environment, a system comprising:
at least one processor;
at least one memory storing instructions configured to cause the at least one processor to:
convert a tag into a vector representation, the tag defining a semantic class to be located in a digital image;
implement an embedding neural network to generate an attention map based on the digital image and the vector representation, the attention map defining a location in the digital image that corresponds to the semantic class, the embedding neural network trained using image-level tags of respective semantic classes; and
implement a refinement neural network to refine the location of the semantic class in the attention map, the refinement neural network trained using localized tags of semantic classes.
12. The system as described in claim 11 , wherein the image-level tags indicate respective semantic classes are associated with respective digital images as a whole that are used to train the embedding neural network and are not localized to respective portions of the digital images.
13. The system as described in claim 11 , wherein the localized tags of the semantic classes are localized to respective portions of digital images used to train the refinement neural network using respective bounding boxes.
14. The system as described in claim 11 , wherein the localized tags of the semantic classes are localized to respective portions of digital images used to train the refinement neural network using respective segmentation masks.
15. The system as described in claim 11 , wherein the instructions are configured to cause the at least one processor to:
refine the location of the semantic class in the attention map to an initial refined location by an initial refinement neural network, the initial refinement neural network trained using localized tags of semantic classes that are localized using respective bounding boxes; and
refine the initial refined location of the semantic class to generate a subsequent refined location by a subsequent refinement neural network, the subsequent refinement neural network trained using localized that are localized using respective segmentation masks.
16. The system as described in claim 11 , wherein the tag defining the semantic class to be located in the digital image is not one of the image-level tags used to train the embedding neural network and is not one of the localized tags used to train the refinement neural network.
17. In a digital medium semantic class localization environment, a method implemented by a least one computing device, the method comprising:
converting, using a processor of the computing device, a tag defining a semantic class to be located in a digital image into a vector representation;
generating an attention map by an embedding network as part of machine learning based on the digital image and the vector representation, the attention map defining a location in the digital image that correspond to the semantic class, the embedding network trained using image-level tags of respective semantic classes;
refining the location of the semantic class in the attention map to an initial refined location by an initial refinement neural network, the initial refinement neural network trained using localized tags of semantic classes that are localized using respective bounding boxes; and
refining the initial refined location of the semantic class to a subsequent refined location by a subsequent refinement neural network, the subsequent refinement neural network trained using localized tags of semantic classes that are localized using respective segmentation masks.
18. The system as described in claim 17 , wherein the image-level tags indicate respective semantic classes that are associated with respective digital images as a whole that are used to train the embedding neural network and are not localized to respective portions of the digital images.
19. The system as described in claim 17 , wherein the respective segmentation masks are pixel-level segmentation masks.
20. The system as described in claim 17 , wherein:
a number of the localized tags used to train the subsequent refinement neural network is less than a number of the localized tags used to train the initial refinement neural network; and
the number of the localized tags used to train the initial refinement neural network is less than a number of the image-level tags used to train the embedding neural network.