Image captioning with weakly-supervised attention penalty
Techniques for training a machine-learning (ML) model for captioning images are disclosed. A plurality of feature vectors and a plurality of visual attention maps are generated by a visual model of the ML model based on an input image. Each of the plurality of feature vectors correspond to different regions of the input image. A plurality of caption attention maps are generated by an attention model of the ML model based on the plurality of feature vectors. An attention penalty is calculated based on a comparison between the caption attention maps and the visual attention maps. A loss function is calculated based on the attention penalty. One or both of the visual model and the attention model are trained using the loss function.
1. A computer-implemented method of training a machine learning model for captioning images, the computer-implemented method comprising:
receiving an input image;
generating, using a visual model and based on the input image, a feature vector and a visual attention map;
generating, using an attention model and based on the feature vector, a caption attention map;
generating, using a language model and based on one or both of the feature vector and the caption attention map, a predicted caption for the input image;
calculating (i) a caption loss based on a comparison between the predicted caption and a manual caption label and (ii) an attention penalty based on a comparison between the caption attention map and the visual attention map; and
calculating a loss function based on the caption loss and the attention penalty.
2. The computer-implemented method of claim 1 , wherein one or more of the visual model, the attention model, and the language model is trained by:
modifying a set of weights of one or more of the visual model, the attention model, and the language model proportional to a magnitude of the loss function.
3. The computer-implemented method of claim 1 , further comprising:
parsing the predicted caption to identify one or more words related to visual concepts using a category mapping dictionary.
4. The computer-implemented method of claim 3 , wherein the visual model is a pre-trained convolutional neural network.
5. The computer-implemented method of claim 3 , wherein the visual model is fine-tuned using the visual concepts.
6. The computer-implemented method of claim 5 , wherein one or more different attention maps are generated for one or more different visual concepts in the predicted caption.
7. The computer-implemented method of claim 1 , wherein the language model is a long short-term memory model.
8. The computer-implemented method of claim 1 , wherein the attention model comprises one or more of a long short-term memory model and a multi-layer perceptron.
9. A hardware storage device having stored thereon computer-executable instructions that, when executed by one or more processors, configure a computer system to perform at least the following:
receive an input image;
generate, using a visual model and based on the input image, a feature vector and a visual attention map;
generate, using an attention model and based on the feature vector, a caption attention map;
generate, using a language model and based on one or both of the feature vector and the caption attention map, a predicted caption for the input image;
calculate (i) a caption loss based on a comparison between the predicted caption and a manual caption label and (ii) an attention penalty based on a comparison between the caption attention map and the visual attention map; and
calculate a loss function based on the caption loss and the attention penalty.
10. The hardware storage device of claim 9 , wherein the computer-executable instructions further configure the computer system to:
train one or more of the visual model, the attention model, and the language model by modifying a set of weights of one or more of the visual model, the attention model, and the language model proportional to a magnitude of the loss function.
11. The hardware storage device of claim 10 , wherein the visual attention map and the caption attention map are probability distributions.
12. The hardware storage device of claim 9 , wherein a gradients-based weakly supervised model and bottom-up and top-down attention mechanism are used to generate the visual attention map and the caption attention map, respectively.
13. The hardware storage device of claim 9 , wherein the visual model is fine-tuned using a multi-label multi-class classification objective.
14. The hardware storage device of claim 9 , wherein the attention model is a LSTM model configured to generate the caption attention map using the feature vector as inputs.
15. The hardware storage device of claim 9 , wherein the language model is a LSTM model configured to generate the predicted caption using one or more of the feature vector or the caption attention map as inputs.
16. The hardware storage device of claim 9 , wherein for each visual word in the predicted caption, the visual attention map is generated by computing gradients of visual model outputs with respect to a last convolutional layer.
17. A system comprising:
one or more processors; and
one or more hardware storage devices having stored thereon computer-executable instructions that, when executed by the one or more processors, configure the system to perform at least the following:
receive an input image;
generate, using a visual model and based on the input image, a feature vector and a visual attention map;
generate, using an attention model and based on the feature vector, a caption attention map;
generate, using a language model and based on one or both of the feature vector and the caption attention map, a predicted caption for the input image;
calculate (i) a caption loss based on a comparison between the predicted caption and a manual caption label and (ii) an attention penalty based on a comparison between the caption attention map and the visual attention map; and
calculate a loss function based on the caption loss and the attention penalty.
18. The system of claim 17 , wherein the one or more of the visual model, the attention model, and the language model is trained by performing at least the following:
modify a set of weights of one or more of the visual model, the attention model, and the language model proportional to a magnitude of the loss function.
19. The system of claim 18 , wherein the loss function is calculated based on a summation of the caption loss and the attention penalty.
20. The system of claim 17 , wherein the visual attention map is generated by a gradients-based weakly supervised model.