Adaptive cycle consistency multimodal image captioning

1. A computer-implemented method comprising:

encoding, by one or more computer processors, an image utilizing an image encoder, wherein the image is contained within a triplet comprising the image, one or more high-resource captions, and one or more low-resource captions;

generating, by one or more computer processors, one or more high-resource captions utilizing the encoded image and the triplet inputted into a high-resource decoder;

encoding, by one or more computer processors, the one or more generated high-resource captions utilizing a high-resource encoder;

adding, by one or more computer processors, adaptive cycle consistency constraints on a set of calculated attention associated the triplet, wherein the added adaptive cycle consistency constraints align inputs between a cycle of (i) one or more image regions associated with the encoded image, (ii) high-resource captions, and (iii) low-resource captions; and

generating, by one or more computer processors, one or more low-resource captions by simultaneously inputting the encoded image, the encoded high-resource caption, and the triplet into a trained low-resource decoder.

2. The method of claim 1 , wherein adding adaptive cycle consistency constraints on attentions of the triplet, further comprises:

aligning, by one or more computer processors, one or more attention weights associated with one or more triplets fed into a plurality of decoders.

3. The method of claim 2 , wherein the plurality of decoders is associated with an image to low-resource caption decoding, an image to high-resource caption decoding, or a high-resource caption to low-resource caption decoding.

4. The method of claim 1 , further comprising:

adding, by one or more computer processors, a sentinel weight to the set of calculated attention weights associated with one or more triplets fed into a decoder, providing a latent representation of a memory of the decoder.

5. The method of claim 1 , wherein the high-resource encoder and the high-resource decoder, and are each, respectively, a recurrent neural network.

6. The method of claim 1 , wherein the low-resource decoder is an attention-based recurrent neural network trained with the one or more generated high-resource captions and associated low-resource translations.

7. The method of claim 1 , wherein the image encoder is a trained convolutional neural network.

8. A computer program product comprising:

one or more computer readable storage media and program instructions stored on the one or more computer readable storage media, the stored program instructions comprising:

program instructions to encode an image utilizing an image encoder, wherein the image is contained within a triplet comprising the image, one or more high-resource captions, and one or more low-resource captions;

program instructions to generate one or more high-resource captions utilizing the encoded image and the triplet inputted into a high-resource decoder;

program instructions to encode the one or more generated high-resource captions utilizing a high-resource encoder;

program instructions to add adaptive cycle consistency constraints on a set of calculated attention weights associated the triplet, wherein the added adaptive cycle consistency constraints align inputs between a cycle of (i) one or more image regions associated with the encoded image, (ii) high-resource captions, and (iii) low-resource captions; and

program instructions to generate one or more low-resource captions by simultaneously inputting the encoded image, the encoded high-resource caption, and the triplet into a trained low-resource decoder.

9. The computer program product of claim 8 , wherein the program instructions to add adaptive cycle consistency constraints on attentions of the triplet, further comprise:

program instructions to align one or more attention weights associated with one or more triplets fed into a plurality of decoders.

10. The computer program product of claim 8 , wherein the plurality of decoders is associated with an image to low-resource caption decoding, an image to high-resource caption decoding, or a high-resource caption to low-resource caption decoding.

11. The computer program product of claim 8 , wherein the program instructions, stored on the one or more computer readable storage media, further comprise:

program instructions to add a sentinel weight to the set of calculated attention weights associated with one or more triplets fed into a decoder, providing a latent representation of a memory of the decoder.

12. The computer program product of claim 8 , wherein the high-resource encoder and the high-resource decoder, and are each, respectively, a recurrent neural network.

13. The computer program product of claim 8 , wherein the low-resource decoder is an attention-based recurrent neural network trained with the one or more generated high-resource captions and associated low-resource translations.

14. The computer program product of claim 8 , wherein the image encoder is a trained convolutional neural network.

15. A computer system comprising:

one or more computer processors;

one or more computer readable storage media; and

program instructions stored on the computer readable storage media for execution by at least one of the one or more processors, the stored program instructions comprising:

program instructions to encode an image utilizing an image encoder, wherein the image is contained within a triplet comprising the image, one or more high-resource captions, and one or more low-resource captions;

program instructions to generate one or more high-resource captions utilizing the encoded image and the triplet inputted into a high-resource decoder;

program instructions to encode the one or more generated high-resource captions utilizing a high-resource encoder;

program instructions to add adaptive cycle consistency constraints on a set of calculated attention weights associated the triplet, wherein the added adaptive cycle consistency constraints align inputs between a cycle of (i) one or more image regions associated with the encoded image, (ii) high-resource captions, and (iii) low-resource captions; and

program instructions to generate one or more low-resource captions by simultaneously inputting the encoded image, the encoded high-resource caption, and the triplet into a trained low-resource decoder.

16. The computer system of claim 15 , wherein the program instructions to add adaptive cycle consistency constraints on attentions of the triplet, further comprise:

program instructions to align one or more attention weights associated with one or more triplets fed into a plurality of decoders.

17. The computer system of claim 15 , wherein the plurality of decoders is associated with an image to low-resource caption decoding, an image to high-resource caption decoding, or a high-resource caption to low-resource caption decoding.

18. The computer system of claim 15 , wherein the program instructions, stored on the one or more computer readable storage media, further comprise:

program instructions to add a sentinel weight to the set of calculated attention weights associated with one or more triplets fed into a decoder, providing a latent representation of a memory of the decoder.

19. The computer system of claim 15 , wherein the high-resource encoder and the high-resource decoder, and are each, respectively, recurrent neural network.

20. The computer system of claim 15 , wherein the low-resource decoder is an attention-based recurrent neural network trained with the one or more generated high-resource captions and associated low-resource translations.