System and method for training a model using localized textual supervision
Systems and methods for training a model are described herein. In one example, a system for training the model includes a processor and a memory in communication with the processor having a training module. The training module has instructions that cause the processor to determine a contrastive loss using a self-supervised contrastive loss function, adjust, based on the contrastive loss, model weights a visual backbone that generated feature maps and/or a textual backbone that generated feature vectors. The training module also has instructions that cause the processor to determine a localized loss using a supervised loss function that compares an image-caption attention map with visual identifiers and adjust, based on the localized loss, the model weights the visual backbone and/or the textual backbone.
1. A system for training a model comprising:
a processor; and
a memory in communication with the processor having a training module, the training module having instructions that, when executed by the processor, cause the processor to:
determine a contrastive loss using a self-supervised contrastive loss function based on feature maps describing a visual content of an image having objects and feature vectors describing a meaning of words of a caption describing the objects within the image,
adjust, based on the contrastive loss, model weights of at least one of a visual backbone that generated the feature maps and a textual backbone that generated the feature vectors,
determine a localized loss using a supervised loss function that compares an image-caption attention map with visual identifiers, the visual identifiers identifying locations of the objects within the image and are associated with portions of the caption describing the objects, and
adjust, based on the localized loss, the model weights of at least one of the visual backbone and the textual backbone.
2. The system of claim 1 , wherein the training module further includes instructions that, when executed by the processor, cause the processor to:
generate the image-caption attention map based on the feature maps and the feature vectors, the image-caption attention map identifying locations and object types of the objects within the image, and
determine the localized loss by comparing the locations and object types of the objects defined by the image-caption attention map with the visual identifiers.
3. The system of claim 2 , wherein the training module further includes instructions that, when executed by the processor, cause the processor to:
transform the feature vectors and the feature maps using a secondary neural network having a multi-dimensional fully-connected layer to generate transformed feature vectors and transformed feature maps, and
calculate the image-caption attention map as a normalized product between the transformed feature vectors and the transformed feature maps.
4. The system of claim 3 , wherein the training module further includes instructions that, when executed by the processor, cause the processor to adjust, based on the localized loss, the model weights of the secondary neural network.
5. The system of claim 2 , wherein the training module further includes instructions that, when executed by the processor, cause the processor to:
temporally crop portions of the visual identifiers to generate cropped visual identifiers that correspond to the words of the caption associated with each of the objects,
render covered regions of the image associated with the cropped visual identifiers to generate binary masks,
stack the binary masks together to generate a rendered attention, and
determine the localized loss using the supervised loss function that compares the image-caption attention map with the rendered attention.
6. The system of claim 1 , wherein the visual identifiers are mouse traces indicating a location of objects with the image.
7. The system of claim 1 , wherein the training module further includes instructions that, when executed by the processor, cause the processor to pull positive pairs of the feature maps and the feature vectors closer and push non-matching pairs of the feature maps and the feature vectors apart using the self-supervised contrastive loss function to determine the contrastive loss.
8. A method for training a model comprising the step of:
determining a contrastive loss using a self-supervised contrastive loss function based on feature maps describing a visual content of an image having objects and feature vectors describing a meaning of words of a caption describing the objects within the image;
adjusting, based on the contrastive loss, model weights of at least one of a visual backbone that generated the feature maps and a textual backbone that generated the feature vectors;
determining a localized loss using a supervised loss function that compares an image-caption attention map with visual identifiers, the visual identifiers identifying locations of the objects within the image and are associated with portions of the caption describing the objects; and
adjusting, based on the localized loss, the model weights of at least one of the visual backbone and the textual backbone.
9. The method of claim 8 , further comprising the steps of:
generating the image-caption attention map based on the feature maps and the feature vectors, the image-caption attention map identifying locations and object types of the objects within the image; and
determining the localized loss by comparing the locations and object types of the objects defined by the image-caption attention map with the visual identifiers.
10. The method of claim 9 , further comprising the steps of:
transforming the feature vectors and the feature maps using a secondary neural network having a multi-dimensional fully-connected layer to generate transformed feature vectors and transformed feature maps; and
calculating the image-caption attention map as a normalized product between the transformed feature vectors and the transformed feature maps.
11. The method of claim 10 , further comprising the step of adjusting, based on the localized loss, the model weights of the secondary neural network.
12. The method of claim 9 , further comprising the steps of:
temporally cropping portions of the visual identifiers to generate cropped visual identifiers that correspond to the words of the caption associated with each of the objects;
rendering covered regions of the image associated with the cropped visual identifiers to generate binary masks;
stacking the binary masks together to generate a rendered attention; and
determining the localized loss using the supervised loss function that compares the image-caption attention map with the rendered attention.
13. The method of claim 8 , wherein the visual identifiers are mouse traces indicating a location of objects with the image.
14. The method of claim 8 , further comprising the step of pulling positive pairs of the feature maps and the feature vectors closer and pushing non-matching pairs of the feature maps and the feature vectors apart using the self-supervised contrastive loss function to determine the contrastive loss.
15. A non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause the processor to:
determine a contrastive loss using a self-supervised contrastive loss function based on feature maps describing a visual content of an image having objects and feature vectors describing a meaning of words of a caption describing the objects within the image;
adjust, based on the contrastive loss, model weights of at least one of a visual backbone that generated the feature maps and a textual backbone that generated the feature vectors;
determine a localized loss using a supervised loss function that compares an image-caption attention map with visual identifiers, the visual identifiers identifying locations of the objects within the image and are associated with portions of the caption describing the objects; and
adjust, based on the localized loss, the model weights of at least one of the visual backbone and the textual backbone.
16. The non-transitory computer-readable medium of claim 15 , further comprising instructions that, when executed by the processor, cause the processor to:
generate the image-caption attention map based on the feature maps and the feature vectors, the image-caption attention map identifying locations and object types of the objects within the image; and
determine the localized loss by comparing the locations and object types of the objects defined by the image-caption attention map with the visual identifiers.
17. The non-transitory computer-readable medium of claim 16 , further comprising instructions that, when executed by a processor, cause the processor to:
transform the feature vectors and the feature maps using a secondary neural network having a multi-dimensional fully-connected layer to generate transformed feature vectors and transformed feature maps; and
calculate the image-caption attention map as a normalized product between the transformed feature vectors and the transformed feature maps.
18. The non-transitory computer-readable medium of claim 17 , further comprising instructions that, when executed by a processor, cause the processor to adjust, based on the localized loss, the model weights of the secondary neural network.
19. The non-transitory computer-readable medium of claim 16 , further comprising instructions that, when executed by a processor, cause the processor to:
temporally crop portions of the visual identifiers to generate cropped visual identifiers that correspond to the words of the caption associated with each of the objects;
render covered regions of the image associated with the cropped visual identifiers to generate binary masks;
stack the binary masks together to generate a rendered attention; and
determine the localized loss using the supervised loss function that compares the image-caption attention map with the rendered attention.
20. The non-transitory computer-readable medium of claim 15 , further comprising instructions that, when executed by a processor, cause the processor to pull positive pairs of the feature maps and the feature vectors closer and push non-matching pairs of the feature maps and the feature vectors apart using the self-supervised contrastive loss function to determine the contrastive loss.