System and method for enhancing machine learning model for audio/video understanding using gated multi-level attention and temporal adversarial training

1. A method comprising:

obtaining, using at least one processor, audio/video content; and

processing, using the at least one processor, the audio/video content with a trained attention-based machine learning model to classify the audio/video content;

wherein processing the audio/video content comprises, using the trained attention-based machine learning model:

generating a global representation of the audio/video content based on the audio/video content;

generating a local representation of the audio/video content based on different portions of the audio/video content; and

combining the global representation of the audio/video content and the local representation of the audio/video content to generate an output representation of the audio/video content, the audio/video content classified based on the output representation.

2. The method of claim 1 , further comprising:

generating multiple matrices based on the audio/video content; and

dividing each of the multiple matrices into multiple attention heads;

wherein the global and local representations of the audio/video content are based on the attention heads.

3. The method of claim 2 , wherein generating the global representation of the audio/video content comprises:

for each of multiple collections of attention heads, using the attention heads to generate a global attention map and generate a global output representation of the audio/video content based on the global attention map;

combining the global output representations of the audio/video content across the collections of attention heads to generate a combined global output representation of the audio/video content;

multiplying the combined global output representation of the audio/video content with a first learnable matrix to generate the global representation of the audio/video content; and

multiplying the combined global output representation of the audio/video content with a second learnable matrix to generate a first weight matrix.

4. The method of claim 3 , wherein generating the local representation of the audio/video content comprises:

for each of the multiple collections of attention heads:

dividing each of the attention heads into multiple chunks in a temporal domain, the chunks of each attention head associated with the different portions of the audio/video content;

using the chunks of the attention heads to generate local attention maps and generate first local output representations of the different portions of the audio/video content based on the local attention maps; and

combining the first local output representations across the temporal domain to generate a second local output representation of the audio/video content;

combining the second local output representations of the audio/video content across the collections of attention heads to generate a combined local output representation of the audio/video content;

multiplying the combined local output representation of the audio/video content with a third learnable matrix to generate the local representation of the audio/video content; and

multiplying the combined local output representation of the audio/video content with a fourth learnable matrix to generate a second weight matrix.

5. The method of claim 4 , wherein combining the global representation of the audio/video content and the local representation of the audio/video content to generate the output representation of the audio/video content comprises:

using the first and second weight matrices to generate a global weight matrix for the global representation of the audio/video content and a local weight matrix for the local representation of the audio/video content;

multiplying the global representation of the audio/video content with the global weight matrix to generate a weighted global representation of the audio/video content;

multiplying the local representation of the audio/video content with the local weight matrix to generate a weighted local representation of the audio/video content; and

combining the weighted global representation of the audio/video content and the weighted local representation of the audio/video content to generate the output representation of the audio/video content.

6. The method of claim 1 , wherein:

the audio/video content comprises first content provided to a user; and

the method further comprises identifying or providing second content to the user based on a classification of the audio/video content.

7. The method of claim 1 , wherein the different portions of the audio/video content have different lengths.

8. The method of claim 1 , wherein the trained attention-based machine learning model is trained by:

obtaining training samples;

generating adversarial counterpart samples for the training samples; and

training the attention-based machine learning model using the training samples and the adversarial counterpart samples to (i) reduce or minimize differences between attention maps generated by the attention-based machine learning model for the training samples and the adversarial counterpart samples and (ii) reduce or minimize differences between output representations generated by the attention-based machine learning model for the training samples and the adversarial counterpart samples.

9. An apparatus comprising:

at least one processor configured to:

obtain audio/video content; and

process the audio/video content with a trained attention-based machine learning model to classify the audio/video content;

wherein, to process the audio/video content, the at least one processor is configured to use the trained attention-based machine learning model to:

generate a global representation of the audio/video content based on the audio/video content;

generate a local representation of the audio/video content based on different portions of the audio/video content; and

combine the global representation of the audio/video content and the local representation of the audio/video content to generate an output representation of the audio/video content, the audio/video content classified based on the output representation.

10. The apparatus of claim 9 , wherein the at least one processor is further configured to:

generate multiple matrices based on the audio/video content; and

divide each of the multiple matrices into multiple attention heads, the global and local representations of the audio/video content based on the attention heads.

11. The apparatus of claim 10 , wherein, to generate the global representation of the audio/video content, the at least one processor is configured to:

for each of multiple collections of attention heads, use the attention heads to generate a global attention map and generate a global output representation of the audio/video content based on the global attention map;

combine the global output representations of the audio/video content across the collections of attention heads to generate a combined global output representation of the audio/video content;

multiply the combined global output representation of the audio/video content with a first learnable matrix to generate the global representation of the audio/video content; and

multiply the combined global output representation of the audio/video content with a second learnable matrix to generate a first weight matrix.

12. The apparatus of claim 11 , wherein, to generate the local representation of the audio/video content, the at least one processor is configured to:

for each of the multiple collections of attention heads:

divide each of the attention heads into multiple chunks in a temporal domain, the chunks of each attention head associated with the different portions of the audio/video content;

use the chunks of the attention heads to generate local attention maps and generate first local output representations of the different portions of the audio/video content based on the local attention maps; and

combine the first local output representations across the temporal domain to generate a second local output representation of the audio/video content;

combine the second local output representations of the audio/video content across the collections of attention heads to generate a combined local output representation of the audio/video content;

multiply the combined local output representation of the audio/video content with a third learnable matrix to generate the local representation of the audio/video content; and

multiply the combined local output representation of the audio/video content with a fourth learnable matrix to generate a second weight matrix.

13. The apparatus of claim 12 , wherein, to combine the global representation of the audio/video content and the local representation of the audio/video content to generate the output representation of the audio/video content, the at least one processor is configured to:

use the first and second weight matrices to generate a global weight matrix for the global representation of the audio/video content and a local weight matrix for the local representation of the audio/video content;

multiply the global representation of the audio/video content with the global weight matrix to generate a weighted global representation of the audio/video content;

multiply the local representation of the audio/video content with the local weight matrix to generate a weighted local representation of the audio/video content; and

combine the weighted global representation of the audio/video content and the weighted local representation of the audio/video content to generate the output representation of the audio/video content.

14. The apparatus of claim 9 , wherein:

the audio/video content comprises first content provided to a user; and

the at least one processor is further configured to identify or provide second content to the user based on a classification of the audio/video content.

15. The apparatus of claim 9 , wherein the different portions of the audio/video content have different lengths.

16. The apparatus of claim 9 , wherein the trained attention-based machine learning model is trained by:

obtaining training samples;

generating adversarial counterpart samples for the training samples; and

training the attention-based machine learning model using the training samples and the adversarial counterpart samples to (i) reduce or minimize differences between attention maps generated by the attention-based machine learning model for the training samples and the adversarial counterpart samples and (ii) reduce or minimize differences between output representations generated by the attention-based machine learning model for the training samples and the adversarial counterpart samples.

17. A non-transitory computer readable medium containing instructions that when executed cause at least one processor to:

obtain audio/video content; and

process the audio/video content with a trained attention-based machine learning model to classify the audio/video content;

wherein the instructions that when executed cause the at least one processor to process the audio/video content comprise instructions that when executed cause the at least one processor to:

generate a global representation of the audio/video content based on the audio/video content;

generate a local representation of the audio/video content based on different portions of the audio/video content; and

combine the global representation of the audio/video content and the local representation of the audio/video content to generate an output representation of the audio/video content, the audio/video content classified based on the output representation.

18. The non-transitory computer readable medium of claim 17 , further containing instructions that when executed cause the at least one processor to:

generate multiple matrices based on the audio/video content; and

divide each of the multiple matrices into multiple attention heads, the global and local representations of the audio/video content based on the attention heads.

19. The non-transitory computer readable medium of claim 18 , wherein the instructions that when executed cause the at least one processor to generate the global representation of the audio/video content comprise instructions that when executed cause the at least one processor to:

for each of multiple collections of attention heads, use the attention heads to generate a global attention map and generate a global output representation of the audio/video content based on the global attention map;

combine the global output representations of the audio/video content across the collections of attention heads to generate a combined global output representation of the audio/video content;

multiply the combined global output representation of the audio/video content with a first learnable matrix to generate the global representation of the audio/video content; and

multiply the combined global output representation of the audio/video content with a second learnable matrix to generate a first weight matrix.

20. The non-transitory computer readable medium of claim 19 , wherein the instructions that when executed cause the at least one processor to generate the local representation of the audio/video content comprise instructions that when executed cause the at least one processor to:

for each of the multiple collections of attention heads:

divide each of the attention heads into multiple chunks in a temporal domain, the chunks of each attention head associated with the different portions of the audio/video content;

use the chunks of the attention heads to generate local attention maps and generate first local output representations of the different portions of the audio/video content; and

combine the first local output representations across the temporal domain to generate a second local output representation of the audio/video content;

combine the second local output representations of the audio/video content across the collections of attention heads to generate a combined local output representation of the audio/video content;

multiply the combined local output representation of the audio/video content with a third learnable matrix to generate the local representation of the audio/video content; and

multiply the combined local output representation of the audio/video content with a fourth learnable matrix to generate a second weight matrix.

21. The non-transitory computer readable medium of claim 20 , wherein the instructions that when executed cause the at least one processor to combine the global representation of the audio/video content and the local representation of the audio/video content to generate the output representation of the audio/video content comprise instructions that when executed cause the at least one processor to:

use the first and second weight matrices to generate a global weight matrix for the global representation of the audio/video content and a local weight matrix for the local representation of the audio/video content;

multiply the global representation of the audio/video content with the global weight matrix to generate a weighted global representation of the audio/video content;

multiply the local representation of the audio/video content with the local weight matrix to generate a weighted local representation of the audio/video content; and

combine the weighted global representation of the audio/video content and the weighted local representation of the audio/video content to generate the output representation of the audio/video content.

22. A method comprising:

obtaining training samples;

generating adversarial counterpart samples for the training samples; and

training an attention-based machine learning model to (i) generate a global representation of audio/video content based on the audio/video content, (ii) generate a local representation of the audio/video content based on different portions of the audio/video content, and (iii) combine the global representation of the audio/video content and the local representation of the audio/video content to generate an output representation of the audio/video content, the audio/video content classified based on the output representation;

wherein the attention-based machine learning model is trained using the training samples and the adversarial counterpart samples to (i) reduce or minimize differences between attention maps generated by the attention-based machine learning model for the training samples and the adversarial counterpart samples and (ii) reduce or minimize differences between output representations generated by the attention-based machine learning model for the training samples and the adversarial counterpart samples.

23. The method of claim 22 , wherein training the attention-based machine learning model comprises using an adversarial loss function that includes an attention map-based regularization term.

24. The method of claim 23 , wherein training the attention-based machine learning model comprises:

performing a first forward pass using the attention-based machine learning model to generate the attention maps for the training samples and determine a first loss representing a cross-entropy loss associated with the training samples;

performing a second forward pass using the attention-based machine learning model to generate the attention maps for the adversarial counterpart samples;

determining a second loss based on the attention map-based regularization term;

determining a third loss representing a cross-entropy loss associated with the adversarial counterpart samples;

determining a final loss by combining the first, second, and third losses; and

performing back propagation to update one or more model parameters of the attention-based machine learning model based on the final loss.

25. The method of claim 23 , wherein the attention map-based regularization term comprises at least one of:

a distance measure; and

a divergence measure.

26. The method of claim 22 , wherein:

the training samples comprise audio samples and video samples;

the adversarial counterpart samples comprise adversarial counterpart audio samples and adversarial counterpart video samples;

the attention-based machine learning model comprises separate encoders for the audio samples and the video samples; and

the encoders are trained during the training of the attention-based machine learning model.

27. The method of claim 22 , wherein training the attention-based machine learning model comprises using an adversarial loss function that includes attention map-based regularization terms associated with global and local attention maps generated using the attention-based machine learning model.