Natural language-guided music audio recommendation for video using machine learning

1 . A computer-implemented method, comprising:

receiving an input, the input including a query video sequence and natural language text, the natural language text including multiple parameters for different portions of a recommended music audio sequence for pairing with the query video sequence, the parameters including a description of one or more of: a mood, a genre, and instrumentation;

concatenating a visual embedding corresponding to the query video sequence and a text embedding corresponding to the natural language text to generate a concatenated embedding;

generating a fused visual-text embedding by processing the concatenated embedding through a transformer-based model;

comparing audio embeddings for music audio sequences of a music audio sequences database with the fused visual-text embedding;

determining a first music audio sequence from the music audio sequences database as the recommended music audio sequence for pairing with the query video sequence based on a similarity metric calculated between a first audio embedding for the first music audio sequence and the fused visual-text embedding being above a threshold value; and

outputting a media sequence, wherein the media sequence is a combination of the query video sequence and the first music audio sequence.

2 . The computer-implemented method of claim 1 , further comprising:

generating the visual embedding for the query video sequence by:

extracting, for each frame of the query video sequence, base visual features, and

generating, by a visual transformer, the visual embedding for the query video sequence using the extracted base visual features.

3 . The computer-implemented method of claim 1 , further comprising:

generating the text embedding for the natural language text further by:

extracting base text features from the natural language text, and

generating, by a text transformer, the text embedding for the natural language text using the extracted base text features.

4 . The computer-implemented method of claim 1 , further comprising:

for each music audio sequence in the music audio sequences database:

extracting, for each frame of a music audio sequence, base audio features,

generating, by an audio transformer, an audio embedding for the music audio sequence using the extracted base audio features, and

storing the audio embedding in the music audio sequences database.

5 . The computer-implemented method of claim 1 , wherein the natural language text includes first parameters for a first portion of the recommended music audio sequence for pairing with the query video sequence and second parameters for a second portion of the recommended music audio sequence for pairing with the query video sequence.

6 . A non-transitory computer-readable storage medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:

receiving an input, the input including a query video sequence and natural language text, the natural language text including multiple parameters for different portions of a recommended music audio sequence for pairing with the query video sequence, the parameters including a description of one or more of: a mood, a genre, and instrumentation;

concatenating a visual embedding corresponding to the query video sequence and a text embedding corresponding to the natural language text to generate a concatenated embedding;

generating a fused visual-text embedding by processing the concatenated embedding through a transformer-based model;

determining a first music audio sequence from a music audio sequence database as the recommended music audio sequence for pairing with the query video sequence based on a similarity metric calculated between a first audio embedding for the first music audio sequence and the fused visual-text embedding generated from the query video sequence and the natural language text being above a threshold value; and

outputting a media sequence, wherein the media sequence is a combination of the query video sequence and the first music audio sequence.

7 . The non-transitory computer-readable storage medium of claim 6 , wherein the instructions further cause the processing device to perform operations comprising:

generating a visual embedding for the query video sequence by:

extracting, for each frame of the query video sequence, base visual features, and

generating, by a visual transformer, the visual embedding for the query video sequence using the extracted base visual features;

generating a text embedding for the natural language text further by:

extracting base text features from the natural language text, and

generating, by a text transformer, the text embedding for the natural language text using the extracted base text features; and

generating, by a transformer, the fused visual-text embedding using the visual embedding and the text embedding.

8 . The non-transitory computer-readable storage medium of claim 6 , wherein the instructions further cause the processing device to perform operations comprising:

for each music audio sequence in the music audio sequences database:

extracting, for each frame of a music audio sequence, base audio features, and

generating, by an audio transformer, an audio embedding for the music audio sequence using the extracted base audio features.

9 . The non-transitory computer-readable storage medium of claim 6 , wherein the natural language text includes first parameters for a first portion of the recommended music audio sequence for pairing with the query video sequence and second parameters for a second portion of the recommended music audio sequence for pairing with the query video sequence.