Training a sound effect recommendation network
A Sound effect recommendation network is trained using a machine learning algorithm with a reference image, a positive audio embedding and a negative audio embedding as inputs to train a visual-to-audio correlation neural network to output a smaller distance between the positive audio embedding and the reference image than the negative audio embedding and the reference image. The visual-to-audio correlation neural network is trained to identify one or more visual elements in the reference image and map the one or more visual elements to one or more sound categories or subcategories within an audio database.
1. A method for training a Sound Effect Recommendation Network, comprising:
a) generating a positive audio embedding from a positive audio signal wherein the positive audio signal is related to a reference image;
b) generating a negative audio embedding from a negative audio signal; and
c) using a machine learning algorithm with the reference image, the positive audio embedding and the negative audio embedding as inputs to train a visual-to-audio correlation neural network to output a smaller distance between the positive audio embedding and the reference image than the negative audio embedding and the reference image, wherein the visual-to-audio correlation neural network is trained to identify one or more visual elements in the reference image and map the one or more visual elements to one or more sound categories or subcategories within an audio database,
wherein a reference audio signal is part of the audio database and wherein the reference audio signal is part of an audio visual sequence having the reference image, and wherein the positive audio signal is the reference audio signal.
2. The method of claim 1 , further comprising determining the negative audio signal using the trained similarity neural network with the audio database wherein the negative signal is not related to the positive audio signal.
3. The method of claim 1 wherein features are extracted from the positive signal, reference signal and negative signal before being used in training with the machine learning algorithm.
4. The method of claim 1 wherein the positive audio signal includes noise.
5. The method of claim 1 wherein the machine learning algorithm used to train the visual-to-audio correlation neural network includes a pairwise loss function.
6. The method of claim 1 wherein the positive signal is part of an audio visual sequence that includes the reference image, wherein the positive signal includes noise signals and wherein the noise signals are other sounds occurring in the audio visual sequence and wherein the negative signal includes noise signals.
7. The method of claim 1 wherein the machine learning algorithm is a self-supervised learning algorithm and wherein the positive, negative and correlated audio are unlabeled or unannotated inputs.
8. A system for training a Sound Effect Recommendation Network, comprising:
a Processor;
a Memory coupled to the Processor;
Non-transitory instructions embedded in the memory that when executed cause the processor to carry out the method for training a sound effect recommendation network comprising;
a) generating a positive audio embedding from a positive audio signal wherein the positive audio signal is related to a reference image;
b) generating a negative audio embedding from a negative audio signal;
c) using a machine learning algorithm with the reference image, the positive audio embedding and the negative audio embedding as inputs to train an image-to-audio correlation neural network to output a smaller distance between the positive audio embedding and the reference image than the negative audio embedding and the reference image, wherein the visual-to-audio correlation neural network is trained to identify one or more visual elements in the reference image and map the one or more visual elements to one or more sound categories or subcategories within an audio database,
wherein a reference audio signal is part of the audio database and wherein the reference audio signal is part of an audio visual sequence having the reference image, and wherein the positive audio signal is the reference audio signal.
9. The system of claim 8 further comprising determining the negative audio signal using the trained similarity neural network with the audio database wherein the negative signal is not related to the reference audio signal.
10. The system of claim 8 wherein the audio features are extracted from the positive signal and negative signal before being used in training with the machine learning algorithm.
11. The system of claim 8 wherein the positive audio signal includes noise.
12. The system of claim 8 wherein the machine learning algorithm used to train the visual-to-audio correlation neural network includes a pairwise loss function.
13. The system of claim 8 wherein the positive signal is part of an audio visual sequence that includes the reference image, wherein positive signal includes noise signals and wherein the noise signals are other sounds occurring in the audio visual sequence and wherein the negative signal includes noise signals.
14. The method of claim 8 wherein the machine learning algorithm is a self-supervised learning algorithm and wherein the positive, negative and correlated audio are unlabeled or unannotated inputs.
15. Non-transitory instructions embedded in a computer readable medium that when executed by a computer cause the computer to carry out a method for training a Sound Recommendation Network comprising:
a) generating a positive audio embedding from a positive audio signal wherein the positive audio signal is related to a reference image;
b) generating a negative audio embedding from a negative audio signal;
c) using a machine learning algorithm with the reference image, the positive audio embedding and a negative audio embedding as inputs to train a visual-to-audio correlation neural network to output a smaller distance between the positive audio embedding and the reference image than the negative audio embedding and the reference image, wherein the visual-to-audio correlation neural network is trained to identify one or more visual elements in the reference image and map the one or more visual elements to one or more sound categories or subcategories within an audio database,
wherein a reference audio signal is part of the audio database and wherein the reference audio signal is part of an audio visual sequence having the reference image, and wherein the positive audio signal is the reference audio signal.