Machine learning based audio watermarking for videoconferencing
Systems and methods for machine learning based audio watermarking for videoconferencing are provided. For example, a computing device accesses an original audio signal and a watermark to be embedded into the original audio signal and extracts, using an audio encoder, a set of audio features from the original audio signal. The audio encoder is a machine learning model. The computing device further extracts, using a watermark encoder, a set of watermark features from the watermark. The watermark encoder is also a machine learning model. The computing device combines the set of audio features and the set of watermark features to generate a set of features, compresses the set of features, and transmits the compressed set of features.
1 . A method performed by a computing device, the method comprising:
accessing an original audio signal and a watermark to be embedded into the original audio signal;
extracting, using an audio encoder, a set of audio features from the original audio signal, wherein the audio encoder is a first machine learning model;
extracting, using a watermark encoder, a set of watermark features from the watermark, wherein the watermark encoder is a second machine learning model;
combining the set of audio features and the set of watermark features to generate a set of features;
compressing the set of features; and
transmitting the compressed set of features.
2 . The method of claim 1 , wherein the audio encoder and the watermark encoder are trained via a training process, the training process comprising:
generating training audio features by applying the audio encoder on a training original audio signal;
generating training watermark features by applying the watermark encoder on a training watermark;
combining the training watermark features with the training audio features to generate combined training features;
generating a training watermarked audio signal by applying an audio decoder on the combined training features;
generating an extracted training watermark by applying a watermark extractor on the training watermarked audio signal or a processed training watermarked audio signal; and
adjusting parameters of the audio encoder, the watermark encoder, the audio decoder, and the watermark extractor to minimize a loss function.
3 . The method of claim 2 , wherein the loss function comprises a first term representing a difference between the training original audio signal and the training watermarked audio signal, a second term representing a difference between the training watermark and the extracted training watermark, and a third term representing a generative adversarial loss defined based on the training original audio signal and the training watermarked audio signal.
4 . The method of claim 2 , wherein the processed training watermarked audio signal is generated by applying on the training watermarked audio signal one or more of additive noise, compression, or a low pass filter.
5 . The method of claim 2 , wherein at least one of the audio encoder, the watermark encoder, the audio decoder, or the watermark extractor is a neural network model.
6 . The method of claim 1 , wherein compressing the set of features comprises quantizing the set of features using vector quantization, and wherein the compressed set of features comprise indices of individual features in the set of features.
7 . The method of claim 1 , further comprising:
receiving a second compressed set of features;
decompressing the second compressed set of features to generate a second set of features;
generating a watermarked audio signal by applying an audio decoder on the second set of features; and
playing the watermarked audio signal.
8 . The method of claim 1 , wherein the watermark is one or more of an audio signal, an image, a text, or a number.
9 . A computing device, comprising:
a non-transitory computer-readable medium; and
a processor communicatively coupled to the non-transitory computer-readable medium, the processor configured to execute processor-executable instructions stored in the non-transitory computer-readable medium to:
access an original audio signal and a watermark to be embedded into the original audio signal;
extract, using an audio encoder, a set of audio features from the original audio signal, wherein the audio encoder is a first machine learning model;
extract, using a watermark encoder, a set of watermark features from the watermark, wherein the watermark encoder is a second machine learning model;
combine the set of audio features and the set of watermark features to generate a set of features;
compress the set of features; and
transmit the compressed set of features.
10 . The computing device of claim 9 , wherein the audio encoder and the watermark encoder are trained via a training process, the training process comprising:
generating training audio features by applying the audio encoder on a training original audio signal;
generating training watermark features by applying the watermark encoder on a training watermark;
combining the training watermark features with the training audio features to generate combined training features;
generating a training watermarked audio signal by applying an audio decoder on the combined training features;
generating an extracted training watermark by applying a watermark extractor on the training watermarked audio signal or a processed training watermarked audio signal; and
adjusting parameters of the audio encoder, the watermark encoder, the audio decoder, and the watermark extractor to minimize a loss function.
11 . The computing device of claim 10 , wherein the loss function comprises a first term representing a difference between the training original audio signal and the training watermarked audio signal, a second term representing a difference between the training watermark and the extracted training watermark, and a third term representing a generative adversarial loss defined based on the training original audio signal and the training watermarked audio signal.
12 . The computing device of claim 10 , wherein the processed training watermarked audio signal is generated by applying on the training watermarked audio signal one or more of additive noise, compression, or a low pass filter.
13 . The computing device of claim 10 , wherein at least one of the audio encoder, the watermark encoder, the audio decoder, or the watermark extractor is a neural network model.
14 . The computing device of claim 9 , wherein compressing the set of features comprises quantizing the set of features using vector quantization, and wherein the compressed set of features comprise indices of individual features in the set of features.
15 . The computing device of claim 9 , wherein the processor is configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to:
receive a second compressed set of features;
decompress the second compressed set of features to generate a second set of features;
generate a watermarked audio signal by applying an audio decoder on the second set of features; and
play the watermarked audio signal.
16 . A non-transitory computer-readable medium comprising processor-executable instructions configured to cause one or more processors to:
access an original audio signal and a watermark to be embedded into the original audio signal;
extract, using an audio encoder, a set of audio features from the original audio signal, wherein the audio encoder is a first machine learning model;
extract, using a watermark encoder, a set of watermark features from the watermark, wherein the watermark encoder is a second machine learning model;
combine the set of audio features and the set of watermark features to generate a set of features;
compress the set of features; and
transmit the compressed set of features.
17 . The non-transitory computer-readable medium of claim 16 , wherein the audio encoder and the watermark encoder are trained via a training process, the training process comprising:
generating training audio features by applying the audio encoder on a training original audio signal;
generating training watermark features by applying the watermark encoder on a training watermark;
combining the training watermark features with the training audio features to generate combined training features;
generating a training watermarked audio signal by applying an audio decoder on the combined training features;
generating an extracted training watermark by applying a watermark extractor on the training watermarked audio signal or a processed training watermarked audio signal; and
adjusting parameters of the audio encoder, the watermark encoder, the audio decoder, and the watermark extractor to minimize a loss function.
18 . The non-transitory computer-readable medium of claim 17 , wherein the loss function comprises a first term representing a difference between the training original audio signal and the training watermarked audio signal, a second term representing a difference between the training watermark and the extracted training watermark, and a third term representing a generative adversarial loss defined based on the training original audio signal and the training watermarked audio signal.
19 . The non-transitory computer-readable medium of claim 17 , wherein the processed training watermarked audio signal is generated by applying on the training watermarked audio signal one or more of additive noise, compression, or a low pass filter.
20 . The non-transitory computer-readable medium of claim 16 , wherein compressing the set of features comprises quantizing the set of features using vector quantization, and wherein the compressed set of features comprise indices of individual features in the set of features.