Voice modification
A computing system that receives an audio waveform representing speech from an individual and produces as output a modified version of the audio waveform that maintains the speaker's speech characteristics as well as prosody for specific utterances (e.g., voice timbre, intonation, timing, intensity). The system uses a bottleneck-based autoencoder with speech spectrograms as input and output. To produce the output audio waveform, the system includes a reconstruction error-based loss function with two additional loss functions. The second loss function is speaker “real vs fake” discriminator that penalizes for the output not sounding like the speaker. The third loss function is a speech intelligibility scorer that penalizes the output for speech that is difficult for the target population to understand. The produced modified audio waveform is an enhanced speech output that delivers speech m a target accent without sacrificing the personality of the speaker.
1 . A system comprising:
an input device configured to receive an input audio waveform comprising utterances by a speaker; and
a computation engine comprising processing circuitry for executing a machine learning system,
wherein the machine learning system is configured to implement an autoencoder, the autoencoder comprising a decoder trained to characterize an accent for a target accent, the decoder configured to modify the utterances by the speaker based on the target accent and a differential weighting between a speaker identification (ID) loss function that applies a penalty based on an identity for the speaker and a text loss function that applies a penalty for loss of intelligibility,
wherein the machine learning system is configured to process the input audio waveform to generate an output audio waveform that includes the modified utterances, and
wherein the machine learning system is configured to output the output audio waveform.
2 . The system of claim 1 ,
wherein the autoencoder is configured to encode the input audio waveform based at least in part on derived features of the utterances by the speaker, and
wherein the derived features comprise one or more of pitch, emphasis, phonemes, energy or pauses.
3 . The system of claim 1 ,
wherein the machine learning system is configured to receive a user input, and
wherein the machine learning system is configured to, in response to the user input, adjust the differential weighting between the text loss function and the speaker identification loss function.
4 . The system of claim 3 ,
wherein the target accent is a first target accent, and the output audio waveform is a first output audio waveform,
wherein the decoder is further trained using constraints for a second target accent, wherein the constraints for the second target accent are different from the constraints for the first target accent, and
wherein, in response to user input, the machine learning system is configured to output a second output audio waveform different from the first output audio waveform based on the constraints for the second target accent.
5 . The system of claim 1 , wherein the machine learning system is configured to generate the output audio waveform to adjust intelligibility of utterances by the speaker based on the target accent and retain recognizable individual language characteristics of the speaker.
6 . The system of claim 5 ,
wherein the machine learning system is configured to generate the output audio waveform to retain recognizable individual language characteristics of the speaker, and
wherein the speaker ID loss function is configured to apply the penalty for misidentifying the speaker.
7 . The system of claim 1 ,
wherein to conform to the target accent, the machine learning system is configured to generate the output audio waveform to disguise recognizable individual language characteristics of the speaker, and
wherein the speaker ID loss function is configured to apply the penalty for correctly identifying the speaker.
8 . The system of claim 1 ,
wherein the processing circuitry further executes a converter, and
wherein the converter is configured to receive the input audio waveform and provide an output to the autoencoder, the output comprising a spectrogram.
9 . The system of claim 1 , wherein the autoencoder comprises a neural network.
10 . The system of claim 1 , wherein the autoencoder is configured to encode and decode the input audio waveform at least in part at a phoneme trigram level.
11 . The system of claim 1 , wherein the decoder is further configured to modify the utterances by the speaker based on a signal loss function.
12 . The system of claim 1 ,
wherein at least one of the speaker ID loss function or the text loss function is a separate, pre-trained loss function, and
wherein the decoder is trained based on the at least one separate, pre-trained loss function.
13 . The system of claim 1 , wherein the autoencoder is trained along with the speaker ID loss function or the text loss function as an adversarial network.
14 . The system of claim 1 , wherein the decoder is a first decoder of a plurality of decoders stored at a decoder library operatively coupled to the machine learning system.
15 . The system of claim 14 , wherein the computation engine is configured to:
receive user input indicating a desired accent,
select a decoder of the plurality of decoders that characterizes the desired accent, and
apply the selected decoder to generate the output audio waveform.
16 . A method comprising:
processing, with an encoder of a machine learning system, an input audio waveform comprising first utterances by a speaker to generate an encoder output, the first utterances having a first accent;
processing, with a decoder of the machine learning system, the encoder output to generate an output audio waveform comprising second utterances, the second utterances having a second accent different from the first accent;
computing, with an identification loss discriminator of the machine learning system, based on the input audio waveform and the output audio waveform, an identification loss for the output audio waveform;
computing, with a text loss discriminator of the machine learning system, based on the input audio waveform and the output audio waveform, a text loss for the output audio waveform; and
training the decoder using the identification loss and the text loss, wherein the trained decoder is configured to modify speaker utterances based on a target accent and a differential weighting between a speaker identification loss function that applies a penalty based on a speaker identity and a text loss function that applies a penalty for loss of intelligibility.
17 . The method of claim 16 , further comprising encoding, by the machine learning system the input audio waveform based at least in part on derived features of the first utterances by the speaker, wherein the derived features comprise one or more of: pitch, emphasis, phonemes and pauses.
18 . The method of claim 17 , wherein the decoder is a first decoder of a plurality of decoders stored at a decoder library operatively coupled to the machine learning system, the method further comprising, selecting, by the machine learning system, a second decoder from the decoder library based on input from a user.
19 . A non-transitory computer-readable storage medium comprising instructions that, when executed, cause one or more processors of a computing device to:
receive an input audio waveform comprising utterances by a speaker; and
execute a machine learning system,
wherein the machine learning system is configured to implement an autoencoder, the autoencoder comprising a decoder trained to characterize an accent for a target accent, the decoder configured to modify the utterances by the speaker based on the target accent and a differential weighting between a speaker identification (ID) loss function that applies a penalty based on an identity for the speaker and a text loss function that applies a penalty for loss of intelligibility,
wherein the machine learning system is configured to process the input audio waveform to generate an output audio waveform that includes the modified utterances, and
wherein the machine learning system is configured to output the output audio waveform.
20 . The non-transitory computer-readable storage medium of claim 19 , further comprising instructions for causing the one or more processors to generate the output audio waveform to:
adjust intelligibility of utterances by the speaker based on the target accent; and
retain recognizable individual language characteristics of the speaker.