Neural pitch-shifting and time-stretching
Methods for modifying audio data include operations for accessing audio data having a first prosody, receiving a target prosody differing from the first prosody, and computing acoustic features representing samples. Computing respective acoustic features for a sample includes computing a pitch feature as a quantized pitch value of the sample by assigning a pitch value, of the target prosody or the audio data, to at least one of a set of pitch bins having equal widths in cents. Computing the respective acoustic features further includes computing a periodicity feature from the audio data. The respective acoustic features for the sample include the pitch feature, the periodicity feature, and other acoustic features. A neural vocoder is applied to the acoustic features to pitch-shift and time-stretch the audio data from the first prosody toward the target prosody.
1. A method in which one or more processing devices perform operations comprising:
identifying, using a transcript of a speaker, audio data from an edit region of an audio recording of the speaker as distinct from an unedited audio portion of the audio recording, the audio data having a first prosody;
applying a phoneme duration model to the audio data to predict phoneme durations;
applying, using the phoneme durations, a pitch generation model to the audio data to provide a target prosody for the audio data, wherein the target prosody differs from the first prosody;
computing acoustic features representing samples, wherein computing respective acoustic features for a sample of the samples comprises:
computing a pitch feature as a quantized pitch value of the sample by assigning a pitch value of at least one of the target prosody or the audio data to at least one of a set of pitch bins, wherein each pitch bin of the set of pitch bins has an equal width in cents; and
computing, from the audio data, a periodicity feature and additional acoustic features for the sample,
wherein the respective acoustic features for the sample comprise the pitch feature, the periodicity feature, and the additional acoustic features;
applying a neural vocoder to the acoustic features to pitch-shift and time-stretch the audio data from the first prosody toward the target prosody to produce edited audio data; and
combining the edited audio data with the unedited audio portion of the audio recording.
2. The method of claim 1 , wherein computing the pitch feature and the periodicity feature further comprises applying a prediction model to the audio data.
3. The method of claim 1 , the operations further comprising dithering the pitch feature with random noise from a triangular distribution.
4. The method of claim 1 , wherein computing the pitch feature further comprises using a quantization of a particular frequency range that is equally spaced in base-2 log scale.
5. The method of claim 1 , wherein the neural vocoder is configured to:
generate a probability distribution of possible excitation values for the sample; and
determine a synthesized output sample by sampling the probability distribution using a constant sampling temperature.
6. The method of claim 5 , wherein computing the additional acoustic features for the sample comprises encoding the audio data corresponding to the sample as Bark-frequency cepstral coefficients (BFCCs) for input to the neural vocoder.
7. The method of claim 6 , the operations further comprising:
computing a prediction value via linear predictive coding (LPC), wherein LPC coefficients are derived from the BFCCs,
wherein the neural vocoder is further configured to combine the prediction value with a result of sampling the probability distribution to determine the synthesized output sample.
8. The method of claim 1 , the operations further comprising training the neural vocoder to map the acoustic features to an output audio signal.
9. The method of claim 8 , wherein training the neural vocoder comprises:
accessing a training dataset for the neural vocoder; and
performing resampling data augmentation on the training dataset using a resampling function dependent on one or more values of a pitch shift factor.
10. The method of claim 8 , wherein the speaker of the audio data is distinct from a set of speakers on which the neural vocoder is trained.
11. A system comprising:
a feature extraction subsystem configured to:
receive a transcript of a speaker and audio data from an edit region of an audio recording of the speaker as distinct from an unedited portion of the audio recording
apply a phoneme duration model to the audio data to predict phoneme durations;
apply, using the phoneme durations, a pitch generation model to the audio data to provide a target prosody for the audio data, the target prosody including a target pitch contour; and
compute acoustic features representing samples, wherein respective acoustic features for a sample of the samples comprise (i) a pitch feature representing at least one of the target pitch contour or the audio data and (ii) a periodicity feature and cepstral coefficients representing the audio data;
a neural vocoder configured to:
input the acoustic features for the sample;
generate a probability distribution of possible excitation values for the sample based on the acoustic features; and
determine a synthesized output sample by sampling the probability distribution using a constant sampling temperature; and
an operations subsystem configured to provide output audio data including the synthesized output sample, the output audio data configured to be combined with the unedited portion of the audio recording.
12. The system of claim 11 , wherein the feature extraction subsystem is further configured to compute the pitch feature by quantizing a pitch value corresponding to the sample by assigning the pitch value to at least one of a set of pitch bins, wherein each pitch bin of the set of pitch bins has an equal width in cents.
13. The system of claim 11 , wherein the neural vocoder is configured to pitch-shift and time-stretch the audio data from a first prosody toward the target prosody.
14. The system of claim 11 , wherein the neural vocoder comprises:
a frame-rate network configured to generate an embedding representing the samples; and
a sample-rate network configured to compute the probability distribution of possible excitation values.
15. The system of claim 11 , further comprising a training system configured to:
access a dataset for training the neural vocoder;
generate a training dataset by performing resampling data augmentation on the dataset using a resampling function dependent on one or more values of a pitch shift factor; and
train the neural vocoder using the training dataset.
16. The system of claim 15 , wherein the speaker of the audio data is distinct from a set of speakers on which the neural vocoder is trained.
17. A non-transitory computer-readable medium embodying program code for pitch-shifting and time-stretching audio data, the program code comprising instructions that, when executed by a processor, cause the processor to perform operations comprising:
identifying, using a transcript of a speaker, audio data from an edit region of an audio recording of the speaker as distinct from an unedited audio portion of the audio recording, the audio data having a first prosody;
applying a phoneme duration model to the audio data to predict phoneme durations;
applying, using the phoneme durations, a pitch generation model to the audio data to provide a target prosody for the audio data, wherein the target prosody differs from the first prosody;
a step for computing acoustic features representing samples, wherein computing respective acoustic features for a sample of the samples comprises:
computing a pitch feature as a quantized pitch value of the sample by assigning a pitch value of at least one of the target prosody and the audio data to at least one of a set of pitch bins, wherein each pitch bin of the set of pitch bins has an equal width in cents; and
computing, from the audio data, a periodicity feature and additional acoustic features for the sample,
wherein the respective acoustic features for the sample comprise the pitch feature, the periodicity feature, and the additional acoustic features;
a step for applying a neural vocoder to the acoustic features to pitch-shift and time-stretch the audio data from the first prosody toward the target prosody; and
combining the edited audio data with the unedited audio portion of the audio recording.
18. The non-transitory computer-readable medium of claim 17 , wherein computing the pitch feature and the periodicity feature further comprises:
applying a prediction model to the audio data to determine the quantized pitch value; and
dithering the quantized pitch value with random noise.
19. The non-transitory computer-readable medium of claim 17 , wherein the neural vocoder is configured to:
generate a probability distribution of possible excitation values for the sample; and
determine a synthesized output sample by combining (i) a sampling of the probability distribution using a constant sampling temperature and (ii) a prediction value computed via linear predictive coding (LPC) with LPC coefficients derived from cepstral coefficients representing the audio data.
20. The non-transitory computer-readable medium of claim 17 , the operations further comprising:
accessing a dataset for training the neural vocoder;
generating a training dataset by performing resampling data augmentation on the dataset using a resampling function dependent on one or more values of a pitch shift factor; and
training the neural vocoder using the training dataset.