Modifying an audio signal to incorporate a natural-sounding intonation
Techniques for modifying intonations in audio files are disclosed. A first audio waveform and a second audio waveform are accessed. A first intonation in the first audio waveform is identified, and a second intonation in the second audio waveform is identified. The first and second intonations correspond to the same unit of pronunciation. The first intonation is modified until it sufficiently matches the second intonation.
1 . A method for dynamically modifying intonations in a first audio waveform to match intonations that are detected in a second audio waveform, said method comprising:
accessing a first audio waveform representing output from a text-to-speech generator operating on a source of text, wherein the text-to-speech generator is configured to generate the first audio waveform based on a source of text and a set of parameters comprising a rate characteristic, a pitch characteristic, a volume characteristic, and a break characteristic;
accessing a second audio waveform representing a recording of a human reading the source of text;
identifying a first set of intonations embodied within the first audio waveform, the first set of intonations being enunciated for syllables within the source of text;
identifying a second set of intonations embodied within the second waveform, the second set of intonations being enunciated for the same syllables;
for each respective syllable in said syllables, detecting a corresponding matching pair of intonations, wherein said corresponding matching pair of intonations includes an intonation from the first set of intonations for said each respective syllable and an intonation from the second set of intonations for said each respective syllable; and
for each respective syllable in said syllables, modifying, using a frequency-domain transformation or programmatically generated markup commands, said each respective syllable's corresponding matching pair of intonations by causing that matching pair's intonation from the first set of intonations to match, within a predefined threshold, that matching pair's intonation from the second set of intonations, and wherein said modifying includes adjusting one or more of the set of parameters comprising the rate characteristic, the pitch characteristic, the volume characteristic, and the break characteristic.
2 . The method of claim 1 , wherein the method is performed by a service.
3 . The method of claim 2 , wherein the service includes one or more of a machine learning engine or a generative pre-trained model.
4 . The method of claim 1 , wherein said modifying includes converting the first waveform to a frequency domain using the frequency-domain transformation and modifying frequency characteristics of the first waveform in the frequency domain.
5 . The method of claim 1 , wherein said modifying includes using the programmatically generated markup commands to mark up the source of text.
6 . The method of claim 5 , wherein, as a result of marking up the source of text, the text-to-speech generator is caused to read the source of text, which now includes programmatically altered language.
7 . The method of claim 1 , wherein the source of text is generated by a speech-to-text generator.
8 . A computer system that dynamically modifies intonations in a first audio waveform to match intonations that are detected in a second audio waveform, said computer system comprising:
a processor system; and
a storage system comprising instructions that are executable by the processor system to cause the computer system to:
access a first audio waveform representing output from a text-to-speech generator operating on a source of text, wherein the text-to-speech generator is configured to generate the first audio waveform based on a source of text and a set of parameters comprising a rate characteristic, a pitch characteristic, a volume characteristic, and a break characteristic;
access a second audio waveform;
identify a first intonation embodied within the first audio waveform, the first intonation being associated with a unit of pronunciation included in the first audio waveform;
identify a second intonation embodied within the second waveform, wherein the second intonation is associated with the same unit of pronunciation, which is also included in the second audio waveform; and
modify, using a frequency-domain transformation or programmatically generated markup commands, the first audio waveform by modifying the first intonation of the first audio waveform until the first intonation matches, in accordance with a pre-defined tolerance, the second intonation from the second audio waveform, and wherein said modifying includes adjusting one or more of the set of parameters comprising the rate characteristic, the pitch characteristic, the volume characteristic, and the break characteristic.
9 . The computer system of claim 8 , wherein execution of the instructions further causes the computer system to perform a pre-processing operation, a real-time operation, or a post-processing operation.
10 . The computer system of claim 8 , wherein the first audio waveform is generated in real-time.
11 . The computer system of claim 8 , wherein the second audio waveform is a pre-saved audio waveform stored in a repository of waveforms.
12 . The computer system of claim 8 , wherein the first audio waveform is output from a text-to-speech generator.
13 . The computer system of claim 8 , wherein the instructions are further executable by the computer system to:
use a speech-to-text generator to generate a transcript of a human who is speaking; and
feed the transcript as input to the text-to-speech generator to generate the first audio waveform.
14 . The computer system of claim 8 , wherein the first audio waveform corresponds to a source of text that includes between 1 and 50 words.
15 . The computer system of claim 8 , wherein the first audio waveform corresponds to a source of text that includes more than 10 words.
16 . A method for dynamically modifying intonations in a first audio waveform to match intonations that are detected in a second audio waveform, said method comprising:
accessing a first audio waveform representing output from a text-to-speech generator operating on a source of text, wherein the text-to-speech generator is configured to generate the first audio waveform based on a source of text and a set of parameters comprising a rate characteristic, a pitch characteristic, a volume characteristic, and a break characteristic;
accessing a second audio waveform;
identifying a first intonation embodied within the first audio waveform, the first intonation being associated with a unit of pronunciation included in the first audio waveform;
identifying a second intonation embodied within the second waveform, wherein the second intonation is associated with the same unit of pronunciation, which is also included in the second audio waveform; and
modifying, using a frequency-domain transformation or programmatically generated markup commands, the first audio waveform by modifying the first intonation of the first audio waveform until the first intonation matches, in accordance with a pre-defined tolerance, the second intonation from the second audio waveform, and wherein said modifying includes adjusting one or more of the set of parameters comprising the rate characteristic, the pitch characteristic, the volume characteristic, and the break characteristic.
17 . The method of claim 16 , wherein the first waveform and the second waveform are associated with the source of text.
18 . The method of claim 16 , wherein the pre-defined tolerance is between 0% and 5%.
19 . The method of claim 16 , wherein the pre-defined tolerance is less than about 5%.