Speech synthesis utilizing audio waveform difference signal(s)
Techniques are disclosed that enable generation of an audio waveform representing synthesized speech based on a difference signal determined using an autoregressive model. Various implementations include using a distribution of the difference signal values to represent sounds found in human speech with a higher level of granularity than sounds not frequently found in human speech. Additional or alternative implementations include using one or more speakers of a client device to render the generated audio waveform.
1. One or more non-transitory storage media encoded with instructions that when executed by a computing device cause the computing device to perform operations comprising:
iteratively generating samples of an audio waveform that is synthesized audio of provided data, wherein generating the samples of the audio waveform comprises:
at each iteration of a plurality of sequential iterations:
generating a respective difference signal for the iteration using an autoregressive model, wherein the respective difference signal is a predicted difference based on a feature of a respective preceding sample of the audio waveform generated in an immediately preceding iteration and a feature of a respective sample for the iteration, wherein an input to the autoregressive model comprises:
a respective representation of at least part of the provided data,
the respective preceding sample of the audio waveform generated in the immediately preceding iteration of the sequential iterations, and
a respective preceding difference signal generated in the immediately preceding iteration; and
determining the respective sample for the iteration using the respective difference signal for the iteration and the respective preceding sample of the audio waveform generated in the immediately preceding iteration, the respective sample for the iteration being one of the samples of the audio waveform; and
causing a client device to render the audio waveform by rendering the samples of the audio waveform.
2. The non-transitory storage media of claim 1 , wherein computing device is the client device, and wherein the audio waveform is rendered using a speaker of the client device.
3. The non-transitory storage media of claim 1 , the operations further comprise:
determining that one or more conditions of the computing device are satisfied; and
in response to determining that the one or more conditions are satisfied:
determining to utilize the autoregressive model to generate the audio waveform based on difference signals generated using the autoregressive model, instead of utilizing an alternative autoregressive model that is more resource intensive to utilize than the autoregressive model.
4. The non-transitory storage media of claim 3 , wherein the one or more conditions of the computing device include the computing device being powered by a battery which is not fully charged.
5. The non-transitory storage media of claim 3 , wherein the one or more conditions of the computing device include one or more processors of the computing device being throttled by heat.
6. The non-transitory storage media of claim 1 , wherein the computing device is a server that is remote from the client device, and wherein causing the client device to render the audio waveform comprises transmitting the samples of the audio waveform from the server to the client device.
7. The non-transitory storage media of claim 1 , wherein the autoregressive model is a recurrent neural network model.
8. The non-transitory storage media of claim 1 , wherein the difference signal generated for the iteration has a smaller size than the respective sample of the audio waveform of the iteration.
9. The non-transitory storage media of claim 1 , wherein the difference signal is a discrete value selected from a difference signal distribution.
10. The non-transitory storage media of claim 9 , wherein the difference signal distribution is a log uniform distribution.
11. The non-transitory storage media of claim 9 , wherein the difference signal distribution includes discrete values, wherein a count of the discrete values is smaller than a threshold.
12. The non-transitory storage media of claim 9 , wherein the difference signal distribution includes at least a first difference signal value and a second difference signal value,
wherein the first difference signal value represents a change in sound corresponding to a sound feature not found in a target audio type, or found in the target audio type with less than a first threshold frequency,
wherein the second difference signal value represents a change in sound found in the target audio type, or found in the target audio type with greater than a second threshold frequency, and
wherein the change in sound represented by the first difference signal value is greater than the change in sound represented by the second difference signal value.
13. The non-transitory storage media of claim 9 , wherein the difference signal distribution excludes a difference signal value representing a sound feature not found in a target audio type, or found in the target audio type with less than a threshold frequency.
14. A client device comprising:
one or more processors;
one or more speakers;
a memory storing an autoregressive model;
wherein the one or more processors are configured to execute instructions that cause a computing system of the client device to perform operations comprising:
iteratively generating samples of an audio waveform that is synthesized audio of provided data, wherein generating the samples of the audio waveform comprises:
at each iteration of a plurality of sequential iterations:
generating a respective difference signal for the iteration using the autoregressive model, wherein the respective difference signal is a predicted difference based on a feature of a respective preceding sample of the audio waveform generated in an immediately preceding iteration and a feature of a respective sample for the iteration, wherein an input to the autoregressive model comprises:
a respective representation of at least part of the provided data,
the respective preceding sample of the audio waveform generated in the immediately preceding iteration of the sequential iterations, and
a respective preceding difference signal generated in the immediately preceding iteration; and
determining the respective sample for the iteration using the respective difference signal for the iteration and the respective preceding sample of the audio waveform generated in the immediately preceding iteration, the respective sample for the iteration being one of the samples of the audio waveform; and
rendering the audio waveform by rendering the samples of the audio waveform.
15. The client device of claim 14 , wherein the audio waveform is rendered using the one or more speakers of the client device.
16. The client device of claim 14 , the operations further comprise:
determining that one or more conditions of the client device are satisfied; and
in response to determining that the one or more conditions are satisfied:
determining to utilize the autoregressive model to generate the audio waveform based on difference signals generated using the autoregressive model, instead of utilizing an alternative autoregressive model that is more resource intensive to utilize than the autoregressive model.
17. The client device of claim 16 , wherein the one or more conditions of the client device include the client device being powered by a battery which is not fully charged.
18. The client device of claim 16 , wherein the one or more conditions of the client device include the one or more processors of the client device being throttled by heat.
19. The client device of claim 14 , wherein the difference signal generated for the iteration has a smaller size than the respective sample of the audio waveform of the iteration.
20. A method implemented by one or more processors, the method comprising:
iteratively generating samples of an audio waveform that is synthesized audio of provided data, wherein generating the samples of the audio waveform comprises:
at each iteration of a plurality of sequential iterations:
generating a respective difference signal for the iteration using an autoregressive model, wherein the respective difference signal is a predicted difference based on a feature of a respective preceding sample of the audio waveform generated in an immediately preceding iteration and a feature of a respective sample for the iteration, wherein an input to the autoregressive model comprises:
a respective representation of at least part of the provided data,
the respective preceding sample of the audio waveform generated in the immediately preceding iteration of the sequential iterations, and
a respective preceding difference signal generated in the immediately preceding iteration; and
determining the respective sample for the iteration using the respective difference signal for the iteration and the respective preceding sample of the audio waveform generated in the immediately preceding iteration, the respective sample for the iteration being one of the samples of the audio waveform; and
causing a client device to render the audio waveform by rendering the samples of the audio waveform.