On-device speech synthesis of textual segments for training of on-device speech recognition model
Processor(s) of a client device can: identify a textual segment stored locally at the client device; process the textual segment, using a speech synthesis model stored locally at the client device, to generate synthesized speech audio data that includes synthesized speech of the identified textual segment; process the synthesized speech, using an on-device speech recognition model that is stored locally at the client device, to generate predicted output; and generate a gradient based on comparing the predicted output to ground truth output that corresponds to the textual segment. In some implementations, the generated gradient is used, by processor(s) of the client device, to update weights of the on-device speech recognition model. In some implementations, the generated gradient is additionally or alternatively transmitted to a remote system for use in remote updating of global weights of a global speech recognition model.
1. A method of personalizing an end-to-end speech recognition model for a user, the method implemented by one or more processors and the method comprising:
identifying a textual segment;
generating synthesized speech audio data that includes synthesized speech of the identified textual segment, wherein generating the synthesized speech audio data comprises processing the textual segment using a speech synthesis model adapted to one or more voice characteristics of the user;
processing, using the end-to-end speech recognition model, the synthesized speech audio data to generate a predicted textual segment;
generating a gradient based on comparing the predicted textual segment to the textual segment; and
updating one or more weights of the end-to-end speech recognition model based on the generated gradient.
2. The method of claim 1 , wherein the one or more processors are of a client device and further comprising:
transmitting, to a remote system, the generated gradient without transmitting any of: the textual segment, the synthesized speech audio data, and the predicted textual segment;
wherein the remote system utilizes the generated gradient, and additional gradients from additional client devices, to update global weights of a global end-to-end speech recognition model.
3. The method of claim 1 , wherein the identified textual segment is a name and wherein identifying the textual segment is based on the name being saved as a contact in a contacts list.
4. The method of claim 1 , wherein the identified textual segment is a name of a media item and wherein identifying the textual segment includes identifying the textual segment from a media playlist.
5. The method of claim 1 , wherein identifying the textual segment is based on determining that the textual segment is new.
6. The method of claim 1 , wherein the one or more processors are of a client device and further comprising:
determining, based on sensor data from one or more sensors of the client device, that a current state of the client device satisfies one or more conditions;
wherein generating the synthesized speech audio data, processing the synthesized speech audio data to generate the predicted textual segment, generating the gradient, and/or updating the one or more weights are performed responsive to determining that the current state of the client device satisfies the one or more conditions.
7. The method of claim 6 , wherein the one or more conditions include at least one of: the client device is charging, the client device has at least a threshold state of charge, or the client device is not being carried.
8. The method of claim 6 , wherein the one or more conditions include that the client device has at least a threshold state of charge and the client device is not being carried.
9. The method of claim 1 , wherein the speech synthesis model is adapted to the one or more voice characteristics of the user based on prior audio data that captures a prior utterance of the user, the prior utterance of the user being provided prior to generating the synthesized speech audio.
10. A system used in personalizing an end-to-end speech recognition model for a user, the system comprising:
memory storing instructions;
one or more processors operable to execute the instructions to:
identify a textual segment;
generate synthesized speech audio data that includes synthesized speech of the identified textual segment, wherein in generating the synthesized speech audio data one or more of the processors are to process the textual segment using a speech synthesis model adapted to one or more voice characteristics of the user;
process, using the end-to-end speech recognition model, the synthesized speech audio data to generate a predicted textual segment;
generate a gradient based on comparing the predicted textual segment to the textual segment; and
update one or more weights of the end-to-end speech recognition model based on the generated gradient.
11. The system of claim 10 , wherein the one or more processors are of a client device and wherein one or more of the processors are further operable to:
transmit, to a remote system, the generated gradient without transmitting any of: the textual segment, the synthesized speech audio data, and the predicted textual segment;
wherein the remote system utilizes the generated gradient, and additional gradients from additional client devices, to update global weights of a global end-to-end speech recognition model.
12. The system of claim 10 , wherein the identified textual segment is a name and wherein in identifying the textual segment one or more of the processors are to identify the textual segment based on the name being saved as a contact in a contacts list.
13. The system of claim 10 , wherein the identified textual segment is a name of a media item and wherein in identifying the textual segment one or more of the processors are to identify the textual segment from a media playlist.
14. The system of claim 10 , wherein in identifying the textual segment one or more of the processors are to identify the textual segment based on determining that the textual segment is new.
15. The system of claim 10 , wherein the one or more processors are of a client device and wherein one or more of the processors are further operable to:
determine, based on sensor data from one or more sensors of the client device, that a current state of the client device satisfies one or more conditions;
wherein generating the synthesized speech audio data, processing the synthesized speech audio data to generate the predicted textual segment, generating the gradient, and/or updating the one or more weights are performed responsive to determining that the current state of the client device satisfies the one or more conditions.
16. The system of claim 15 , wherein the one or more conditions include at least one of: the client device is charging, the client device has at least a threshold state of charge, or the client device is not being carried.
17. The system of claim 15 , wherein the one or more conditions include that the client device has at least a threshold state of charge and the client device is not being carried.
18. The system of claim 10 , wherein the speech synthesis model is adapted to the one or more voice characteristics of the user based on prior audio data that captures a prior utterance of the user, the prior utterance of the user being provided prior to generating the synthesized speech audio.
19. One or more non-transitory computer readable storage media storing computer instructions that are executable by one or more processors to personalize an end-to-end speech recognition model for a user by:
identifying a textual segment;
generating synthesized speech audio data that includes synthesized speech of the identified textual segment, wherein generating the synthesized speech audio data comprises processing the textual segment using a speech synthesis model adapted to one or more voice characteristics of the user;
processing, using the end-to-end speech recognition model, the synthesized speech audio data to generate a predicted textual segment;
generating a gradient based on comparing the predicted textual segment to the textual segment; and
updating one or more weights of the end-to-end speech recognition model based on the generated gradient.