Granted Patent B2

US 12,462,788 · App. 18/312,576 · Granted Nov 4, 2025

Inventors: Vijayaditya Peddinti (San Jose, CA); Bhuvana Ramabhadran (Mt. Kisco, NY); Andrew Rosenberg (Brooklyn, NY); Mateusz Golebiewski (Mountain View, CA)

Assignee: Google LLC

G10L13/08G10L15/187

Abstract

Claims (66)

1 . A computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising:

processing, using a trained automatic speech recognition model, audio data representing a query spoken by a user to identify a particular word present in the query;

receiving a user pronunciation of the particular word present in the query spoken by the user;

receiving a text-to-speech (TTS) pronunciation of the same particular word, the TTS pronunciation of the particular word comprising a verified preferred pronunciation of the particular word that is verified by a person as being an accurate pronunciation of the particular word;

determining, by processing the user pronunciation and the TTS pronunciation using a trained pronunciation decision model, that the verified preferred pronunciation of the particular word is different than the user pronunciation of the particular word;

generating, using the trained pronunciation decision model, a pronunciation decision selecting the verified preferred pronunciation of the particular word, the pronunciation decision comprising:

an indication for the user that the user pronunciation of the particular word is different from the verified preferred pronunciation of the particular word; and

the verified preferred pronunciation of the particular word; and

providing the pronunciation decision to a user device associated with the user.

2 . The method of claim 1 , wherein the operations further comprise, after receiving the TTS pronunciation of the same particular word:

obtaining user pronunciation-related features associated with the user pronunciation of the particular word; and

obtaining TTS pronunciation-related features associated with the TTS pronunciation of the particular word.

3 . The method of claim 2 , wherein selecting the verified preferred pronunciation of the particular word comprises determining that the verified preferred pronunciation of the particular word is associated with a highest confidence for use in TTS audio.

4 . The method of claim 2 , wherein the user pronunciation-related features associated with the user pronunciation of the particular word comprise at least one of:

a geographical area of the user when the query was spoken by the user;

linguistic demographic information associated with the user; or

a frequency of using the user pronunciation when pronouncing the particular word in previous queries spoken by the user and/or other users.

5 . The method of claim 2 , wherein receiving the TTS pronunciation of the particular word comprises:

receiving, as input to a trained TTS system, a TTS input comprising a textual representation of a response to the query;

generating, as output from the trained TTS system, an initial sample of TTS audio comprising an initial synthesized speech representation of the response to the query; and

extracting a TTS acoustic representation of the particular word from the initial sample of the TTS audio, the TTS acoustic representation conveying the TTS pronunciation of the particular word.

6 . The method of claim 2 , wherein receiving the TTS pronunciation of the particular word comprises processing a textual representation of a response to the query to generate a TTS phoneme representation that conveys the TTS pronunciation of the particular word.

7 . The method of claim 1 , wherein the operations further comprise:

receiving audio data corresponding to the query spoken by the user; and

processing, using an automated speech recognizer (ASR), the audio data to generate a transcription of the query.

8 . The method of claim 7 , wherein receiving the user pronunciation of the particular word comprises at least one of:

extracting the user pronunciation of the particular word from an intermittent state of the ASR while using the ASR to process the audio data;

extracting a user acoustic representation of the particular word from the audio data, the user acoustic representation conveying the user pronunciation of the particular word; or

processing the audio data to generate a user phoneme representation that conveys the user pronunciation of the particular word.

9 . The method of claim 1 , wherein the operations further comprise, after generating the pronunciation decision selecting the verified preferred pronunciation of the particular word:

receiving explicit feedback from the user indicating which one of the user pronunciation of the particular word or the TTS pronunciation of the particular word the user prefers for pronouncing the particular word in subsequent TTS outputs; and

updating the trained pronunciation decision model based on the explicit feedback from the user.

10 . A system comprising:

data processing hardware; and

memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations comprising:

processing, using an automatic speech recognition (ASR) model, audio data representing a query spoken by a user to identify a particular word present in the query;

receiving a user pronunciation of the particular word present in the query spoken by the user;

receiving a text-to-speech (TTS) pronunciation of the same particular word, the TTS pronunciation of the particular word comprising a verified preferred pronunciation of the particular word that is verified by a person as being an accurate pronunciation of the particular word;

determining, by processing the user pronunciation and the TTS pronunciation using a trained pronunciation decision model, that the verified preferred pronunciation of the particular word is different than the user pronunciation of the particular word;

generating, using the trained pronunciation decision model, a pronunciation decision selecting the verified preferred pronunciation of the particular word, the pronunciation decision comprising:

an indication for the user that the user pronunciation of the particular word is different from the verified preferred pronunciation of the particular word; and

the verified preferred pronunciation of the particular word; and

providing the pronunciation decision to a user device associated with the user.

11 . The system of claim 10 , wherein the operations further comprise, after receiving the TTS pronunciation of the same particular word:

obtaining user pronunciation-related features associated with the user pronunciation of the particular word; and

obtaining TTS pronunciation-related features associated with the TTS pronunciation of the particular word.

12 . The system of claim 11 , wherein selecting the verified preferred pronunciation of the particular word comprises determining that the verified preferred pronunciation of the particular word is associated with a highest confidence for use in TTS audio.

13 . The system of claim 11 , wherein the user pronunciation-related features associated with the user pronunciation of the particular word comprise at least one of:

a geographical area of the user when the query was spoken by the user;

linguistic demographic information associated with the user; or

a frequency of using the user pronunciation when pronouncing the particular word in previous queries spoken by the user and/or other users.

14 . The system of claim 11 , wherein receiving the TTS pronunciation of the particular word comprises:

receiving, as input to a trained TTS system, a TTS input comprising a textual representation of a response to the query;

generating, as output from the trained TTS system, an initial sample of TTS audio comprising an initial synthesized speech representation of the response to the query; and

extracting a TTS acoustic representation of the particular word from the initial sample of the TTS audio, the TTS acoustic representation conveying the TTS pronunciation of the particular word.

15 . The system of claim 11 , wherein receiving the TTS pronunciation of the particular word comprises processing a textual representation of a response to the query to generate a TTS phoneme representation that conveys the TTS pronunciation of the particular word.

16 . The system of claim 10 , wherein the operations further comprise:

receiving audio data corresponding to the query spoken by the user; and

processing, using an automated speech recognizer (ASR), the audio data to generate a transcription of the query.

17 . The system of claim 16 , wherein receiving the user pronunciation of the particular word comprises at least one of:

extracting the user pronunciation of the particular word from an intermittent state of the ASR while using the ASR to process the audio data;

extracting a user acoustic representation of the particular word from the audio data, the user acoustic representation conveying the user pronunciation of the particular word; or

processing the audio data to generate a user phoneme representation that conveys the user pronunciation of the particular word.

18 . The system of claim 10 , wherein the operations further comprise, after generating the pronunciation decision selecting the verified preferred pronunciation of the particular word:

receiving explicit feedback from the user indicating which one of the user pronunciation of the particular word or the TTS pronunciation of the particular word the user prefers for pronouncing the particular word in subsequent TTS outputs; and

updating the pronunciation decision model based on the explicit feedback from the user.

Continuity (2)

References Cited (40)