Granted Patent B1

US 12,400,645 · App. 18/082,838 · Granted Aug 26, 2025

Inventors: Anjali Yuan Narayan-Chen (Santa Clara, CA); Jiao Sun (Los Angeles, CA); Shereen Oraby (Santa Clara, CA); Shuyang Gao (Sunnyvale, CA); Tagyoung Chung (Palo Alto, CA); Jing Huang (Mountain View, CA); Yang Liu (Los Altos, CA); Nanyung Peng (Los Angeles, CA); Arindam Mandal (Redwood City, CA); Premkumar Natarajan (Rolling Hills Estates, CA)

Assignee: Amazon Technologies, Inc.

G10L15/183G10L13/02G10L15/22G10L2015/223

Abstract

Claims (65)

1. A computer-implemented method comprising:

receiving, from a first device, first audio data representing an utterance corresponding to a dialog between a user of the first device and a system;

performing speech processing using the first audio data to determine first data representing a plurality of words in the dialog;

processing the first data using a bidirectional encoder representations from transformers (BERT) model configured to use input data to classify a candidate word pair as suitable for generating a pun that is contextually related to the input data;

using an output of the BERT model to select a first word pair suitable for generating a pun;

processing the first word pair and the first data using a text-to-text transfer transformer (T5) model to generate second data representing a first pun that is contextually related to the first data;

sending the second data to a dialog management component;

sending, from the dialog management component to a speech synthesis component, the second data;

processing the second data by the speech synthesis component to generate second audio data representing the first pun; and

sending the second audio data to the first device.

2. The computer-implemented method of claim 1 , wherein the T5 model generates the first pun based at least in part on a first meaning of a first word of the first word pair and a second meaning of a second word of the first word pair.

3. The computer-implemented method of claim 1 , further comprising, prior to receiving the first audio data:

training the BERT model using training data including third data representing one or more keywords, fourth data representing a second word pair, and an indication whether the second word pair is suitable for generating a pun contextually related to the one or more keywords, with the indication being used to supervise the training.

4. The computer-implemented method of claim 1 , further comprising:

training the T5 model using training data including at least third data representing a text prefix corresponding to pun generation, fourth data representing a second word pair, fifth data representing one or more keywords, and sixth data representing a second pun, with the sixth data being used to supervise the training.

5. A computer-implemented method, comprising:

determining first data representing one or more first keywords;

processing the first data by at least one first component to determine a first word pair suitable for generating a pun;

processing the first word pair and the one or more first keywords using a first machine learning model to generate a first pun that is contextually related to the one or more first keywords, wherein the first machine learning model is trained using training data including at least:

second data representing a text prefix corresponding to pun generation,

third data representing a second word pair,

fourth data representing one or more second keywords, and

fifth data representing a second pun; and

causing a first device to output a representation of the first pun.

6. The computer-implemented method of claim 5 , wherein the first machine learning model generates the first pun based at least in part on a first meaning of a first word of the first word pair and a second meaning of a second word of the first word pair.

7. The computer-implemented method of claim 5 , wherein the first machine learning model comprises a text-to-text transfer transformer model.

8. The computer-implemented method of claim 7 , further comprising:

training the text-to-text transfer transformer model using the training data wherein the fifth data is second pun, wherein the fifth data is used to supervise the training.

9. The computer-implemented method of claim 5 , wherein the at least one first component comprises a second machine learning model that uses the first data to classify respective candidate word pairs as suitable for generating a pun that is contextually related to the first data.

10. The computer-implemented method of claim 9 , wherein the at least one first component further comprises a pun word selector configured to select the first word pair based on an output of the second machine learning model.

11. The computer-implemented method of claim 5 , further comprising:

receiving, by a dialog management component, a first communication;

determining the first data based on the first communication;

generating, by the dialog management component, a response representing the first pun; and

sending, to the first device, a second communication corresponding to the response to cause the first device to output the representation of the first pun.

12. The computer-implemented method of claim 5 , wherein:

the first machine learning model is configured to generate an output pun based at least in part on one or more input keywords and an input word pair such that the output pun is contextually related to the one or more input keywords.

13. A system, comprising:

at least one processor; and

at least one memory comprising instructions that, when executed by the at least one processor, cause the system to:

determine first data representing one or more first keywords;

process the first data by at least one first component to determine a first word pair suitable for generating a pun;

process the first word pair and the one or more first keywords using a first machine learning model to generate a first pun that is contextually related to the one or more first keywords, wherein the first machine learning model is trained using training data including at least:

second data representing a text prefix corresponding to pun generation,

third data representing a second word pair,

fourth data representing one or more second keywords, and

fifth data representing a second pun;

receive, from the first machine learning model, sixth data representing the first pun; and

cause, based on the sixth data, a first device to output a representation of the first pun.

14. The system of claim 13 , wherein the at least one memory further comprises additional instructions that, when executed by the at least one processor, cause the system to:

process a first meaning of a first word of the first word pair and a second meaning of a second word of the first word pair with the first machine learning model to generate the first pun.

15. The system of claim 13 , wherein the first machine learning model comprises a text-to-text transfer transformer model.

16. The system of claim 15 , wherein the at least one memory further comprises additional instructions that, when executed by the at least one processor, cause the system to:

train the text-to-text transfer transformer model using the training data, wherein the fifth data is used to supervise the training.

17. The system of claim 13 , wherein the at least one first component comprises a second machine learning model, and the at least one memory further comprises additional instructions that, when executed by the at least one processor, cause the system to:

process the first data with the second machine learning model to classify respective candidate word pairs as suitable for generating a pun that is contextually related to the first data.

18. The system of claim 17 , wherein the at least one first component further comprises a pun word selector, and the at least one memory further comprises additional instructions that, when executed by the at least one processor, cause the system to:

use the pun word selector to select the first word pair based on an output of the second machine learning model.

19. The system of claim 13 , wherein the at least one memory further comprises additional instructions that, when executed by the at least one processor, cause the system to:

receive, by a dialog management component, a first communication;

determine the first data based on the first communication;

generate, by the dialog management component and based on the sixth data, a response representing the first pun; and

send, to the first device, a second communication corresponding to the response to cause the first device to output the representation of the first pun.

20. The system of claim 19 , wherein:

the first machine learning model is configured to generate an output pun based at least in part on one or more input keywords and an input word pair such that the output pun is contextually related to the one or more input keywords.

Assignments (2)

Continuity (1)

References Cited (82)