IP Library Granted Patent US 12,456,015
Granted Patent B1
US 12,456,015 · App. 18/193,693 · Granted Oct 28, 2025

Natural language question generation

Inventors: Anoop Kumar (Fountain Valley, CA); Yi Fung (Quincy, MA); Premkumar Natarajan (Rolling Hills Estates, CA); Aram Galstyan (Los Angeles, CA); Heng Ji (Champaign, IL)
Assignee: Amazon Technologies, Inc.
G06F40/30G06F16/3329G06F40/40
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,456,015
App. No.
18/193,693
Granted
Oct 28, 2025
Kind
B1
Abstract

Techniques for generating a natural language prompt to further a goal of a dialog, are described. During a dialog, the system receives one or more user inputs including a user question, a user response to the question, and a request to generate a further question following the response. The system determines ASR output data corresponding to the user inputs, and determines dialog history data of the dialog. Using the ASR output data and the dialog history data, the system determines a category and an explanation of relevance corresponding to the category. Using the ASR output data, the dialog history, the category, and the explanation, the system determines the further question to be output to the user.

Claims (108)

1. A computer-implemented method comprising:

receiving first input data representing a first question corresponding to a first dialog in a natural language;

receiving second input data representing a first response to the first question;

receiving third input data corresponding to a request to generate a second question following the first response;

using a first machine learning (ML) component configured to generate data associated with a predicted question of a dialog, processing the first input data, the second input data, and the third input data to determine:

a first category corresponding to the second question, and

a first explanation of relevance of the first category;

processing, using a second ML component, the first input data, the second input data, the third input data, the first category, and the first explanation to generate the second question; and

outputting the second question.

2. The computer-implemented method of claim 1 , further comprising:

receiving fourth input data representing a third question corresponding to a second dialog in the natural language;

receiving fifth input data representing a second response to the third question;

processing, by a first preliminary ML component, the fourth input data and the fifth input data to determine:

a second category corresponding to a fourth question following the second response, and

a second explanation of relevance of the second category;

determining a first comparison of the second category and a training category corresponding to the third question;

determining a second comparison of the second explanation and a training natural explanation corresponding to the third question; and

based on the first comparison and the second comparison, determining a third ML component.

3. The computer-implemented method of claim 2 , further comprising:

after determining the third ML component, processing, using the third ML component, the fourth input data, the fifth input data, the second category, and the second explanation to generate the third question;

determining a third comparison of the third question and a training question; and

based on the third comparison, determining a fourth ML component, wherein the first ML component and the second ML component correspond to the fourth ML component.

4. The computer-implemented method of claim 1 , further comprising:

prior to receiving the first input data, receiving a transcript of a description of a first event associated with the first dialog;

using a third ML component, processing the transcript to generate a third question associated with the first event; and

outputting the third question.

5. A computer-implemented method comprising:

receiving first input data corresponding to a first natural language question;

receiving second input data corresponding to first natural language response to the first natural language question;

using a first machine learning (ML) component configured to generate data associated with a predicted question of a dialog, processing the first input data and the second input data to determine:

a first category corresponding to a second natural language question following the first natural language response, and

a first explanation of relevance of the first category;

using the first input data, the second input data, the first category, and the first explanation, generating the second natural language question; and

outputting the second natural language question.

6. The computer-implemented method of claim 5 , further comprising:

prior to receiving the first input data, receiving third input data corresponding to a natural language description of a first event;

using a second ML component, processing the third input data to generate a third natural language question associated with the first event; and

outputting the third natural language question.

7. The computer-implemented method of claim 5 , wherein:

the first input data and the second input data are associated with a first dialog involving a first user and a second user,

the first natural language question corresponds to a question asked by the first user to the second user,

the first natural language response corresponds to a response of the second user to the question asked by the first user, and

the second natural language question is configured to be addressed to the second user.

8. The computer-implemented method of claim 5 , wherein:

generating the second natural language question comprises processing the first input data, the second input data, the first category, and the first explanation using a second ML component.

9. The computer-implemented method of claim 5 , further comprising:

determining third input data corresponding to a third natural language question previously presented, wherein:

determining the first category is further based on processing the third input data,

determining the first explanation is further based on processing the third input data, and

determining the second natural language question is further based on processing the third input data.

10. The computer-implemented method of claim 5 , further comprising:

after outputting the second natural language question, receiving third input data corresponding to a second natural language response to the second natural language question;

receiving fourth input data corresponding to the second natural language question;

using the first ML component, processing the third input data and the fourth input data to determine:

a second category corresponding to a third natural language question following the second natural language response, and

a second explanation of relevance of the second category;

processing the third input data, the fourth input data, the second category, and the second explanation to generate the third natural language question;

determining that the third natural language question is semantically similar to the second natural language question; and

ceasing processing with respect to the third natural language question.

11. The computer-implemented method of claim 5 , further comprising:

after outputting the second natural language question, receiving third input data corresponding to a request to output an explanation of relevance of the second natural language question; and

based on the third input data corresponding to the request to output the explanation of relevance of the second natural language question, outputting the first explanation.

12. The computer-implemented method of claim 5 , wherein:

the first input data and the second input data are associated with a first dialog, the first dialog corresponding to a first goal to be achieved during the first dialog, and

the method further comprises:

using the first ML component, processing the first input data, the second input data, the first category, and the first explanation to generate the second natural language question, wherein the second natural language question is configured to result in third input data associated with the first goal.

13. A computing system comprising:

at least one processor; and

at least one memory comprising instructions that, when executed by the at least one processor, cause the computing system to:

receive first input data corresponding to a first natural language question;

receive second input data corresponding to a first natural language response to the first natural language question;

using a first machine learning (ML) component configured to generate data associated with a predicted question of a dialog, process the first input data and the second input data to determine:

a first category corresponding to a second natural language question following the first natural language response, and

a first explanation of relevance of the first category;

using the first input data, the second input data, the first category, and the first explanation, generating the second natural language question; and

outputting the second natural language question.

14. The computing system of claim 13 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to:

prior to receiving the first input data, receive third input data corresponding to a natural language description of a first event;

using a second ML component, process the third input data to generate a third natural language question associated with the first event; and

output the third natural language question.

15. The computing system of claim 13 , wherein:

the first input data and the second input data are associated with a first dialog involving a first user and a second user,

the first natural language question corresponds to a question asked by the first user to the second user,

the first natural language response corresponds to a response of the second user to the question asked by the first user, and

the second natural language question is configured to be addressed to the second user.

16. The computing system of claim 13 , wherein:

generating the second natural language question comprises processing the first input data, the second input data, the first category, and the first explanation using a second ML component.

17. The computing system of claim 13 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to:

determine third input data corresponding to a third natural language question previously presented, wherein:

determining the first category is further based on processing the third input data,

determining the first explanation is further based on processing the third input data, and

determining the second natural language question is further based on processing the third input data.

18. The computing system of claim 13 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to:

after outputting the second natural language question, receive third input data corresponding to a second natural language response to the second natural language question;

receive fourth input data corresponding to the second natural language question;

using the first ML component, process the third input data and the fourth input data to determine:

a second category corresponding to a third natural language question following the second natural language response, and

a second explanation of relevance of the second category;

process the third input data, the fourth input data, the second category, and the second explanation to generate the third natural language question;

determine that the third natural language question is semantically similar to the second natural language question; and

cease processing with respect to the third natural language question.

19. The computing system of claim 13 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to:

after outputting the second natural language question, receive third input data corresponding to a request to output an explanation of relevance of the second natural language question; and

based on the third input data corresponding to the request to output the explanation of relevance of the second natural language question, outputting the first explanation.

20. The computing system of claim 13 , wherein:

the first input data and the second input data are associated with a first dialog, the first dialog corresponding to a first goal to be achieved during the first dialog, and

the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to:

using the first ML component, process the first input data, the second input data, the first category, and the first explanation to generate the second natural language question, wherein the second natural language question is configured to result in third input data associated with the first goal.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 25, 2024
From: FUNG, YI; NATARAJAN, PREMKUMAR
To: AMAZON TECHNOLOGIES, INC.
Reel/Frame 067837/0201 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 31, 2023
From: KUMAR, ANOOP; GALSTYAN, ARAM; JI, HENG
To: AMAZON TECHNOLOGIES, INC.
Reel/Frame 063182/0441 →
References Cited (70)
US 7693705B1 · Jamieson · 2010 [cited by examiner]
US 10170014B2 · Beason · 2019 [cited by examiner]
US 10387576B2 · Beller · 2019 [cited by examiner]
US 11687734B2 · Harper · 2023 [cited by examiner]
US 11734510B2 · Hu · 2023 [cited by examiner]
US 11741306B2 · Galley · 2023 [cited by examiner]
US 11741316B2 · Galitsky · 2023 [cited by examiner]
US 11783131B2 · Yuan · 2023 [cited by examiner]
US 12141535B2 · Galitsky · 2024 [cited by examiner]
US 12182518B2 · Galitsky · 2024 [cited by examiner]
US 20020169595A1 · Agichtein · 2002 [cited by examiner]
US 20180075366A1 · Dole · 2018 [cited by examiner]
US 20180144047A1 · Beller · 2018 [cited by examiner]
US 20180260472A1 · Kelsey · 2018 [cited by examiner]
US 20180293302A1 · Ko · 2018 [cited by examiner]
US 20180357324A1 · Chakraborty · 2018 [cited by examiner]
US 20190163745A1 · Beller · 2019 [cited by examiner]
US 20190197415A1 · Bulut · 2019 [cited by examiner]
US 20200089768A1 · Kim · 2020 [cited by examiner]
US 20200125658A1 · Gupta · 2020 [cited by examiner]
US 20210056150A1 · Karandish · 2021 [cited by examiner]
US 20210209139A1 · Wu · 2021 [cited by examiner]
US 20210286833A1 · Verstraete · 2021 [cited by examiner]
US 20210406467A1 · Li · 2021 [cited by examiner]
US 20220067281A1 · Hu · 2022 [cited by examiner]
US 20220067486A1 · Klein · 2022 [cited by examiner]
US 20220245838A1 · Wu · 2022 [cited by examiner]
US 20220300718A1 · Chen · 2022 [cited by examiner]
US 20220300756A1 · Sekar · 2022 [cited by examiner]
US 20220391849A1 · O'Connor · 2022 [cited by examiner]
US 20230061773A1 · Patil · 2023 [cited by examiner]
US 20230061906A1 · Gaur · 2023 [cited by examiner]
US 20230081891A1 · Kulkarni · 2023 [cited by examiner]
US 20230169362A1 · Beucher · 2023 [cited by examiner]
US 20230222164A1 · Silverstein · 2023 [cited by examiner]
US 20230281393A1 · Shroff · 2023 [cited by examiner]
US 20240045990A1 · Boyer · 2024 [cited by examiner]
US 20240248918A1 · Bhagavan · 2024 [cited by examiner]
US 20240282298A1 · Koneru · 2024 [cited by examiner]
US 20240311403A1 · Bursztyn · 2024 [cited by examiner]
US 20240320445A1 · Mallick · 2024 [cited by examiner]
US 20240419908A1 · Alkhalifa · 2024 [cited by examiner]
US 20250078484A1 · Nguyen · 2025 [cited by examiner]
Michael J. Bommarito II, et al., “LexNLP: Natural language processing and information extraction for legal and regulatory texts,” arXiv preprint arXiv:1806.03688v1, 2018, 7 pages. [cited by applicant]
Kevin Ros, et al., “Generation of student questions for inquiry-based learning,” Proceedings of the 15th International Conference on Natural Language Generation, 2022, pp. 186-195. [cited by applicant]
Brown, et al., “Language Models Are Few-Shot Learners”, Advances in Neural Information Processing Systems, vol. 33, pp. 877-1901, 2020. [cited by applicant]
Chalkidis, et al., “MultiEURLEX—a multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer”, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Pr… [cited by applicant]
Chalkidis, et al., “Legal-Bert: The muppets straight out of law school”, Findings of the Association for Computational Linguistics: EMNLP 2020, Online, pp. 2898-2904, 2020. [cited by applicant]
Chalkidis, et al., “LexGLUE: A benchmark dataset for legal language understanding in English”, Proceedings for the 60th Annual Meeting of the Association for Computational Linguistics (vol. 1: Long Papers), pp. 4310-433… [cited by applicant]
Choi, et al., “QuAC: Question answering in context”, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2174-2184, 2018. [cited by applicant]
Du, et al., “Identifying where to focus in reading comprehension for neural question generation”, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2067-2073, 2017. [cited by applicant]
Du, et al., “Learning to ask: Neural question generation for reading comprehension”, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (vol. 1: Long Papers), pp. 1342-1352, 2017. [cited by applicant]
Duan, et al. “Question generation for question answering”, Proceedings of the 2017 Conference of Empirical Methods in Natural Language Processing, pp. 866-874, 2017. [cited by applicant]
Forbes, et al., “Social chemistry 101: Learning to reason about social and moral norms”, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 653-670, 2020. [cited by applicant]
Fung, et al., “Normsage: Multi-lingual multi-cultural norm discovery from conversations on-the-fly”, 2022, arXiv preprint arXiv:2210.08604. [cited by applicant]
Lee, et al., “Does GPT-3 generate empathetic dialogues? A novel in-context example selection method and automatic evaluation metric for empathetic dialogue generation”, Proceedings of the 29th International Conference o… [cited by applicant]
Lewis, et al., “BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension”, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics… [cited by applicant]
Limsopatham, “Effectively leveraging BERT for legal document classification”, Proceedings of the Natural Legal Language Processing Workshop 2021, pp. 210-216, 2021. [cited by applicant]
Majumder, et al., “Interview: Large-scale modeling of media dialog with discourse patterns and knowledge grounding”, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 81… [cited by applicant]
Majumder, et al., “Ask what's missing and what's useful: Improving clarification questions generation using global knowledge”, Proceedings of the 2021 Conference of the North American Chapter of the Association for Comp… [cited by applicant]
Polsley, et al., “CaseSummarizer: A system for automated summarization of legal texts”, Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: System Demonstrations, pp. 258-262, 201… [cited by applicant]
Qi, et al., “Stay hungry, stay focused: Generating informative and specific questions in information-seeking conversations”, Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 25-40, Online, 2020. [cited by applicant]
Rajpurkar, et al., “SQaAD: 100,000+ questions for machine comprehension of text”, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, 2016. [cited by applicant]
Rao, et al., “Learning to ask good questions: Ranking clarification questions using neural expected value of perfect Information”, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics … [cited by applicant]
Ravichander, et al., “Question answering for privacy policies: Combining computational and legal perspectives”, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Internat… [cited by applicant]
Roller, et al., “Recipes for building an open-domain chatbot”, Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 300-325, 2021. [cited by applicant]
Sellam, et al., “BLEURT: Learning robust metrics for text generation”, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7881-7892, Online, 2020. [cited by applicant]
Williams, et al., “A broad-coverage challenge corpus for sentence understanding through inference”, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Huma… [cited by applicant]
Wu, et al., “De-biased court's view generation with causality”, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 763-780, Online, 2020. [cited by applicant]
Zhong, et al., “How does NLP benefit legal system: A summary of legal artificial intelligence”, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5218-5230, Online, 2020. [cited by applicant]