IP Library › Granted Patent US 12,277,937
Granted Patent B2
US 12,277,937 · App. 18/378,038 · Granted Apr 15, 2025

Systems, methods, and apparatuses for resuming dialog sessions via automated assistant

Inventors: Vikram Aggarwal (Palo Alto, CA); Jung Eun Kim (San Francisco, CA); Deniz Binay (Sunnyvale, CA)
Assignee: GOOGLE LLC
G10L15/222G06F3/04842G06F9/485G10L25/51G06F9/461G06F9/4856G06F16/3325G06F16/90332G06F40/35G10L2015/223G10L2015/225G10L2015/228
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,277,937
App. No.
18/378,038
Granted
Apr 15, 2025
Kind
B2
Abstract

Methods, apparatus, systems, and computer-readable media are provided for storing incomplete dialog sessions between a user and an automated assistant in order that the dialog sessions can be completed in furtherance of certain actions. While interacting with an automated assistant, a user can become distracted and not complete the interaction to the point of the automated assistant performing some action. In response, the automated assistant can store the interaction as a dialog session. Subsequently, the user may express interest, directly or indirectly, in completing the dialog session, and the automated assistant can provide the user with a selectable element that, when selected, causes the dialog session to be reopened. The user can then continue the dialog session with the automated assistant in order that the originally intended action can be performed by the automated assistant.

Claims (75)

1. A method implemented by one or more processors, the method comprising:

receiving natural language input from a user as part of a dialog session between the user and an automated assistant via an automated assistant interface of a client device of the user;

determining, based on processing the natural language input, an action to be performed by the automated assistant during the dialog session;

identifying, based on the action to be performed by the automated assistant during the dialog session, a third party agent for the automated assistant to engage with in furtherance of performing the action;

causing the automated assistant to engage with the third party agent in furtherance of performing the action;

while the automated assistant is engaged with the third party agent in furtherance of performing the action:

causing output to be provided for presentation to the user via the automated assistant interface of the client device of the user, the output requesting information to be utilized in furtherance of performing the action;

determining that no additional natural language input is received from the user subsequent to causing the output to be provided for presentation to the user; and

in response to determining that no additional natural language input is received from the user subsequent to causing the output to be provided for presentation to the user:

generating data that characterizes a state of the dialog session, the data that characterizes the state of the dialog session including a transcript of the dialog and an indication of the third party agent that engaged with the automated assistant in furtherance of performing the action; and

storing the data that characterizes the state of the dialog session; and

subsequent to storing the data that characterizes the state of the dialog session:

receiving a user selection of a given selectable icon associated with the dialog session via the automated assistant interface of the client device of the user, the given selectable icon being one of a plurality of disparate selectable icons that are displayed via the automated assistant interface of the client device of the user, and each of the plurality of disparate selectable icons being associated with a corresponding incomplete dialog session from among a plurality of incomplete dialog sessions that were previously conducted at the client device of the user;

in response to receiving the user selection of the selectable icon via the automated assistant interface of the client device of the user:

loading the data that characterizes the state of the dialog session, wherein loading the data that characterizes the state of the dialog session comprises:

causing the transcript of the dialog to be visually rendered via the automated assistant interface of the client device; and

causing the automated assistant to re-engage with the third party agent in furtherance of performing the action; and

receiving the additional natural language input from the user as part of the dialog session between the user and the automated assistant via the automated assistant interface of the client device of the user, the additional natural language input including the information to be utilized in furtherance of performing the action.

2. The method of claim 1 , wherein the automated assistant interface of the client device of the user includes the transcript of the dialog session.

3. The method of claim 1 , wherein the third party agent is associated with an application or a website.

4. The method of claim 1 , wherein the third party agent generates the output that requests the information to be utilized in furtherance of performing the action.

5. The method of claim 1 , wherein the natural language input received from the user is typed input received via the automated assistant interface of the client device of the user.

6. The method of claim 1 , wherein the natural language input received from the user is spoken input received via the automated assistant interface of the client device of the user.

7. The method of claim 1 , wherein determining that no additional natural language input is received from the user subsequent to causing the output to be provided for presentation to the user is based on determining that no additional natural language input is received from the user subsequent to causing the output to be provided for presentation to the user.

8. The method of claim 1 , wherein the plurality of disparate selectable icons are rendered according to a ranking of the plurality of incomplete dialog sessions that were previously conducted at the client device of the user.

9. The method of claim 8 , wherein the ranking of the plurality of incomplete dialog sessions that were previously conducted at the client device of the user is determined by a computing device that is separate from the client device of the user.

10. A system comprising:

at least one processor; and

memory storing instructions that, when executed, cause the at least one processor to be operable to:

receive natural language input from a user as part of a dialog session between the user and an automated assistant via an automated assistant interface of a client device of the user;

determine, based on processing the natural language input, an action to be performed by the automated assistant during the dialog session;

identify, based on the action to be performed by the automated assistant during the dialog session, a third party agent for the automated assistant to engage with in furtherance of performing the action;

cause the automated assistant to engage with the third party agent in furtherance of performing the action;

while the automated assistant is engaged with the third party agent in furtherance of performing the action:

cause output to be provided for presentation to the user via the automated assistant interface of the client device of the user, the output requesting information to be utilized in furtherance of performing the action;

determine that no additional natural language input is received from the user subsequent to causing the output to be provided for presentation to the user; and

in response to determining that no additional natural language input is received from the user subsequent to causing the output to be provided for presentation to the user:

generate data that characterizes a state of the dialog session, the data that characterizes the state of the dialog session including a transcript of the dialog and an indication of the third party agent that engaged with the automated assistant in furtherance of performing the action; and

store the data that characterizes the state of the dialog session; and

subsequent to storing the data that characterizes the state of the dialog session:

receive a user selection of a given selectable icon associated with the dialog session via the automated assistant interface of the client device of the user, the given selectable icon being one of a plurality of disparate selectable icons that are displayed via the automated assistant interface of the client device of the user, and each of the plurality of disparate selectable icons being associated with a corresponding incomplete dialog session from among a plurality of incomplete dialog sessions that were previously conducted at the client device of the user;

in response to receiving the user selection of the selectable icon via the automated assistant interface of the client device of the user:

load the data that characterizes the state of the dialog session, wherein, in loading the data that characterizes the state of the dialog session, the at least one processor is operable to:

 cause the transcript of the dialog to be visually rendered via the automated assistant interface of the client device; and

 cause the automated assistant to re-engage with the third party agent in furtherance of performing the action; and

receive the additional natural language input from the user as part of the dialog session between the user and the automated assistant via the automated assistant interface of the client device of the user, the additional natural language input including the information to be utilized in furtherance of performing the action.

11. The system of claim 10 , wherein the automated assistant interface of the client device of the user includes the transcript of the dialog session.

12. The system of claim 10 , wherein the third party agent is associated with an application or a website.

13. The system of claim 10 , wherein the third party agent generates the output that requests the information to be utilized in furtherance of performing the action.

14. The system of claim 10 , wherein the natural language input received from the user is typed input received via the automated assistant interface of the client device of the user.

15. The system of claim 10 , wherein the natural language input received from the user is spoken input received via the automated assistant interface of the client device of the user.

16. A non-transitory computer-readable storage medium storing instructions that, when executed, cause at least one processor to be operable to perform operations, the operations comprising:

receiving natural language input from a user as part of a dialog session between the user and an automated assistant via an automated assistant interface of a client device of the user;

determining, based on processing the natural language input, an action to be performed by the automated assistant during the dialog session;

identifying, based on the action to be performed by the automated assistant during the dialog session, a third party agent for the automated assistant to engage with in furtherance of performing the action;

causing the automated assistant to engage with the third party agent in furtherance of performing the action;

while the automated assistant is engaged with the third party agent in furtherance of performing the action:

causing output to be provided for presentation to the user via the automated assistant interface of the client device of the user, the output requesting information to be utilized in furtherance of performing the action;

determining that no additional natural language input is received from the user subsequent to causing the output to be provided for presentation to the user; and

in response to determining that no additional natural language input is received from the user subsequent to causing the output to be provided for presentation to the user:

generating data that characterizes a state of the dialog session, the data that characterizes the state of the dialog session including a transcript of the dialog and an indication of the third party agent that engaged with the automated assistant in furtherance of performing the action; and

storing the data that characterizes the state of the dialog session; and

subsequent to storing the data that characterizes the state of the dialog session:

receiving a user selection of a given selectable icon associated with the dialog session via the automated assistant interface of the client device of the user, the given selectable icon being one of a plurality of disparate selectable icons that are displayed via the automated assistant interface of the client device of the user, and each of the plurality of disparate selectable icons being associated with a corresponding incomplete dialog session from among a plurality of incomplete dialog sessions that were previously conducted at the client device of the user;

in response to receiving the user selection of the selectable icon via the automated assistant interface of the client device of the user:

loading the data that characterizes the state of the dialog session, wherein loading the data that characterizes the state of the dialog session comprises:

causing the transcript of the dialog to be visually rendered via the automated assistant interface of the client device; and

causing the automated assistant to re-engage with the third party agent in furtherance of performing the action; and

receiving the additional natural language input from the user as part of the dialog session between the user and the automated assistant via the automated assistant interface of the client device of the user, the additional natural language input including the information to be utilized in furtherance of performing the action.

17. The non-transitory computer-readable storage medium of claim 16 , wherein the automated assistant interface of the client device of the user includes the transcript of the dialog session.

18. The non-transitory computer-readable storage medium of claim 16 , wherein the third party agent is associated with an application or a website.

19. The non-transitory computer-readable storage medium of claim 16 , wherein the third party agent generates the output that requests the information to be utilized in furtherance of performing the action.

20. The non-transitory computer-readable storage medium of claim 16 ,

wherein the natural language input received from the user is typed input received via the automated assistant interface of the client device of the user; or

wherein the natural language input received from the user is spoken input received via the automated assistant interface of the client device of the user.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 24, 2023
From: AGGARWAL, VIKRAM; KIM, JUNG EUN; BINAY, DENIZ
To: GOOGLE INC.
Reel/Frame 065324/0444 →
CHANGE OF NAME Recorded Oct 24, 2023
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 065334/0307 →
Continuity (4)
Continuation 17572261 · Jan 10, 2022
Continuation 16359179 · Mar 20, 2019
Continuation 15597088 · May 16, 2017
Related Publication 20240105180A1 · Mar 28, 2024
References Cited (67)
US 6163794A · Lange et al. · 2000 [cited by applicant]
US 7310803B2 · Haughey · 2007 [cited by applicant]
US 7454351B2 · Jeschke et al. · 2008 [cited by applicant]
US 7984440B2 · James et al. · 2011 [cited by applicant]
US 8214219B2 · Prieto et al. · 2012 [cited by applicant]
US 8448177B1 · Morlen · 2013 [cited by applicant]
US 8631419B2 · Horvitz et al. · 2014 [cited by applicant]
US 8823507B1 · Touloumtzis · 2014 [cited by applicant]
US 9378456B2 · White · 2016 [cited by examiner]
US 9378467B1 · Chaiyochlarb et al. · 2016 [cited by applicant]
US 9696886B2 · Krishnappa et al. · 2017 [cited by applicant]
US 9911415B2 · VanBlon et al. · 2018 [cited by applicant]
US 9922642B2 · Pitschel et al. · 2018 [cited by applicant]
US 9972318B1 · Kelly · 2018 [cited by applicant]
US 10229259B2 · Wang et al. · 2019 [cited by applicant]
US 10269351B2 · Aggarwal et al. · 2019 [cited by applicant]
US 10754685B2 · Gupta · 2020 [cited by examiner]
US 11264033B2 · Aggarwal et al. · 2022 [cited by applicant]
US 20060095267A1 · Yano et al. · 2006 [cited by applicant]
US 20060095268A1 · Yano et al. · 2006 [cited by applicant]
US 20080127201A1 · Mori et al. · 2008 [cited by applicant]
US 20090125813A1 · Shen et al. · 2009 [cited by applicant]
US 20100257526A1 · Zimmet et al. · 2010 [cited by applicant]
US 20110289503A1 · Toub et al. · 2011 [cited by applicant]
US 20140136187A1 · Wolverton et al. · 2014 [cited by applicant]
US 20140142953A1 · Kim et al. · 2014 [cited by applicant]
US 20140143196A1 · White · 2014 [cited by examiner]
US 20140218372A1 · Missig et al. · 2014 [cited by applicant]
US 20140244712A1 · Walters et al. · 2014 [cited by applicant]
US 20140310001A1 · Kalns et al. · 2014 [cited by applicant]
US 20150039292A1 · Suleman et al. · 2015 [cited by applicant]
US 20150142704A1 · London · 2015 [cited by applicant]
US 20150248651A1 · Akutagawa et al. · 2015 [cited by applicant]
US 20160042735A1 · Vibbert et al. · 2016 [cited by applicant]
US 20160117593A1 · London · 2016 [cited by applicant]
US 20160232625A1 · Akutagawa et al. · 2016 [cited by applicant]
US 20160259656A1 · Sumner et al. · 2016 [cited by applicant]
US 20160308980A1 · Singh et al. · 2016 [cited by applicant]
US 20170031576A1 · Saoji et al. · 2017 [cited by applicant]
US 20170053110A1 · Wang et al. · 2017 [cited by applicant]
US 20180284958A1 · Alphin, III et al. · 2018 [cited by applicant]
US 20220130391A1 · Aggarwal et al. · 2022 [cited by applicant]
US 20220172717A1 · Luo · 2022 [cited by examiner]
CN 1937608 · 2007 [cited by applicant]
CN 102542410 · 2012 [cited by applicant]
JP 2005309185A · 2005 [cited by examiner]
JP 2006188098 · 2006 [cited by applicant]
JP 2007148654 · 2007 [cited by applicant]
JP 2015129793 · 2015 [cited by applicant]
JP 2015528140 · 2015 [cited by applicant]
WO 2016025277 · 2016 [cited by applicant]
WO 2016110954 · 2016 [cited by applicant]
Translation of JP2005309185A. (Year: 2005). [cited by examiner]
China National Intellectual Property Administration; Notice of Grant issued for Application No. 201880032818.8, 6 pages, dated Nov. 15, 2023. [cited by applicant]
China National Intellectual Property Administration; Notification of Second Office Action issued in Application No. 201880032818.8; 15 pages; dated Aug. 12, 2023. [cited by applicant]
Japan Patent Office; Notice of Reasons for Rejection issued in Application No. 2022-030967, 8 pages, dated Mar. 30, 2023. [cited by applicant]
China National Intellectual Property Administration; Notification of First Office Action issued in Application No. 201880032818.8; 18 pages; dated Jan. 9, 2023. [cited by applicant]
European Patent Office; Extended Search Report issued in Application No. 22179403.5; 8 pages; dated Nov. 10, 2022. [cited by applicant]
Japanese Patent Office; Notice of Allowance issued in Application No. 2019-563499, 4 pages, dated Jan. 31, 2022. [cited by applicant]
European Patent Office; Intention to Grant issued in Application No. 18732517.0, 41 pages, dated Feb. 16, 2022. [cited by applicant]
European Patent Office; Preliminary Opinion issued in EP18732517.0; 11 pages; dated Dec. 23, 2021. [cited by applicant]
European Patent Office; Summons to attend oral proceedings pursuant to Rule 115(1) EPC issued in Application No. 18732517.0; 11 pages; dated Sep. 8, 2021. [cited by applicant]
Japanese Patent Office; Final Rejection issue in Application No. 2019-563499; 11 pages; dated Jul. 28, 2021. [cited by applicant]
Korea Patent Office; Office Action issue in Application No. 10-2019-7037013; 13 pages; dated Jan. 27, 2021. [cited by applicant]
Japanese Patent Office; Notice of Reasons for Rejection issue in Application No. 2019-563499; 9 pages; dated Feb. 1, 2021. [cited by applicant]
European Patent Office; Examination Report issued for U.S. Appl. No. 18/732,517 dated Feb. 19, 2020 (8 Pages). [cited by applicant]
European Patent Office; International Search Report and Written Opinion of PCT Ser. No. PCT/US2018/032784; 17 pages; dated Aug. 31, 2018. [cited by applicant]