IP Library Granted Patent US 12,230,272
Granted Patent B2
US 12,230,272 · App. 18/540,377 · Granted Feb 18, 2025

Proactive incorporation of unsolicited content into human-to-computer dialogs

Inventors: Ibrahim Badr (Zurich, CH); Zaheed Sabur (Baar, CH); Vladimir Vuskovic (Zollikerberg, CH); Adrian Zumbrunnen (Zurich, CH); Lucas Mirelmann (Zurich, CH)
Assignee: GOOGLE LLC
G10L15/22G06F16/90335G06N3/006G10L15/1815G10L15/1822G10L2015/223
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,230,272
App. No.
18/540,377
Granted
Feb 18, 2025
Kind
B2
Abstract

Methods, apparatus, and computer readable media are described related to automated assistants that proactively incorporate, into human-to-computer dialog sessions, unsolicited content of potential interest to a user. In various implementations, in an existing human-to-computer dialog session between a user and an automated assistant, it may be determined that the automated assistant has responded to all natural language input received from the user. Based on characteristic(s) of the user, information of potential interest to the user or action(s) of potential interest to the user may be identified. Unsolicited content indicative of the information of potential interest to the user or the action(s) may be generated and incorporated by the automated assistant into the existing human-to-computer dialog session. In various implementations, the incorporating may be performed in response to the determining that the automated assistant has responded to all natural language input received from the user during the human-to-computer dialog session.

Claims (44)

1. A method implemented using one or more processors, comprising:

determining that in an existing human-to-computer dialog session between a user and an automated assistant occurring at one or more computing devices operated by the user, the automated assistant has responded to all natural language input received from the user during the human-to-computer dialog session;

determining a current context of the user;

analyzing search queries submitted by others to identify spikes, trends, or other patterns in the search queries that are submitted by others in contexts similar to the current context of the user;

based on the spikes, trends, or other patterns, selecting one or more of the search queries submitted by others;

searching one or more online sources for information responsive to the selected one or more search queries;

generating, by one or more of the processors, unsolicited content indicative of the information responsive to the selected one or more search queries; and

incorporating, by the automated assistant into the existing human-to-computer dialog session, the unsolicited content;

wherein at least the incorporating is performed in response to the determining that the automated assistant has responded to all natural language input received from the user during the human-to-computer dialog session.

2. The method of claim 1 , wherein the unsolicited content comprises unsolicited natural language content.

3. The method of claim 1 , further comprising determining a desirability measure that is indicative of the user's desire to receive unsolicited content, wherein the desirability measure is determined based at least in part on the current context of the user, and wherein at least the incorporating is performed in response to a determination that the desirability measure satisfies one or more thresholds.

4. The method of claim 1 , wherein the current context of the user is determined based on one or more signals generated by one or more sensors integral with one or more of the computing devices.

5. The method of claim 4 , wherein the current context of the user comprises position coordinates generated by a position coordinate sensor of one or more of the computing devices operated by the user.

6. The method of claim 4 , wherein the current context of the user comprises sensor data generated by an accelerometer of one or more of the computing devices operated by the user.

7. The method of claim 1 , wherein the current context of the user comprises a location of the user.

8. The method of claim 7 , wherein the location of the user is determined based on a schedule or calendar of the user.

9. The method of claim 7 , wherein the selected one or more search queries seek information about the location of the user.

10. A system comprising one or more processors and memory storing instructions that, in response to execution by the one or more processors, cause the one or more processors to:

determine that in an existing human-to-computer dialog session between a user and an automated assistant occurring at one or more computing devices operated by the user, the automated assistant has responded to all natural language input received from the user during the human-to-computer dialog session;

determine a current context of the user;

analyze search queries submitted by others to identify spikes, trends, or other patterns in the search queries that are submitted by others in contexts similar to the current context of the user;

based on the spikes, trends, or other patterns, select one or more of the search queries submitted by others;

search one or more online sources for information responsive to the selected one or more search queries;

generate, by one or more of the processors, unsolicited content indicative of the information responsive to the selected one or more search queries; and

incorporate, by the automated assistant into the existing human-to-computer dialog session, the unsolicited content;

wherein at least the incorporation is performed in response to the determination that the automated assistant has responded to all natural language input received from the user during the human-to-computer dialog session.

11. The system of claim 10 , wherein the unsolicited content comprises unsolicited natural language content.

12. The system of claim 10 , further comprising instructions to determine a desirability measure that is indicative of the user's desire to receive unsolicited content, wherein the desirability measure is determined based at least in part on the current context of the user, and wherein at least the incorporating is performed in response to a determination that the desirability measure satisfies one or more thresholds.

13. The system of claim 10 , wherein the current context of the user is determined based on one or more signals generated by one or more sensors integral with one or more of the computing devices.

14. The system of claim 13 , wherein the current context of the user comprises position coordinates generated by a position coordinate sensor of one or more of the computing devices operated by the user.

15. The system of claim 13 , wherein the current context of the user comprises sensor data generated by an accelerometer of one or more of the computing devices operated by the user.

16. The system of claim 10 , wherein the current context of the user comprises a location of the user.

17. The system of claim 16 , wherein the location of the user is determined based on a schedule or calendar of the user.

18. The system of claim 16 , wherein the selected one or more search queries seek information about the location of the user.

19. At least one non-transitory computer-readable medium comprising instructions that, in response to execution by one or more processors, cause the one or more processors to:

determine that in an existing human-to-computer dialog session between a user and an automated assistant occurring at one or more computing devices operated by the user, the automated assistant has responded to all natural language input received from the user during the human-to-computer dialog session;

determine a current context of the user;

analyze search queries submitted by others to identify spikes, trends, or other patterns in the search queries that are submitted by others in contexts similar to the current context of the user;

based on the spikes, trends, or other patterns, select one or more of the search queries submitted by others;

search one or more online sources for information responsive to the selected one or more search queries;

generate, by one or more of the processors, unsolicited content indicative of the information responsive to the selected one or more search queries; and

incorporate, by the automated assistant into the existing human-to-computer dialog session, the unsolicited content;

wherein at least the incorporation is performed in response to the determination that the automated assistant has responded to all natural language input received from the user during the human-to-computer dialog session.

20. The at least one non-transitory computer-readable medium of claim 19 , wherein the unsolicited content comprises unsolicited natural language content.

Assignments (2)
CHANGE OF NAME Recorded Mar 18, 2024
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 066816/0663 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 16, 2024
From: BADR, IBRAHIM; SABUR, ZAHEED; VUSKOVIC, VLADIMIR; ZUMBRUNNEN, ADRIAN; MIRELMANN, LUCAS
To: GOOGLE INC.
Reel/Frame 066135/0174 →
Continuity (4)
Continuation 17572293 · Jan 10, 2022
Continuation 16829323 · Mar 25, 2020
Continuation 15466422 · Mar 22, 2017
Related Publication 20240112679A1 · Apr 4, 2024
References Cited (148)
US 6731307B1 · Strubbe et al. · 2004 [cited by applicant]
US 8612226B1 · Epstein et al. · 2013 [cited by applicant]
US 9117447B2 · Gruber et al. · 2015 [cited by applicant]
US 9368114B2 · Larson et al. · 2016 [cited by applicant]
US 9412392B2 · Lindahl · 2016 [cited by applicant]
US 9449052B1 · Rivard · 2016 [cited by examiner]
US 9865260B1 · Vuskovic et al. · 2018 [cited by applicant]
US 10049664B1 · Indyk · 2018 [cited by applicant]
US 10482882B2 · Vuskovic et al. · 2019 [cited by applicant]
US 10636418B2 · Badr · 2020 [cited by examiner]
US 11114100B2 · Vuskovic et al. · 2021 [cited by applicant]
US 11232792B2 · Badr et al. · 2022 [cited by applicant]
US 20040162724A1 · Hill et al. · 2004 [cited by applicant]
US 20050054381A1 · Lee · 2005 [cited by applicant]
US 20050131695A1 · Lucente et al. · 2005 [cited by applicant]
US 20070001636A1 · Murphree · 2007 [cited by applicant]
US 20070201636A1 · Gilbert et al. · 2007 [cited by applicant]
US 20080115068A1 · Smith · 2008 [cited by applicant]
US 20080189110A1 · Freeman et al. · 2008 [cited by applicant]
US 20080289002A1 · Portele · 2008 [cited by applicant]
US 20090150156A1 · Kennewick et al. · 2009 [cited by applicant]
US 20100088100A1 · Lindahl · 2010 [cited by applicant]
US 20100205541A1 · Rapaport et al. · 2010 [cited by applicant]
US 20100217657A1 · Gazdzinski · 2010 [cited by applicant]
US 20100241963A1 · Kulis et al. · 2010 [cited by applicant]
US 20100250672A1 · Vance et al. · 2010 [cited by applicant]
US 20110066634A1 · Phillips et al. · 2011 [cited by applicant]
US 20110271194A1 · Lin et al. · 2011 [cited by applicant]
US 20120290950A1 · Rapaport et al. · 2012 [cited by applicant]
US 20130110505A1 · Gruber et al. · 2013 [cited by applicant]
US 20130159377A1 · Nash · 2013 [cited by applicant]
US 20130275164A1 · Gruber et al. · 2013 [cited by applicant]
US 20130317823A1 · Mengibar · 2013 [cited by applicant]
US 20140129651A1 · Gelfenbeyn et al. · 2014 [cited by applicant]
US 20140164312A1 · Lynch et al. · 2014 [cited by applicant]
US 20140229471A1 · Galvin, Jr. et al. · 2014 [cited by applicant]
US 20140236953A1 · Rapaport et al. · 2014 [cited by applicant]
US 20140278400A1 · Coussemaeker et al. · 2014 [cited by applicant]
US 20140310001A1 · Kalns et al. · 2014 [cited by applicant]
US 20140310002A1 · Nitz et al. · 2014 [cited by applicant]
US 20150046147A1 · Waibel et al. · 2015 [cited by applicant]
US 20150162000A1 · Di Censo et al. · 2015 [cited by applicant]
US 20150169284A1 · Quast et al. · 2015 [cited by applicant]
US 20150169336A1 · Harper et al. · 2015 [cited by applicant]
US 20150172262A1 · Ortiz et al. · 2015 [cited by applicant]
US 20150178388A1 · Winnemoeller et al. · 2015 [cited by applicant]
US 20150254058A1 · Klein et al. · 2015 [cited by applicant]
US 20150269612A1 · Cucerzan · 2015 [cited by applicant]
US 20160049149A1 · Lacher · 2016 [cited by applicant]
US 20160217784A1 · Gelfenbeyn et al. · 2016 [cited by applicant]
US 20160294739A1 · Stoehr et al. · 2016 [cited by applicant]
US 20160321573A1 · Vangala et al. · 2016 [cited by applicant]
US 20160373891A1 · Ramer et al. · 2016 [cited by applicant]
US 20170006356A1 · Krasadakis · 2017 [cited by applicant]
US 20170076327A1 · Filippini et al. · 2017 [cited by applicant]
US 20170083628A1 · Frenkel et al. · 2017 [cited by applicant]
US 20170180276A1 · Gershony et al. · 2017 [cited by applicant]
US 20170255989A1 · Calio · 2017 [cited by applicant]
US 20180053114A1 · Adjaoute · 2018 [cited by applicant]
US 20180061400A1 · Carbune et al. · 2018 [cited by applicant]
US 20180061421A1 · Sarikaya · 2018 [cited by applicant]
US 20180063384A1 · Kudo · 2018 [cited by applicant]
US 20180068656A1 · Lehman et al. · 2018 [cited by applicant]
US 20180082682A1 · Erickson et al. · 2018 [cited by applicant]
US 20180098030A1 · Morabia et al. · 2018 [cited by applicant]
US 20180137856A1 · Gilbert · 2018 [cited by applicant]
US 20180183748A1 · Zhang et al. · 2018 [cited by applicant]
US 20180373405A1 · Donahue et al. · 2018 [cited by applicant]
US 20220130386A1 · Badr et al. · 2022 [cited by applicant]
CN 101588323 · 2009 [cited by applicant]
CN 102750270 · 2012 [cited by applicant]
CN 102792320 · 2012 [cited by applicant]
CN 102947823 · 2013 [cited by applicant]
CN 103443853 · 2013 [cited by applicant]
CN 103577531 · 2014 [cited by applicant]
CN 103620605 · 2014 [cited by applicant]
CN 104254867 · 2014 [cited by applicant]
CN 104603830 · 2015 [cited by applicant]
CN 104769584 · 2015 [cited by applicant]
CN 104813311 · 2015 [cited by applicant]
CN 104871150 · 2015 [cited by applicant]
CN 104951428 · 2015 [cited by applicant]
CN 105247511 · 2016 [cited by applicant]
CN 105359138 · 2016 [cited by applicant]
CN 105830048 · 2016 [cited by applicant]
CN 105930367 · 2016 [cited by applicant]
CN 106020488 · 2016 [cited by applicant]
CN 103226949 · 2017 [cited by applicant]
CN 106559321 · 2017 [cited by applicant]
EP 2219141 · 2010 [cited by applicant]
EP 2884409 · 2015 [cited by applicant]
JP H10288532 · 1998 [cited by applicant]
JP 2001188784 · 1999 [cited by applicant]
JP 2001290493 · 2001 [cited by applicant]
JP 2003108191 · 2003 [cited by applicant]
JP 2005167628 · 2005 [cited by applicant]
JP 2009116552 · 2009 [cited by applicant]
JP 2010191486 · 2010 [cited by applicant]
JP 2015115069 · 2015 [cited by applicant]
JP 2015156231 · 2015 [cited by applicant]
JP 2016071192 · 2016 [cited by applicant]
JP 2016191791 · 2016 [cited by applicant]
KR 20130000423 · 2013 [cited by applicant]
WO 2011086053 · 2011 [cited by applicant]
WO 2015187048 · 2015 [cited by applicant]
WO 2016129276 · 2016 [cited by applicant]
Japanese Patent Office; Notice of Reasons for Rejection issued in App. No. 2023039772; 10 pages, dated Apr. 15, 2024. [cited by applicant]
Koichiro Yoshino et al.; Evaluation of Dialogue System based on Information Extraction from Web Text, Information Processing, No. 82; Information Processing Society of Japan; 6 pages; dated Aug. 15, 2010. [cited by applicant]
Korean Patent Office; Decision of Rejection issued in Application No. 20197031202; 8 pages; dated Oct. 26, 2021. [cited by applicant]
China National Intellectual Property Administration; Notification of First Office Action issued in Application No. 201711146949.2: 16 pages; dated Jun. 3, 2021. [cited by applicant]
Korean Patent Office: Office Action issued in Application No. 20197031202; 15 pages; dated Mar. 31, 2021. [cited by applicant]
Japanese Patent Office: Office Action issued for Application No. 2019-560296 dated Nov. 2, 2020. [cited by applicant]
Yoshino et al., Spoken Dialogue System Using Information Extraction from the Web (No. 82, pp. 1-6), Information Processing Society of Japan, Spoken Language Processing (SLP) dated Oct. 15, 2010. [cited by applicant]
Yoshino et al., Spoken Dialogue System Based on Information Extraction and Presentation Using Similarity of Predicate Argument Structures (vol. 52, No. 12, pp. 3386-3397), Journal of Information Processing Society of Ja… [cited by applicant]
Yoshino et al., “Spoken Dialogue System based on Information Extraction from Web Text” The Special Interest Group Technical Reports of IPSJ Spoken Language Information Processing (SLP) No. 82. Aug. 15, 2010, pp. 1-6 201… [cited by applicant]
Japanese Patent Office; Notice of Allowance issue in Application No. 2019-560296; 3 pages; dated Jan. 25, 2021. [cited by applicant]
“Bots: An introduction for developers;” 13 sheets [online] [found on May 15, 2017], available in the Internet as URL; https://core.telegram.org/bots 2017. [cited by applicant]
Constine, Josh “Facebook will launch group chatbots at F8;” Posted Mar. 29, 2017, TechCrunch, 9 sheets [online] [found on May 15, 2017], available in the Internet as URL: https://techcrunch.com/2017/03/29/facebook-group… [cited by applicant]
McHugh, Molly “Slack is Overrun with Bots. Friendly, Wonderful Bots;” Aug. 21, 2015, Then One/Wired, 10 sheets [online] [found on May 15, 2017], available in the Internet as URL: https://www.wired.com/2015/08/slack-over… [cited by applicant]
Metz, Rachel “Messaging App Adds and Assistant to the Conversation:” Apr. 4, 2014, MIT Technology Review, 9 sheets [online] [found on May 15, 2017], available in the Internet as URL: https://www.technologyreview.com/s/5… [cited by applicant]
Minker, W., et al. “Next-generation human-computer interfaces—towards intelligent, adaptive and proactive spoken language dialogue systems.” In Intelligent Environments, 2006. IE 06. 2nd IET International Conference on … [cited by applicant]
European Patent Office; Invitation to Pay Additional Fees in International Patent Application No. PCT/US2017/059110; 13 pages; dated Feb. 22, 2018. [cited by applicant]
Becker, Christian, et al. “Simulating the Emotion Dynamics of a Multimodal Conversational Agent.” In Tutorial and Research Workshop on Affective Dialogue Systems, pp. 154-165. Springer Berlin Heidelberg, 2004. [cited by applicant]
L'Aabbate, Marcello. “Modelling Proactive Behaviour of Conversational Interfaces.” PhD diss., Technische Universität; 170 pages. 2006. [cited by applicant]
European Patent Office: International Search Report and Written Opinion of PCT Ser. No. PCT/US2018/040057; 15 pages; dated Sep. 28, 2018. [cited by applicant]
International Search Report and Written Opinion fo PCT Ser. No. PCT/US2017/059110, dated May 30, 2018; 16 pages. [cited by applicant]
European Patent Office; Communication pursuant to Article 94(3) EPC issued in Application No. 18727097.0; 8 pages; dated Nov. 8, 2021. [cited by applicant]
Korean Patent Office; Notice of Allowance issued in Application No. 10-2019-7035659; 4 pages; dated Sep. 29, 2021. [cited by applicant]
China National Intellectual Property Administration; Notification of First Office Action issued in Application No. 201880035874.7; 21 pages; dated Apr. 26, 2021. [cited by applicant]
Japanese Patent Office; Notice of Allowance issued in Application No. 2019-552127; 3 pages; dated May 24, 2021. [cited by applicant]
Intellectual Property India; Office Action issued in Application No. 201927043964; 7 pages; dated May 13, 2021. [cited by applicant]
Korean Patent Office; Notice of Office Action issued in Application No. 10-2019-7035659; 6 pages; dated Mar. 2, 2021. [cited by applicant]
European Patent Office; Communication issued in Application No. 20213009.2; 9 pages; dated Mar. 19, 2021. [cited by applicant]
Japanese Patent Office: Office Action issued for Application No. 2019-552127 dated Nov. 2, 2020. [cited by applicant]
Japanese Patent Office: Office Action issued in Application No. 2019-552127 dated Nov. 2, 2020. [cited by applicant]
European Patent Office; Intention to Grant issued in Application No. 18749908.2; 48 pages; dated Oct. 13, 2020. [cited by applicant]
The European Patent Office; Examination Report issued in Application No. 18749908.2 dated Sep. 27, 2019. Sep. 27, 2019. [cited by applicant]
The European Patent Office; Examination Report issued in Application No. 18749908.2 dated Sep. 27, 2019. [cited by applicant]
International Search Report and Written Opinion of PCT Ser. No. PCT/US2018/030317, dated Jul. 25, 2018; 16 pages Jul. 25, 2018. [cited by applicant]
European Patent Office; Communication pursuant to Article 64(3) EPC issued in Application No. 17805003.5; 5 pages; dated Mar. 14, 2023. [cited by applicant]
European Patent Office; Communication pursuant to Article 164 (2)(b) and Article 94(3) EPC issued in Application No. 17805003.5; 33 pages; dated Jul. 1, 2021. [cited by applicant]
Korean Patent Office; Notice of Office Action issued in Application No. KR10-2022-7014294; 11 pages; dated Sep. 19, 2022. [cited by applicant]
Japanese Patent Office; Notice of Reasons for Rejection issued in App. No. 2021-104078, 6 pages, dated Jul. 25, 2022. [cited by applicant]
Koichiro Yoshino, Spoken Dialogue System based on Information Extraction from Web Text, Information Processing Society of Japan Research Report, H22. [CD ROM], Japan, Information Processing Society; dated 2010. [cited by applicant]
China National Intellectual Property Administration: Decision of Rejection issued for Application No. 201711146949.2, 5 pages, dated Feb. 8, 2022. [cited by applicant]
Korean Patent Office; Notice of Allowance issued in Application No. 10-2019-7031202; 3 pages; dated Jan. 28, 2022. [cited by applicant]
Japanese Patent Office; Decision of Rejection issued in App. No. 2023039772; 10 pages, dated Aug. 13, 2024. [cited by applicant]
European Patent Office; Intention to Grant issued in Application No. 17805003.5; 55 pages; dated Sep. 30, 2024. [cited by applicant]