IP Library Granted Patent US 12,387,110
Granted Patent B2
US 12,387,110 · App. 18/378,033 · Granted Aug 12, 2025

Proactive virtual assistant

Inventors: Vikram Aggarwal (Palo Alto, CA); Deniz Binay (Mountain View, CA)
Assignee: GOOGLE LLC
G06N5/02G06Q10/10H04L51/02
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,387,110
App. No.
18/378,033
Granted
Aug 12, 2025
Kind
B2
Abstract

An assistant executing at, at least one processor, is described that determines content for a conversation with a user of a computing device and selects, based on the content and information associated with the user, a modality to signal initiating the conversation with the user. The assistant is further described that causes, in the modality, a signaling of the conversation with the user.

Claims (76)

1. A method implemented by one or more processors, the method comprising:

maintaining a list of computing devices that are associated with a user and that each provide access to an assistant;

determining, by the assistant, and without any user interaction from the user, content for a conversation with the user, wherein determining the content for the conversation comprises:

automatically generating, based on contextual information associated with one or more of the computing devices, a search query; and

determining, based on search results obtained responsive to executing the search query, the content for the conversation with the user; and

in response to determining the content for the conversation with the user:

determining, by the assistant, and from the list of computing devices, a given computing device to signal initiating of the conversation with the user;

determining, by the assistant, whether to signal the initiating of the conversation with the user; and

in response to determining to signal the initiating of the conversation with the user:

determining, by the assistant, a given modality to signal the initiating of the conversation with the user at the given computing device; and

causing, by the assistant, and in the given modality, a signaling of the conversation with the user via the given computing device.

2. The method of claim 1 , wherein determining the computing device to signal initiating of the conversation with the user comprises:

determining, by the assistant, that the user is travelling in a vehicle associated with the user; and

in response to determining that the user is travelling in the vehicle associated with the user:

determining, by the assistant, and from the list of computing devices, a vehicle computing device as the given computing device.

3. The method of claim 2 , wherein determining whether to signal the initiating of the conversation with the user comprises:

determining, by the assistant, whether a ranking of the content for the conversation satisfies a threshold; and

in response to determining that the ranking of the content for the conversation satisfies the threshold:

determining, by the assistant, to signal the initiating of the conversation with the user via the vehicle computing device.

4. The method of claim 3 , wherein determining the given modality to signal the initiating of the conversation with the user at the given computing device comprises:

determining, by the assistant, to signal the initiating of the conversation with the user via the vehicle computing device and in an audible modality.

5. The method of claim 4 , further comprising:

in response to determining that the ranking of the content for the conversation fails to satisfy the threshold:

determining, by the assistant, to refrain from signaling the initiating of the conversation with the user.

6. The method of claim 1 , wherein determining the computing device to signal initiating of the conversation with the user comprises:

determining, by the assistant, and from the list of computing devices, to utilize a most recently accessed computing device as the given computing device.

7. The method of claim 1 , wherein determining the computing device to signal initiating of the conversation with the user comprises:

determining, by the assistant, and from the list of computing devices, to utilize a most proximate computing device, that is most proximate to the user, as the given computing device.

8. The method of claim 7 , wherein determining whether to signal the initiating of the conversation with the user comprises:

determining, by the assistant, whether a ranking of the content for the conversation satisfies a threshold; and

in response to determining that the ranking of the content for the conversation satisfies the threshold:

determining, by the assistant, to signal the initiating of the conversation with the user via the most proximate computing device.

9. The method of claim 8 , wherein determining the given modality to signal the initiating of the conversation with the user at the given computing device comprises:

determining, by the assistant, capabilities of the most proximate computing device; and

selecting, by the assistant, and based on the ranking of the content and based on the capabilities of the most proximate computing device, the given modality.

10. The method of claim 1 , wherein determining the computing device to signal initiating of the conversation with the user comprises:

determining, by the assistant, and from the list of computing devices, the given computing device based on a time of day and/or location of the user.

11. The method of claim 10 , wherein determining whether to signal the initiating of the conversation with the user comprises:

determining, by the assistant, whether a ranking of the content for the conversation satisfies a threshold; and

in response to determining that the ranking of the content for the conversation satisfies the threshold:

determining, by the assistant, to signal the initiating of the conversation with the user via the given computing device.

12. The method of claim 11 , wherein determining given modality to signal the initiating of the conversation with the user at the given computing device comprises:

determining, by the assistant, capabilities of the given computing device; and

selecting, by the assistant, and based on the ranking of the content and based on the capabilities of the given computing device, the given modality.

13. The method of claim 1 , wherein the signaling of the conversation occurs without providing the content for the conversation with the user.

14. The method of claim 13 , further comprising:

subsequent to causing the signaling of the conversation without providing the content for the conversation with the user:

receiving, by the assistant, an indication, from the user via the given computing device, to initiate the conversation.

15. The method of claim 14 , further comprising:

responsive to receiving the indication to initiate the conversation from the user via the given computing device, causing, by the assistant, the content of the conversation with the user to be provided via the given computing device.

16. The method of claim 1 , wherein the contextual information associated with one or more of the computing devices comprises one or more of: sensor information obtained by one or more sensors of one or more of the computing devices, communication information sent to one or more of the computing devices or received from one or more of the computing devices, or application usage information associated with applications accessible by one or more of the computing devices.

17. The method of claim 16 , wherein the search results obtained responsive to executing the search query comprise one or more of: weather, traffic conditions, news, business operating hours, stock prices, sports scores, user schedules, transportation schedules, or retail prices.

18. A system, comprising:

one or more processors; and

memory storing instructions that, when executed, cause one or more of the processors to be operable to:

maintain a list of computing devices that are associated with a user and that each provide access to an assistant;

determine, by the assistant, and without any user interaction from the user, content for a conversation with the user, wherein the instructions to determine the content for the conversation comprise instructions to:

automatically generate, based on contextual information associated with one or more of the computing devices, a search query; and

determine, based on search results obtained responsive to executing the search query, the content for the conversation with the user;

in response to determining the content for the conversation with the user:

determine, by the assistant, and from the list of computing devices, a given computing device to signal initiating of the conversation with the user;

determine, by the assistant, whether to signal the initiating of the conversation with the user; and

in response to determining to signal the initiating of the conversation with the user:

determine, by the assistant, a given modality to signal the initiating of the conversation with the user at the given computing device; and

cause, by the assistant, and in the given modality, a signaling of the conversation with the user via the given computing device.

19. A non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors to perform operations, the operations comprising:

maintaining a list of computing devices that are associated with a user and that each provide access to an assistant;

determining, by the assistant, and without any user interaction from the user, content for a conversation with the user, wherein determining the content for the conversation comprises:

automatically generating, based on contextual information associated with one or more of the computing devices, a search query;

determining, based on search results obtained responsive to executing the search query, the content for the conversation with the user; and

in response to determining the content for the conversation with the user:

determining, by the assistant, and from the list of computing devices, a given computing device to signal initiating of the conversation with the user;

determining, by the assistant, whether to signal the initiating of the conversation with the user; and

in response to determining to signal the initiating of the conversation with the user:

determining, by the assistant, a given modality to signal the initiating of the conversation with the user at the given computing device; and

causing, by the assistant, and in the given modality, a signaling of the conversation with the user via the given computing device.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 24, 2023
From: AGGARWAL, VIKRAM; BINAY, DENIZ
To: GOOGLE INC.
Reel/Frame 065324/0367 →
CHANGE OF NAME Recorded Oct 24, 2023
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 065334/0297 →
Continuity (4)
Continuation 16779782 · Feb 3, 2020
Continuation 15381544 · Dec 16, 2016
Provisional Application 62408105 · Oct 14, 2016
Related Publication 20240037414A1 · Feb 1, 2024
References Cited (107)
US 6353827B1 · Davies · 2002 [cited by examiner]
US 6804332B1 · Miner et al. · 2004 [cited by applicant]
US 9275641B1 · Gelfenbeyn et al. · 2016 [cited by applicant]
US 10367950B2 · Davis · 2019 [cited by examiner]
US 10552742B2 · Aggarwal et al. · 2020 [cited by applicant]
US 20020178019A1 · Anderson et al. · 2002 [cited by applicant]
US 20030046421A1 · Horvitz et al. · 2003 [cited by applicant]
US 20030167167A1 · Gong · 2003 [cited by applicant]
US 20050054381A1 · Lee et al. · 2005 [cited by applicant]
US 20060240803A1 · Valeriano et al. · 2006 [cited by applicant]
US 20070130277A1 · Roskind et al. · 2007 [cited by applicant]
US 20080171559A1 · Frank et al. · 2008 [cited by applicant]
US 20080224883A1 · Mock · 2008 [cited by applicant]
US 20120265528A1 · Gruber et al. · 2012 [cited by applicant]
US 20130159441A1 · Appelman et al. · 2013 [cited by applicant]
US 20130275164A1 · Gruber et al. · 2013 [cited by applicant]
US 20130275875A1 · Gruber et al. · 2013 [cited by applicant]
US 20130311997A1 · Gruber et al. · 2013 [cited by applicant]
US 20140052680A1 · Nitz · 2014 [cited by applicant]
US 20140052681A1 · Nitz et al. · 2014 [cited by applicant]
US 20140119531A1 · Tuchman et al. · 2014 [cited by applicant]
US 20140164400A1 · Kruglick · 2014 [cited by applicant]
US 20140207469A1 · Dykstra-Erickson et al. · 2014 [cited by applicant]
US 20140222436A1 · Binder et al. · 2014 [cited by applicant]
US 20140282003A1 · Gruber et al. · 2014 [cited by applicant]
US 20140310005A1 · Brown et al. · 2014 [cited by applicant]
US 20140317502A1 · Brown et al. · 2014 [cited by applicant]
US 20140351744A1 · Jeon · 2014 [cited by applicant]
US 20150123898A1 · Kim et al. · 2015 [cited by applicant]
US 20150134761A1 · Sharma · 2015 [cited by examiner]
US 20150141055A1 · Barrand · 2015 [cited by applicant]
US 20150163188A1 · Faaborg et al. · 2015 [cited by applicant]
US 20150269009A1 · Faaborg et al. · 2015 [cited by applicant]
US 20150288745A1 · Moghaddam et al. · 2015 [cited by applicant]
US 20150296276A1 · Umetani · 2015 [cited by examiner]
US 20150350146A1 · Cary · 2015 [cited by examiner]
US 20150382147A1 · Clark · 2015 [cited by applicant]
US 20160173631A1 · McKay et al. · 2016 [cited by applicant]
US 20160180259A1 · Marianko · 2016 [cited by applicant]
US 20160205253A1 · Williams et al. · 2016 [cited by applicant]
US 20160210363A1 · Rambhia et al. · 2016 [cited by applicant]
US 20160246885A1 · Aravamudhan · 2016 [cited by examiner]
US 20160260431A1 · Newendorp et al. · 2016 [cited by applicant]
US 20160306798A1 · Guo · 2016 [cited by examiner]
US 20160335138A1 · Surti et al. · 2016 [cited by applicant]
US 20170091612A1 · Gruber et al. · 2017 [cited by applicant]
US 20170160813A1 · Divakaran et al. · 2017 [cited by applicant]
US 20170289329A1 · Yim et al. · 2017 [cited by applicant]
US 20170311053A1 · Ganjam et al. · 2017 [cited by applicant]
US 20170337035A1 · Choudhary et al. · 2017 [cited by applicant]
US 20170358305A1 · Kudurshian · 2017 [cited by examiner]
US 20180052824A1 · Ferrydiansyah · 2018 [cited by examiner]
US 20180063814A1 · Yuan · 2018 [cited by examiner]
US 20180096072A1 · He et al. · 2018 [cited by applicant]
CN 101043391 · 2007 [cited by applicant]
CN 102792320 · 2012 [cited by applicant]
CN 104335234 · 2015 [cited by applicant]
CN 105027197 · 2015 [cited by applicant]
CN 105122353 · 2015 [cited by applicant]
CN 105247511 · 2016 [cited by applicant]
CN 105264524 · 2016 [cited by applicant]
CN 105830048 · 2016 [cited by applicant]
CN 103226949 · 2017 [cited by applicant]
KR 20160105995 · 2016 [cited by applicant]
WO 200296126 · 2002 [cited by applicant]
WO 2008012537 · 2008 [cited by applicant]
China National Intellectual Property Administration; Notice of First Office Action issue for Application No. 202110325476.2; 13 pages; dated Oct. 20, 2023. [cited by applicant]
Weeratunga, A. M. et al; Project Nethra—An Intelligent Assistant for the Visually Disabled to Interact with Internet Services; IEEE 10th Conference on Industrial and Information Systems (ICIIS); 5 pages; dated Dec. 2015. [cited by applicant]
Huang, J.; What is Siri and how does it work?; Retrieved from the Internet: https://zhidao.baidu.com/question/134968451412138605.html; 4 pages; dated Oct. 23, 2013. [cited by applicant]
German Patent Office; Decision to Grant received in Application No. 10 2017 120 732.5, 11 pages, dated May 9, 2022. [cited by applicant]
Germany Patent Office; Examination Report issued in Application No. 10 2017 120 732.5; 8 pages; dated Oct. 11, 2021. [cited by applicant]
Deutsches Patent Office; Summons issued in Application No. 102017120732.5; 12 pages; dated Jul. 26, 2021. [cited by applicant]
Germany Patent Office; Office Action issued in Application No. 10 2017 120 732.5; 11 pages; dated Mar. 24, 2021. [cited by applicant]
China National Intellectual Property Administration; Notice of Allowance issue for Application No. 201710915581.5; 6 pages; dated Jan. 15, 2021. [cited by applicant]
Intellectual Property Office of Singapore; Notice of Allowance issued in Application No. 10201708331V; 6 pages; dated Dec. 18, 2021. [cited by applicant]
Gil, Lory; How to teach Siri your family and friends relationships; Retrieved from the Internet <URL: https://www.idownloadblog.com/2013/08/01siri-family-friends>; dated Aug. 1, 2013. [cited by applicant]
“How Siri Remembers my Family”, [online], Searched on Jun. 10, 2020, the Internet, <URL: http://gonta55.com/2015/09/22/739> dated Sep. 22, 2015. [cited by applicant]
China National Intellectual Property Administration; First Office Action issue for Application No. 201710915581.5; 20 pages; dated May 28, 2020. [cited by applicant]
Intellectual Property Office of Singapore; Invitation to Respond to Written Opinion issue in Application No. 10201904396W; 11 pages; dated Apr. 30, 2020. [cited by applicant]
Great Britain Intellectual Property Office; Examination Report issued in Application No. GB1715898.1 dated Apr. 21, 2020. [cited by applicant]
European Patent Office; Communication Pursuant to Article 94(3) issue in U.S. Appl. No. 17/783,682; dated Mar. 23, 2020. [cited by applicant]
Fu, R.Y. et al. “A framework for device capability on demand and virtual device user experience”; IBM J. Res. & Dev., pp. 635-648, vol. 48, No. 5/6 Sep./Nov. 2004. [cited by applicant]
Great Britain Intellectual Property Office; Examination Report issued in Application No. GB1715898.1 dated Aug. 8, 2019. [cited by applicant]
International Bureau of WIPO; International Preliminary Report on Patentability of Ser. No. PCT/US2017/054147; 7 pages; dated Apr. 25, 2019. [cited by applicant]
Intellectual Property Office of Singapore; Search Report and Written Opinion issued in Application No. 10201708331V dated Apr. 23, 2018. [cited by applicant]
European Patent Office; Written Opinion of the International Preliminary Examining Authority of PCT Ser. No. PCT/US2017/054147; 6 pages; dated Sep. 21, 2018. [cited by applicant]
European Patent Office; Written Opinion of the International Preliminary Examining Authority of PCT Ser. No. PCT/US2017/052285; 5 pages; dated Aug. 24, 2018. [cited by applicant]
International Search Report and Written Opinion of PCT Ser. No. PCT/US2017/054147; 12 pages dated Nov. 28, 2017. [cited by applicant]
Amazon, “Alexa Skills”, retrieved from: https://www.amazon.com/b?ie=UTF&&node=13727911011 on Jan. 26, 2017, 3 pages. [cited by applicant]
API.AI, “Build Conversation Actions for Google Assistant Users,” retrieved from: https://api.ai/blog/2016/12/08/build-conversations-actions-for-the-google-assistant/, Dec. 8, 2016, 2 Pages. [cited by applicant]
Badcock, “First Alexa Third-Party Skills Now Available for Amazon Echo,” retrieved from: https://developer.amazon.com/blogs/post/TxC2VHKFEIZ9SG/first-alexa-third-party-skills-now-available-for-amazon-echo, Aug. 17, 2015… [cited by applicant]
Newman, “It's On! 2017 is the Year The Virtual Assistant Wars Get Real,” retrieved from: https://www.fastcompany.com/3066831/its-on-2017-is-the-virtual-assistant-wars-get-real, Dec. 28, 2016. 9 Pages. [cited by applicant]
Intellectual Property Office of Singapore; Notice of Grant and Examination Report issued in Application No. 10201708331V dated Mar. 18, 2019. [cited by applicant]
International Search Report and Written Opinion of PCT Ser. No. PCT/US2017/052285; 11 pages dated Nov. 6, 2017. [cited by applicant]
United Kingdom Intellectual Property Office; Examination Report issued in Application No. 1715898.1 dated Mar. 22, 2018. [cited by applicant]
Irish Patent Office; Search Report issued in Application No. 2017/0177 dated May 11, 2018. [cited by applicant]
European Patent Office; International Preliminary Report on Patentability of PCT Ser. No. PCT/US2017/052285; 9 pages; dated Dec. 12, 2018. [cited by applicant]
European Patent Office; Examination Report issued for Application No. 17778044 dated Feb. 3, 2020 (6 Pages). [cited by applicant]
Japanese Patent Office; Notice of Reasons for Rejection issue in Application 2019-517895; 8 pages; dated Jun. 22, 2020. [cited by applicant]
European Patent Office: Summons to Attend Oral Proceedings issued for Application No. 17778044.2 dated Aug. 7, 2020. [cited by applicant]
Korean Intellectual Property Office: Reason For Refusal issued for Application No. 10-2019-7012954 dated Jul. 24, 2020. [cited by applicant]
Intellectual Property India; Examination Report issue in Application No. 201847014571; 7 pages; dated Nov. 5, 2020. [cited by applicant]
Koren Patent Office; Notification of Decision of Rejection issue in Application No. 10-2019-7012954; 14 pages; dated Jan. 26, 2021. [cited by applicant]
Japanese Patent Office, Notice of Reasons for Rejection issued in Application No. 2021-09071 dated Apr. 11, 2022. [cited by applicant]
Japanese Patent Office; Notice of Allowance issued in App. No. 2019-517895, 2 pages, dated Jul. 11, 2022. [cited by applicant]
Yousef, A. et al.; Activity-based cloud sending: Push services for user device multiplicity; 12th Annual IEEE Consumer Communications and Networking Conference (CCNC); dated 2015. [cited by applicant]
China National Intellectual Property Administration; Notice of Allowance issue for Application No. 202110325476.2; 4 pages; dated Apr. 21, 2024. [cited by applicant]