Granted Patent B2

US 12,118,325 · App. 18/232,144 · Granted Oct 15, 2024

Inventors: Matthew K. Gray (Reading, MA); John Blitzer (Mountain View, CA); Corinn Herrick (Mountain View, CA); Srinivasan Venkatachary (Sunnyvale, CA); Jayant Madhavan (San Francisco, CA); Sam Oates (Cambridge, MA); Phiroze Parakh (San Jose, CA); Aditya Shah (Mountain View, CA); Mahsan Rofouei (Menlo Park, CA); Ibrahim Badr (Zurich, CH)

Assignee: GOOGLE LLC

G06F40/40G06F16/3328

Abstract

Claims (54)

1. A method implemented by one or more processors, the method comprising:

receiving a query formulated based on user interface input at a client device;

determining, based on processing the query and/or based on processing at least one search result document of multiple search result documents that are responsive to the query, whether to utilize a large language model (LLM) in generating a natural language (NL) based summary to render responsive to receiving the query, wherein determining whether to utilize the LLM in generating the NL based summary to render responsive to receiving the query is based on processing the query and comprises:

determining that the query includes or omits one or more terms; and

determining that the inclusion or the omission of the one or more terms indicates that the LLM is to be utilized;

in response to determining to utilize the LLM in generating the NL based summary to render responsive to receiving the query:

causing a given NL based summary to be generated based on the query and using the LLM, and

causing the given NL based summary to be rendered at the client device in response to receiving the query and to be rendered at the client device along with rendering of search results that correspond to at least some of the search result documents that are responsive to the query.

2. The method of claim 1 , wherein determining whether to utilize the LLM in generating the NL based summary comprises:

processing the query using a classifier to generate classifier output; and

determining, based on the classifier output, to utilize the LLM.

3. The method of claim 1 , wherein determining whether to utilize the LLM in generating the NL based summary to render responsive to receiving the query is based on processing the at least one search result document.

4. The method of claim 3 , wherein determining whether to utilize the LLM in generating the NL based summary comprises:

determining, based on a type of the at least one search result document, to utilize the LLM in generating the NL based summary.

5. The method of claim 1 , wherein determining whether to utilize the LLM in generating the NL based summary to render responsive to receiving the query is based on processing the query and is based on processing the at least one search result document.

6. The method of claim 1 , wherein causing a given NL based summary to be generated based on the query and using the LLM comprises:

generating a prompt based on the query; and

causing the prompt to be processed, utilizing the LLM, to generate LLM output that reflects the NL based summary.

7. The method of claim 6 , wherein generating the prompt further comprises:

generating the prompt further based on corresponding content from a set of the search result documents that are responsive to the query.

8. The method of claim 7 , wherein the prompt includes the query, the corresponding content, and one or more additional terms that are in addition to those of the query and the corresponding content.

9. The method of claim 1 , further comprising

determining that a first portion of content, of the NL based summary, is verifiable based on first document content of a first document of the search result documents; and

determining that a second portion of content, of the NL based summary, is verifiable based on second document content of a second document of the search result documents, the second portion of content being disparate from the first portion of content;

wherein causing the NL based summary to be rendered at the client device comprises:

causing a first selectable graphical element to be rendered immediately after the first portion of content, wherein the first selectable graphical element, when selected, causes navigation to the first document, and wherein the first selectable graphical element is rendered immediately after the first portion of content responsive to determining that the first portion of content is verifiable based on the first document content of the first document, and

causing a second selectable graphical element to be rendered immediately after the second portion of content, wherein the second selectable graphical element, when selected, causes navigation to the second document, and wherein the second selectable graphical element is rendered immediately after the second portion of the content responsive to determining that the second portion of the content is verifiable based on the second document content of the second document.

10. A system, comprising:

memory storing instructions;

one or more processors operable to execute the instruction to:

receive a query associated with a client device;

in response to receiving the query:

determine, based on processing the query and/or based on processing at least one search result document of multiple search result documents that are responsive to the query, whether to utilize a large language model (LLM) in generating a natural language (NL) based summary to render responsive to receiving the query, wherein one or more of the processors are to process the query in determining whether to utilize the LLM in generating the NL based summary to render responsive to receiving the query and wherein in determining whether to utilize the LLM in generating the NL based summary to render responsive to receiving the query one or more of the processors are to:

determine that the query includes or omits one or more terms; and

determine that the inclusion or the omission of the one or more terms indicates that the LLM is to be utilized;

in response to determining to utilize the LLM in generating the NL based summary to render responsive to receiving the query:

generate large language model (LLM) output based on processing the query using the LLM;

generate a natural language (NL) based summary using the LLM output; and

cause the given NL based summary to be rendered at the client device in response to receiving the query and to be rendered at the client device along with rendering of search results that correspond to at least some of the search result documents that are responsive to the query.

11. The system of claim 10 , wherein one or more of the processors are to process the at least one search result document in determining whether to utilize the LLM in generating the NL based summary to render responsive to receiving the query.

12. The system of claim 10 , wherein one or more of the processors are to process the query and process the at least one search result document in determining whether to utilize the LLM in generating the NL based summary to render responsive to receiving the query.

13. The system of claim 10 , wherein in generating the LLM output based on processing the query using the LLM one or more of the processors are to:

generate a prompt that includes the query and additional content; and

process the prompt, utilizing the LLM, to generate the LLM output.

14. The system of claim 13 , wherein in generating the prompt one or more of the processors are further to:

generate at least some of the additional content, of the prompt, based on corresponding content from a set of the search result documents that are responsive to the query.

15. The system of claim 14 , wherein the additional content includes the corresponding content and includes one or more additional terms that are in addition to those of the query and the corresponding content.

16. The system of claim 10 , wherein one or more processors are further operable to execute the instruction to:

determine that a first portion of content, of the NL based summary, is verifiable based on first document content of a first document of the search result documents; and

determine that a second portion of content, of the NL based summary, is verifiable based on second document content of a second document of the search result documents, the second portion of content being disparate from the first portion of content;

wherein in causing the NL based summary to be rendered at the client device one or more of the processors are to:

cause a first selectable graphical element to be rendered immediately after the first portion of content, wherein the first selectable graphical element, when selected, causes navigation to the first document, and wherein the first selectable graphical element is rendered immediately after the first portion of content responsive to determining that the first portion of content is verifiable based on the first document content of the first document, and

cause a second selectable graphical element to be rendered immediately after the second portion of content, wherein the second selectable graphical element, when selected, causes navigation to the second document, and wherein the second selectable graphical element is rendered immediately after the second portion of the content responsive to determining that the second portion of the content is verifiable based on the second document content of the second document.

17. The system of claim 10 , wherein the query associated with the client device is automatically generated based on one or more of a profile of a user of the client device, a history of the user, and a context of the user.

Assignments (1)

Continuity (3)

Cited By (11)

US 12,322,402 US 12,423,388 US 12,632,514 US 12,632,515 US 12,632,516 US 12,632,517 US 12,632,518 US 12,633,275 US 12,633,276 US 12,639,405 US 12,645,766