IP Library › Granted Patent US 12681964

Granted Patent B2

US 12681964 · App. 18/225,990 · Granted Jul 14, 2026

Blockwise controlled decoding of natural language (NL) based output generated using a large language model (LLM) to reduce latency in rendering thereof

Inventors: Sidharth Mudgal (Mountain View, CA); Ahmad Beirami (New York, NY); Jilin Chen (Cupertino, CA); Alex Beutel (Brooklyn, NY); Harish Ganapathy (Sunnyvale, CA); YaGuang Li (Sunnyvale, CA); Tao Wang (Sunnyvale, CA); Yanping Huang (Mountain View, CA); Trevor Strohman (Sunnyvale, CA)

Assignee: GOOGLE LLC

G06F16/3329 G06F40/284

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12681964
App. No.: 18/225,990
Granted: Jul 14, 2026
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

Implementations relate to reducing latency in generating and/or rendering a given stream of natural language (NL) based output generated using a large language model (LLM). Processor(s) of a system can: receive NL based input associated with a client device, generate the stream of NL based output utilizing the LLM that is responsive to the NL based input and that is for a given dialog context of an ongoing dialog, and cause the stream of NL based output to be rendered at the client device. Notably, the processor(s) can employ attribute classifier(s) and a multi-objective scorer to implement a blockwise controlled decoding technique in generating the stream of NL based output utilizing the LLM. By implementing the blockwise controlled decoding technique in generating the stream of NL based output utilizing the LLM, the processor(s) can reduce latency in generating and/or of the stream of NL based output generated utilizing the LLM.

Claims (71)

1 . A method implemented by one or more processors, the method comprising:

receiving natural language (NL) based input associated with a client device and as part of an ongoing dialog;

generating, based on processing the NL based input using a large language model (LLM), a given stream of NL based output that is responsive to the NL based input on a segment-by-segment basis and that is for a given dialog context of the ongoing dialog, wherein the given stream of NL based output includes a plurality of segments, and wherein generating a given segment, of the plurality of segments, of the given stream of NL based output that is responsive to the NL based input and based on processing the NL based input using the LLM comprises:

processing, using the LLM, the NL based input to decode a stream of LLM output;

determining, based on the stream of LLM output, a plurality of candidate segments for the given segment as the stream of LLM output is being decoded;

processing, using a plurality of attribute classifiers, each of the plurality of candidate segments, for the given segment, to generate a corresponding set of a plurality of attribute scores for each of the plurality of candidate segments;

processing, using a multi-objective scorer, the corresponding sets of the plurality of attribute scores, for each of the plurality of candidate segments, to select the given segment, from among the plurality of candidate segments, for inclusion in the given stream of NL based output that is responsive to the NL based input and that is for the given dialog context of the ongoing dialog; and

incorporating the given segment into the given dialog context of the ongoing dialog for utilization in determining, based on the stream of LLM output, a plurality of additional candidate segments for a given additional segment, that follows the given segment in the given stream of NL based output, as the stream of LLM output is being decoded; and

causing the given stream of NL based output to be rendered at the client device on the segment-by-segment basis as each of the plurality of segments are selected for inclusion in the given stream of NL based output that is responsive to the NL based input and that is for the given dialog context of the ongoing dialog.

2 . The method of claim 1 , wherein generating the given additional segment, that follows the given segment in the given stream of NL based output comprises:

determining, based on the stream of LLM output, the plurality of additional candidate segments for the given additional segment as the stream of LLM output is being decoded;

processing, using the plurality of attribute classifiers, each of the plurality of additional candidate segments, for the given additional segment, to generate a corresponding additional set of a plurality of attribute scores for each of the plurality of additional candidate segments; and

processing, using the multi-objective scorer, the corresponding additional sets of the plurality of attribute scores, for each of the plurality of additional candidate segments, to select the given additional segment, from among the plurality of additional candidate segments, for inclusion in the given stream of NL based output that is responsive to the NL based input and that is for the given dialog context of the ongoing dialog.

3 . The method of claim 1 , further comprising:

generating, based on processing the NL based input using the LLM, a given alternative stream of NL based output that is also responsive to the NL based input on the segment-by-segment basis and that is for a given alternative dialog context of the ongoing dialog, wherein the given alternative stream of NL based output includes a plurality of alternative segments, and wherein generating a given alternative segment, of the plurality of alternative segments, of the given alternative stream of NL based output that is also responsive to the NL based input and based on processing the NL based input using the LLM comprises:

determining, based on the stream of LLM output, a plurality of alternative candidate segments for the given alternative segment as the stream of LLM output is being decoded;

processing, using the plurality of attribute classifiers, each of the plurality of alternative candidate segments, for the given alternative segment, to generate a corresponding alternative set of a plurality of attribute scores for each of the plurality of alternative candidate segments;

processing, using the multi-objective scorer, the corresponding alternative sets of the plurality of attribute scores, for each of the plurality of alternative candidate segments, to select the given alternative segment, from among the plurality of alternative candidate segments, for inclusion in the given alternative stream of NL based output that is also responsive to the NL based input and that is for the given alternative dialog context of the ongoing dialog; and

incorporating the given alternative segment into the given alternative dialog context of the ongoing dialog for utilization in determining, based on the stream of LLM output, a plurality of additional alternative candidate segments for a given additional alternative segment, that follows the given alternative segment in the given alternative stream of NL based output, as the stream of LLM output is being decoded; and

causing the given alternative stream of NL based output to be rendered at the client device on the segment-by-segment basis as each of the plurality of alternative segments are selected for inclusion in the given alternative stream of NL based output that is responsive to the NL based input and that is for the given alternative dialog context of the ongoing dialog.

4 . The method of claim 3 , wherein generating the given additional alternative segment, that follows the given alternative segment in the given alternative stream of NL based output comprises:

determining, based on the stream of LLM output, the plurality of additional alternative candidate segments for the given additional alternative segment as the stream of LLM output is being decoded;

processing, using the plurality of attribute classifiers, each of the plurality of additional alternative candidate segments, for the given additional alternative segment, to generate a corresponding additional alternative set of a plurality of attribute scores for each of the plurality of additional alternative candidate segments; and

processing, using the multi-objective scorer, the corresponding additional alternative sets of the plurality of attribute scores, for each of the plurality of additional alternative candidate segments, to select the given additional alternative segment, from among the plurality of additional alternative candidate segments, for inclusion in the given alternative stream of NL based output that is also responsive to the NL based input and that is for the given alternative dialog context of the ongoing dialog.

5 . The method of claim 4 , wherein both the given stream of NL based output that is for the given dialog context of the ongoing dialog and the given alternative stream of NL based output that is for the given alternative dialog context of the ongoing dialog are rendered at the client device.

6 . The method of claim 5 , wherein the given stream of NL based output that is for the given dialog context of the ongoing dialog is rendered at the client device more prominently than the given alternative stream of NL based output that is for the given alternative dialog context of the ongoing dialog.

7 . The method of claim 1 , wherein the plurality of attribute classifiers include at least a filtering classifier and one or more ranking classifiers, and wherein processing a given candidate segment, of the plurality of candidate segments, to generate the corresponding set of the plurality of attribute scores for the given candidate segment and using the plurality of attribute classifiers comprises:

processing, using the filtering classifier, the given candidate segment to determine a corresponding filtering score for the given candidate segment;

processing, using the one or more ranking classifiers, the given candidate segment to determine one or more corresponding ranking scores for the given candidate segment; and

generating, based on the corresponding filtering score for the given candidate segment and the one or more corresponding ranking scores for the given candidate segment, the corresponding set of the plurality of attribute scores for the given candidate segment.

8 . The method of claim 7 , wherein processing the corresponding sets of the plurality of attribute scores, for each of the plurality of candidate segments, to select the given segment, from among the plurality of candidate segments, for inclusion in the given stream of NL based output that is responsive to the NL based input and that is for the given dialog context of the ongoing dialog and using the multi-objective scorer comprises:

filtering, based on the corresponding filtering scores for each of the plurality of candidate segments, to generate a subset of the plurality of candidate segments;

ranking, based on the one or more corresponding ranking scores for each of the plurality of candidate segments in the subset, each of the plurality of candidate segments that are included in the subset; and

selecting, based on the ranking, the given segment from among the plurality of candidate segments.

9 . The method of claim 8 , wherein the multi-objective scorer includes a plurality of tunable parameters that are dynamically adjustable throughout the ongoing dialog to dynamically adjust the filtering and the ranking.

10 . The method of claim 7 , wherein the filtering classifier is an assurance filtering classifier that is trained to generate an assurance score as the corresponding filtering scores for each of the plurality of candidate segments.

11 . The method of claim 10 , wherein the one or more ranking classifiers include one or more of: a quality ranking classifier that is trained to generate a quality score as the one or more corresponding ranking scores for each of the plurality of candidate segments, or an accuracy ranking classifier that is trained to generate an accuracy score as the one or more ranking scores for each of the plurality of candidate segments.

12 . The method of claim 1 , wherein the stream of LLM output comprises a probability distribution over a sequence of tokens, and wherein each of the plurality of candidate segments are selected from the sequence of tokens and based on the probability distribution.

13 . The method of claim 12 , wherein each of the plurality of candidate segments includes a fixed number of tokens.

14 . The method of claim 1 , wherein the multi-objective scorer is trained prior to the ongoing dialog being initiated and based on a plurality of training instances.

15 . A system comprising:

one or more hardware processors; and

memory storing instructions that, when executed by the one or more hardware processors, cause the one or more hardware processors to:

receive natural language (NL) based input associated with a client device and as part of an ongoing dialog;

generate, based on processing the NL based input using a large language model (LLM), a given stream of NL based output that is responsive to the NL based input on a segment-by-segment basis and that is for a given dialog context of the ongoing dialog, wherein the given stream of NL based output includes a plurality of segments, and wherein the instructions to generate a given segment, of the plurality of segments, of the given stream of NL based output that is responsive to the NL based input and based on processing the NL based input using the LLM comprise instructions to:

process, using the LLM, the NL based input to decode a stream of LLM output;

determine, based on the stream of LLM output, a plurality of candidate segments for the given segment as the stream of LLM output is being decoded;

process, using a plurality of attribute classifiers, each of the plurality of candidate segments, for the given segment, to generate a corresponding set of a plurality of attribute scores for each of the plurality of candidate segments;

process, using a multi-objective scorer, the corresponding sets of the plurality of attribute scores, for each of the plurality of candidate segments, to select the given segment, from among the plurality of candidate segments, for inclusion in the given stream of NL based output that is responsive to the NL based input and that is for the given dialog context of the ongoing dialog; and

incorporate the given segment into the given dialog context of the ongoing dialog for utilization in determining, based on the stream of LLM output, a plurality of additional candidate segments for a given additional segment, that follows the given segment in the given stream of NL based output, as the stream of LLM output is being decoded; and

cause the given stream of NL based output to be rendered at the client device on the segment-by-segment basis as each of the plurality of segments are selected for inclusion in the given stream of NL based output that is responsive to the NL based input and that is for the given dialog context of the ongoing dialog.

16 . The system of claim 15 , wherein the instructions to generate the given additional segment, that follows the given segment in the given stream of NL based output comprise instructions to:

determine, based on the stream of LLM output, the plurality of additional candidate segments for the given additional segment as the stream of LLM output is being decoded;

process, using the plurality of attribute classifiers, each of the plurality of additional candidate segments, for the given additional segment, to generate a corresponding additional set of a plurality of attribute scores for each of the plurality of additional candidate segments; and

process, using the multi-objective scorer, the corresponding additional sets of the plurality of attribute scores, for each of the plurality of additional candidate segments, to select the given additional segment, from among the plurality of additional candidate segments, for inclusion in the given stream of NL based output that is responsive to the NL based input and that is for the given dialog context of the ongoing dialog.

17 . The system of claim 15 , wherein the instructions further cause the one or more hardware processors to:

generate, based on processing the NL based input using the LLM, a given alternative stream of NL based output that is also responsive to the NL based input on the segment-by-segment basis and that is for a given alternative dialog context of the ongoing dialog, wherein the given alternative stream of NL based output includes a plurality of alternative segments, and wherein the instructions to generate a given alternative segment, of the plurality of alternative segments, of the given alternative stream of NL based output that is also responsive to the NL based input and based on processing the NL based input using the LLM comprise instructions to:

determine, based on the stream of LLM output, a plurality of alternative candidate segments for the given alternative segment;

process, using the plurality of attribute classifiers, each of the plurality of alternative candidate segments, for the given alternative segment, to generate a corresponding alternative set of a plurality of attribute scores for each of the plurality of alternative candidate segments;

process, using the multi-objective scorer, the corresponding alternative sets of the plurality of attribute scores, for each of the plurality of alternative candidate segments, to select the given alternative segment, from among the plurality of alternative candidate segments, for inclusion in the given alternative stream of NL based output that is also responsive to the NL based input and that is for the given alternative dialog context of the ongoing dialog; and

incorporate the given alternative segment into the given alternative dialog context of the ongoing dialog for utilization in determining, based on the stream of LLM output, a plurality of additional alternative candidate segments for a given additional alternative segment, that follows the given alternative segment in the given alternative stream of NL based output, as the stream of LLM output is being decoded; and

cause the given alternative stream of NL based output to be rendered at the client device on the segment-by-segment basis as each of the plurality of alternative segments are selected for inclusion in the given alternative stream of NL based output that is responsive to the NL based input and that is for the given alternative dialog context of the ongoing dialog.

18 . A non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors to perform operations to:

receive natural language (NL) based input associated with a client device and as part of an ongoing dialog;

generate, based on processing the NL based input using a large language model (LLM), a given stream of NL based output that is responsive to the NL based input on a segment-by-segment basis and that is for a given dialog context of the ongoing dialog, wherein the given stream of NL based output includes a plurality of segments, and wherein the operations to generate a given segment, of the plurality of segments, of the given stream of NL based output that is responsive to the NL based input and based on processing the NL based input using the LLM comprise operations to:

process, using the LLM, the NL based input to decode a stream of LLM output;

determine, based on the stream of LLM output, a plurality of candidate segments for the given segment as the stream of LLM output is being decoded;