Granted Patent B2

US 12,737,648 · App. 19/219,678 · Granted Sep 15, 2026

Inventors: Jason Weng Wei (Mountain View, CA); Dengyong Zhou (Redmond, WA); Dale Eric Schuurmans (Edmonton, CA); Quoc V. Le (Sunnyvale, CA); Maarten Paul Bosma (Cupertino, CA); Ed Huai-Hsin Chi (Palo Alto, CA); Olivier Jean Andrè Bousquet (Zürich, CH); Le Hou (South Setauket, NY); Nathan Kemp Sekiguchi Scales (Baar, CH); David J. Bieber (New York, NY); Charles Aloysius Sutton (Santa Clara, CA); Nathanael Martin Schärli (Pittsburgh, PA); Augustus Quadrozzi Odena (San Francisco, CA); Sharan Ajit Narang (San Francisco, CA); Guy Gur-Ari Krakover (Palo Alto, CA); Aakanksha Chowdhery (Santa Clara, CA); Aitor Lewkowycz (San Jose, CA); Jiageng Luan (San Francisco, CA); David Martin Dohan (San Francisco, CA); Henryk Michalewski (Oxfordshire, GB); Jacob Austin (New York, NY); Anders Johan Andreassen (Princeton, NJ); Maxwell Isaac Nye (San Francisco, CA); Xuezhi Wang (New York, NY)

Assignee: GOOGLE LLC

G06N5/022

Abstract

Claims (39)

1 . A computer-implemented method for increasing consistency of stochastic machine-learned model execution, the method comprising:

providing, by a computing system comprising one or more processors, as input to a machine-learned model, a model input comprising an operative query;

generating, by the computing system, a plurality of operative responses to the operative query, the plurality of operative responses based on a plurality of sampled outputs sampled, based on a probability distribution of outputs, from an output layer of the machine-learned model;

determining, by the computing system, a consistency metric based on the plurality of operative responses; and

generating, by the computing system, an operative response based on the consistency metric.

2 . The method of claim 1 , the consistency metric based on operative responses respectively associated with different operative traces.

3 . The method of claim 1 , the plurality of sampled outputs comprising a plurality of operative traces and the plurality of operative responses, the consistency metric evaluated only over the plurality of operative responses.

4 . The method of claim 1 , the operative query comprising image data for performing image analysis on the image data.

5 . The method of claim 1 , the plurality of sampled outputs sampled using temperature sampling.

6 . The method of claim 1 , the plurality of sampled outputs sampled from a top-k subset of the outputs.

7 . The method of claim 1 , each of the plurality of sampled outputs sampled independently.

8 . The method of claim 1 , the consistency metric comprising:

a plurality vote; or

a majority vote.

9 . The method of claim 1 , wherein the machine-learned model comprises a left-to-right decoder model.

10 . The method of claim 1 , the model input comprising an instructive sequence descriptive of an instructive query, an instructive response, and an instructive trace of intermediate states from the instructive query to the instructive response.

11 . The method of claim 2 , at least one operative trace comprising one or more intermediate states of one or more variables declared by a computer-executable coding language.

12 . The method of claim 1 , wherein the operative query is a first query component and the operative response is a first response component, and wherein the method comprises:

providing, by the computing system, as input to the machine-learned model, a second query component; and

generating, by the computing system, using the machine-learned model and responsive to the second query component, a second response component;

the first query component and the second query component generated using the machine-learned model based on a target query, first query component and the second query component corresponding to a breakdown of the target query.

13 . The method of claim 1 , the model input provided over a network to a server computing system for performing operations of the machine-learned model on a server of the server computing system.

14 . A computing system comprising:

one or more processors; and

one or more memory devices storing non-transitory computer-readable instructions executable to cause the one or more processors to perform operations, the operations comprising:

providing, as input to a machine-learned model, a model input comprising an operative query;

generating a plurality of operative responses to the operative query, the plurality of operative responses based on a plurality of sampled outputs sampled, based on a probability distribution of outputs, from an output layer of the machine-learned model;

determining a consistency metric based on the plurality of operative responses; and

generating an operative response based on the consistency metric.

15 . The computing system of claim 14 , the consistency metric based on operative responses respectively associated with different operative traces.

16 . The computing system of claim 14 , the plurality of sampled outputs comprising a plurality of operative traces and the plurality of operative responses, the consistency metric evaluated only over the plurality of operative responses.

17 . The computing system of claim 14 , the plurality of sampled outputs sampled using temperature sampling.

18 . The computing system of claim 14 , the plurality of sampled outputs sampled from a top-k subset of the outputs.

19 . The computing system of claim 14 , each of the plurality of sampled outputs sampled independently.

20 . One or more memory devices storing non-transitory computer-readable instructions executable to cause one or more processors to perform operations, the operations comprising:

providing, as input to a machine-learned model, a model input comprising an operative query;

generating a plurality of operative responses to the operative query, the plurality of operative responses based on a plurality of sampled outputs sampled, based on a probability distribution of outputs, from an output layer of the machine-learned model;

determining a consistency metric based on the plurality of operative responses; and

generating an operative response based on the consistency metric.

Assignments (1)

Continuity (5)

References Cited (94)