Importance aware selective correction for speculative decoding in large models

1 . A method comprising:

obtaining, via a prompt interface, an input prompt to a large model (LM);

intercepting the input prompt by a draft model;

processing, by the draft model, the input prompt to generate a first set of output tokens in response to the input prompt;

triggering performing, by the LM, a verification of the first set of output tokens to reject the first set of output tokens;

processing, by a pivotal model, the first set of output tokens to classify the first set of output tokens as trivial; and

transmitting, as a first part of a response to the input prompt, the first set of output tokens on a communications network based on the first set of output tokens being classified as trivial regardless of the first set of output tokens being rejected by the LM;

further processing, by the draft model after generating the first set of output tokens, the input prompt to generate a second set of output tokens in response to the input prompt;

triggering performing, by the LM, a verification of the second set of output tokens to reject the second set of output tokens;

processing, by the pivotal model, the second set of output tokens using a context vector encoding a plurality of prior sets of output tokens, wherein processing the second set of output tokens classifies the second set of output tokens as pivotal, and wherein the plurality of prior sets comprises the first set of output tokens;

triggering regenerating, by the LM, the second set of output tokens responsive to the second set of output tokens being rejected by the LM and being marked as pivotal by the pivotal model to generate a set of revised output tokens; and

transmitting, as a second part of the response to the input prompt, the set of revised output tokens on the communications network.

2 . The method of claim 1 , wherein the first set of output tokens comprises a series of output tokens.

3 . The method of claim 2 , wherein the pivotal model processes the series of the output tokens in order to maintain a context of each token within the series.

4 . The method of claim 1 , wherein each output token of the first set of output tokens is an individual term in the response to the input prompt.

5 . The method of claim 1 , wherein the processing the first set of output tokens by the pivotal model is performed responsive to the LM rejecting the first set of output tokens.

6 . The method of claim 1 , wherein the processing by the pivotal model comprises:

executing a first layer on a feed forward neural network to generate an intermediate vector; and

executing a second layer of the feed forward neural network on the intermediate vector to generate a classification result, wherein the classification result identifies the first set of output tokens as trivial.

7 . The method of claim 1 , further comprising:

training the pivotal model on a prelabeled training dataset comprising a training document, wherein the training document comprises a pivotal label and a trivial label for each term of a plurality of terms in the training document.

8 . A system comprising:

at least one computer processor;

a draft model executing on the at least one computer processor and configured to:

intercept an input prompt,

process the input prompt to generate a first set of output tokens in response to the input prompt, and

further process, after generating the first set of output tokens, the input prompt to generate a second set of output tokens in response to the input prompt;

a Large Model (LM) executing on the at least one computer processor and configured to:

perform a verification of the first set of output tokens to reject the first set of output tokens,

perform a verification of the second set of output tokens to reject the second set of output tokens, and

regenerate the second set of output tokens responsive to the second set of output tokens being rejected by the LM and being marked as pivotal by a pivotal model to generate a set of revised output tokens;

the pivotal model executing on the at least one computer processor and configured to:

process the first set of output tokens to classify the first set of output tokens as trivial, and

process the second set of output tokens using a context vector encoding a plurality of prior sets of output tokens, wherein processing the second set of output tokens classifies the second set of output tokens as pivotal, and wherein the plurality of prior sets comprises the first set of output tokens; and

a prompt interface executing on the at least one computer processor and configured to:

obtain the input prompt to a language model,

transmit, as a first part of a response to the input prompt, the first set of output tokens on a communications network based on the first set of output tokens being classified as trivial regardless of the first set of output tokens being rejected by the LM, and

transmit, as a second part of the response to the input prompt, the set of revised output tokens on the communications network.

9 . The system of claim 8 , wherein the first set of output tokens comprises a series of output tokens.

10 . The system of claim 8 , wherein each output token of the first set of output tokens is an individual term in the response to the input prompt.

11 . The system of claim 8 , wherein the processing the first set of output tokens by the pivotal model is performed responsive to the LM rejecting the first set of output tokens.

12 . The system of claim 8 , wherein the processing by the pivotal model comprises:

executing a first layer on a feed forward neural network to generate an intermediate vector; and

executing a second layer of the feed forward neural network on the intermediate vector to generate a classification result, wherein the classification result identifies the first set of output tokens as trivial.

13 . The system of claim 8 , further comprising:

a training controller executing on the at least one computer processor and configured to train the pivotal model on a prelabeled training dataset comprising a training document, wherein the training document comprises a pivotal label and a trivial label for each term of a plurality of terms in the training document.

14 . A method comprising:

obtaining, via a prompt interface, an input prompt to a large model (LM);

intercepting the input prompt by a draft model;

processing, by the draft model, the input prompt to generate a first set of output tokens, a second set of output tokens, and a third set of tokens in response to the input prompt, wherein the generating the second set of output tokens is performed after the first set of output tokens is output by the draft model, and wherein the generating the third set of output tokens is performed after the second set of output tokens is output by the draft model;

triggering performing, by the LM, a verification of the first set of output tokens to reject the first set of output tokens;

processing, by a pivotal model, the first set of output tokens to classify the first set of output tokens as trivial;

transmitting, as a first part of a response to the input prompt, the first set of output tokens on a communications network based on the first set of output tokens being classified as trivial regardless of the first set of output tokens being rejected by the LM;

triggering performing, by the LM, a verification of the second set of output tokens to reject the second set of output tokens;

processing, by the pivotal model, the second set of output tokens using a context vector encoding a plurality of prior sets of output tokens, wherein processing the second set of output tokens classifies the second set of output tokens as pivotal, and wherein the plurality of prior sets comprises the first set of output tokens;

triggering regenerating, by the LM, the second set of output tokens responsive to the second set of output tokens being rejected by the LM and being marked as pivotal by the pivotal model to generate a set of revised output tokens;

transmitting, as a second part of the response to the input prompt, the set of revised output tokens on the communications network;

performing, by the LM, a verification of the third set of output tokens to accept the first set of output tokens; and

transmitting, as a third part of the response to the input prompt, the third set of output tokens on the communications network based on the third set of output tokens being accepted by the LM.

15 . The method of claim 14 , wherein the first set of output tokens comprises a series of output tokens.