Entropy-based set block decoding utilizing an adaptive entropy threshold
Certain aspects of the disclosure provide techniques for language model token prediction. A method generally includes processing an input token sequence to generate, for each respective token position of a plurality of token positions in an output token sequence: a probability distribution for a set of candidate output tokens associated with the respective token position; and an entropy score based on the probability distribution; for each respective token position, determining whether the entropy score satisfies a current adaptive entropy threshold, which is based on the entropy score associated with each respective token position; based on the determination for each respective token position, performing a decoding operation comprising: a set block decoding operation or a single-token decoding operation to generate one or more output tokens that correspond to one or more token positions; and generating a first version of the output token sequence including the one or more output tokens.
1 . A method of token prediction, comprising:
processing an input token sequence to generate, for each respective token position of a plurality of token positions in an output token sequence:
a probability distribution for a set of candidate output tokens associated with the respective token position, wherein the probability distribution comprises a probability score for each respective candidate output token in the set of candidate output tokens; and
an entropy score based on the probability distribution for the set of candidate output tokens;
for each respective token position of a set of token positions from the plurality of token positions in the output token sequence, determining whether the entropy score associated with the respective token position satisfies a current adaptive entropy threshold, wherein the current adaptive entropy threshold is based on the entropy score associated with each token position of the plurality of token positions;
based on the determination for each respective token position of the set of token positions, performing a decoding operation comprising:
a set block decoding operation to generate at least two output tokens that correspond to at least a first token position and a second token position of the plurality of token positions in the output token sequence; or
a single-token decoding operation to generate a single output token that corresponds to the first token position; and
generating a first version of the output token sequence including the at least two output tokens or the single output token.
2 . The method of claim 1 , wherein:
the set of token positions comprises at least two token positions, of the plurality of token positions, that correspond to at least the first token position and a second token position in the output token sequence,
the decoding operation comprises the set block decoding operation based on the entropy score associated with each respective token position of at least two token positions satisfying the current adaptive entropy threshold, and
generating the first version of the output token sequence comprises generating the first version of the output token sequence with the at least two output tokens.
3 . The method of claim 2 , wherein performing the set block decoding operation comprises decoding, in parallel, the sets of candidate output tokens corresponding to the at least two token positions to generate the at least two output tokens.
4 . The method of claim 1 , wherein:
the set of token positions comprises the first token position and the second token position in the output token sequence,
the decoding operation comprises the single-token decoding operation based on:
the entropy score associated with the first token position satisfying the current adaptive entropy threshold; and
the entropy score associated with the second token position not satisfying the current adaptive entropy threshold, and
generating the first version of the output token sequence comprises generating the first version of the output token sequence with the single output token.
5 . The method of claim 1 , wherein:
the set of token positions comprises the first token position in the output token sequence,
the decoding operation comprises the single-token decoding operation based on the entropy score associated with the first token position not satisfying the current adaptive entropy threshold, and
generating the first version of the output token sequence comprises generating the first version of the output token sequence with the single output token.
6 . The method of claim 1 , further comprising:
determining a current decoding cycle entropy score based on the entropy score associated with each token position of the plurality of token positions;
updating a sliding window of entropy scores to include the current decoding cycle entropy score, the sliding window of entropy scores comprising a plurality of previous decoding cycle entropy scores;
determining one or more trend metrics based on the sliding window of entropy scores; and
determining the current adaptive entropy threshold based on the trend metrics.
7 . The method of claim 6 , wherein the current decoding cycle entropy score comprises:
a mean entropy score;
a median entropy score; or
a weighted aggregation entropy score.
8 . The method of claim 6 , wherein the one or more trend metrics comprise at least one of:
a slope; or
a variance.
9 . The method of claim 6 , wherein determining the current adaptive entropy threshold comprises determining a first mapping between the one or more trend metrics and the current adaptive entropy threshold.
10 . The method of claim 6 , wherein determining the current adaptive entropy threshold comprises determining one or more rules.
11 . The method of claim 6 , wherein determining the current adaptive entropy threshold comprises:
determining at least one of the one or more trend metrics satisfies at least one threshold; and
adjusting, based on the determination, a previous adaptive entropy threshold by an increment.
12 . The method of claim 11 , wherein:
the one or more trend metrics comprise a slope,
determining the at least one of the one or more trend metrics satisfies the at least one threshold comprises determining the slope is greater than a slope threshold, and
adjusting the previous adaptive entropy threshold comprises decreasing the previous adaptive entropy threshold by the increment.
13 . The method of claim 11 , wherein:
the one or more trend metrics comprise a slope,
determining the at least one of the one or more trend metrics satisfies the at least one threshold comprises determining the slope is less than a slope threshold, and
adjusting the previous adaptive entropy threshold comprises increasing the previous adaptive entropy threshold by the increment.
14 . The method of claim 6 , wherein:
the method further comprises obtaining decoding feedback for a plurality of previous decoding cycles associated with the plurality of previous decoding cycle entropy scores, and
determining the current adaptive entropy threshold comprises adjusting a previous adaptive entropy threshold by an increment based on the decoding feedback.
15 . The method of claim 14 , wherein the decoding feedback comprises information about at least one of:
the plurality of previous decoding cycle entropy scores;
a latency metric associated with each respective previous decoding cycle of the plurality of previous decoding cycles; or
an accuracy metric associated with each respective previous decoding cycle of the plurality of previous decoding cycles.
16 . A method of adaptive entropy threshold adjustment for token prediction, comprising:
processing an input token sequence to generate, for each respective token position of a plurality of token positions in an output token sequence:
a probability distribution for a set of candidate output tokens associated with the respective token position, wherein the probability distribution comprises a probability score for each candidate output token in the set of candidate output tokens; and
an entropy score based on the probability distribution for the set of candidate output tokens;
determining a current decoding cycle entropy score based on the entropy score associated with each token position of the plurality of token positions;
updating a sliding window of entropy scores to include the current decoding cycle entropy score, the sliding window of entropy scores comprising a plurality of previous decoding cycle entropy scores;
determining one or more trend metrics based on the sliding window of entropy scores;
determining a current adaptive entropy threshold based on the trend metrics; and
predicting one or more tokens, associated with one or more token positions in the plurality of token positions, in the output token sequence based on the current adaptive entropy threshold.
17 . The method of claim 16 , wherein the current decoding cycle entropy score comprises:
a mean entropy score;
a median entropy score; or
a weighted aggregation entropy score.
18 . The method of claim 16 , wherein the one or more trend metrics comprise at least one of:
a slope; or
a variance.
19 . The method of claim 16 , wherein:
the method further comprises obtaining decoding feedback for a plurality of previous decoding cycles associated with the plurality of previous decoding cycle entropy scores, and
determining the current adaptive entropy threshold comprises adjusting a previous adaptive entropy threshold by an increment based on the decoding feedback.
20 . A processing system, comprising: memory comprising computer-executable instructions; and one or more processors configured to execute the computer-executable instructions and cause the processing system to:
process an input token sequence to generate, for each respective token position of a plurality of token positions in an output token sequence:
a probability distribution for a set of candidate output tokens associated with the respective token position, wherein the probability distribution comprises a probability score for each respective candidate output token in the set of candidate output tokens; and
an entropy score based on the probability distribution for the set of candidate output tokens;
for each respective token position of a set of token positions from the plurality of token positions in the output token sequence, determine whether the entropy score associated with the respective token position satisfies a current adaptive entropy threshold, wherein the current adaptive entropy threshold is based on the entropy score associated with each token position of the plurality of token positions;
based on the determination for each respective token position of the set of token positions, perform a decoding operation comprising:
a set block decoding operation to generate at least two output tokens that correspond to at least a first token position and a second token position of the plurality of token positions in the output token sequence; or
a single-token decoding operation to generate a single output token that corresponds to the first token position; and
generate a first version of the output token sequence including the at least two output tokens or the single output token.