Context data reduction in language model processing

1 . A computer-implemented method comprising:

receiving input data comprising first data representing a natural language query and context data to be used to determine a response to the natural language query, the context data comprising a plurality of portions including a first portion and a second portion, wherein the first portion comprises:

a plurality of initial tokens,

a plurality of intermediate tokens following the plurality of initial tokens, and

a plurality of final tokens following the plurality of intermediate tokens;

processing a first token of the plurality of intermediate tokens to determine first position data corresponding to at least one rotary position embedding representing a position of the first token with respect to the context data;

processing a second token of the plurality of intermediate tokens to determine second position data corresponding to at least one rotary position embedding representing a position of the second token with respect to the context data;

processing the first token, the first position data, and the first data, using a large language model (LLM) to determine a first plurality of attention scores and first model output data;

summing the first plurality of attention scores to determine a first context score representing a relevance of the first token to the natural language query;

processing the second token, the second position data, and the first data, using the LLM to determine a second plurality of attention scores and second model output data;

summing the second plurality of attention scores to determine a second context score representing a relevance of the second token to the natural language query;

based at least in part on the first context score, storing the first model output data in a cache associated with the LLM;

based at least in part on the second context score, declining to store the second model output data in the cache; and

processing contents of the cache using the LLM to determine language model output data corresponding to a response to the natural language query.

2 . The computer-implemented method of claim 1 , wherein determination of the first plurality of attention scores and first model output data comprises:

processing the first token, the first position data, and the first data using a first layer of the LLM to determine a first subset of attention scores and first layer output data; and

processing the first layer output data using a further portion of the LLM to determine a second subset of attention scores,

wherein the first plurality of attention scores comprises the first subset and the second subset.

3 . The computer-implemented method of claim 1 , further comprising:

processing the plurality of initial tokens to determine third position data corresponding to at least one rotary position embedding representing at least one position of the plurality of initial tokens with respect to the context data;

processing the third position data, the plurality of initial tokens, and the first data using the LLM to determine third model output data; and

storing the third model output data in the cache.

4 . The computer-implemented method of claim 3 , further comprising:

processing the plurality of final tokens to determine fourth position data corresponding to at least one rotary position embedding representing at least one position of the plurality of final tokens with respect to the context data;

processing the fourth position data, the plurality of final tokens, and the first data using the LLM to determine fourth model output data; and

storing the fourth model output data in the cache.

5 . A computer-implemented method comprising:

receiving first data representing a natural language input;

determining second data representing context data to be used to process the natural language input, wherein an amount of the context data exceeds a size of a storage of a language model;

determining first position data representing a first portion of the second data, wherein the first portion does not exceed the size of the storage;

processing the first position data, the first data, and the first portion of the second data using the language model to determine a plurality of attention scores and first model processing data;

determining, based at least in part on the plurality of attention scores, a first score representing a relevance of the first portion of the second data to the natural language input;

determining the first score fails to satisfy a condition;

based at least in part on the first score failing to satisfy the condition, causing the first model processing data to not be stored in the storage;

determining a remainder of the context data has been processed with regard to relevance to the natural language input; and

processing contents of the storage using the language model to determine language model output data corresponding to a response to the natural language input.

6 . The computer-implemented method of claim 5 , wherein determining the first score comprises:

summing the plurality of attention scores to determine a summation value; and

determining an average attention score using the summation value,

wherein the first score comprises the average attention score.

7 . The computer-implemented method of claim 5 , further comprising:

processing the first data, the first position data, and the first portion of the second data using a first layer of the language model to determine a first attention score, and first layer output data; and

processing the first layer output data using a second layer of the language model to determine a second attention score and a second hidden representation,

wherein the plurality of attention scores comprises the first attention score and the second attention score.

8 . The computer-implemented method of claim 7 , further comprising, prior to determining the remainder of the context data has been processed:

determining second position data representing a second portion of the second data;

processing the first data, the second position data, and the second portion of the second data using the first layer of a language model to determine a third attention score and third layer output data;

processing the third layer output data using the second layer of the language model to determine a fourth attention score and fourth layer output data;

processing the fourth layer output data using a further portion of the language model to determine a second plurality of attention scores and second model processing data corresponding to the second portion of the second data;

determining a second context score using the third attention score, the fourth attention score, and the second plurality of attention scores;

determining the second context score satisfies the condition; and

based at least in part on the second context score satisfying the condition, causing the second model processing data to be stored in the storage.

9 . The computer-implemented method of claim 8 , further comprising:

storing, by the storage, third model processing data corresponding to a third portion of the second data;

determining a third context score representing a relevance of the third portion of the second data to the natural language input;

wherein determining the second context score satisfies the condition comprises determining the second context score is greater than the third context score; and

deleting the third model processing data from the storage.

10 . The computer-implemented method of claim 8 , wherein:

the second portion comprises:

a plurality of initial tokens,

a plurality of intermediate tokens following the plurality of initial tokens, and

a plurality of final tokens following the plurality of intermediate tokens;

the second model processing data corresponds to a subset of the plurality of intermediate tokens; and

the method further comprises:

determining third model processing data corresponding to the plurality of initial tokens;

determining fourth model processing data corresponding to the plurality of final tokens; and

causing the third model processing data and the fourth model processing data to be used to be stored in the storage.

11 . The computer-implemented method of claim 5 , wherein the first model processing data comprises key-value data.

12 . The computer-implemented method of claim 5 , wherein the first position data comprises rotary position embedding data based at least on a position of the first portion with respect to the context data.

13 . A system, comprising:

at least one processor; and

at least one memory comprising instructions that, when executed by the at least one processor, cause the system to:

receive first data representing a natural language input;

determine second data representing context data to be used to process the natural language input, wherein an amount of the context data exceeds a size of a storage of a language model;

determine first position data representing a first portion of the second data, wherein the first portion does not exceed the size of the storage;

process the first position data, the first data, and the first portion of the second data using the language model to determine a plurality of attention scores and first model processing data;

determine, based at least in part on the plurality of attention scores, a first score representing a relevance of the first portion of the second data to the natural language input;

determine the first score fails to satisfy a condition;

based at least in part on the first score failing to satisfy the condition, cause the first model processing data to not be stored in the storage;

determine a remainder of the context data has been processed with regard to relevance to the natural language input; and

process contents of the storage using the language model to determine language model output data corresponding to a response to the natural language input.

14 . The system of claim 13 , wherein the instructions that cause the system to determine the first score comprise instructions that, when executed by the at least one processor, cause the system to:

sum the plurality of attention scores to determine a summation value; and

determine an average attention score using the summation value,

wherein the first score comprises the average attention score.

15 . The system of claim 13 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

process the first data, the first position data, and the first portion of the second data using a first layer of the language model to determine a first attention score, and first layer output data; and

process the first layer output data using a second layer of the language model to determine a second attention score and a second hidden representation,

wherein the plurality of attention scores comprises the first attention score and the second attention score.

16 . The system of claim 15 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to, prior to determination that the remainder of the context data has been processed:

determine second position data representing a second portion of the second data;

process the first data, the second position data, and the second portion of the second data using the first layer of a language model to determine a third attention score and third layer output data;

process the third layer output data using the second layer of the language model to determine a fourth attention score and fourth layer output data;

process the fourth layer output data using a further portion of the language model to determine a second plurality of attention scores and second model processing data corresponding to the second portion of the second data;

determine a second context score using the third attention score, the fourth attention score, and the second plurality of attention scores;

determine the second context score satisfies the condition; and

based at least in part on the second context score failing to satisfy the condition, cause the second model processing data to be stored in the storage.

17 . The system of claim 16 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

store, by the storage, third model processing data corresponding to a third portion of the second data;

determine a third context score representing a relevance of the third portion of the second data to the natural language input;

wherein the instructions that cause the system to determine that the second context score satisfies the condition comprise instructions that, when executed by the at least one processor, cause the system to determine the second context score is greater than the third context score; and

delete the third model processing data from the storage.

18 . The system of claim 16 , wherein:

the second portion comprises:

a plurality of initial tokens,

a plurality of intermediate tokens following the plurality of initial tokens, and

a plurality of final tokens following the plurality of intermediate tokens;

the second model processing data corresponds to a subset of the plurality of intermediate tokens; and

the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

determine third model processing data corresponding to the plurality of initial tokens;

determine fourth model processing data corresponding to the plurality of final tokens; and

cause the third model processing data and the fourth model processing data to be stored in the storage.

19 . The system of claim 13 , wherein the first model processing data comprises key-value data.

20 . The system of claim 13 , wherein the first position data comprises rotary position embedding data based at least on a position of the first portion with respect to the context data.