Caching policy to cache key-value vectors

1 . A system, comprising:

a memory that stores computer-executable components; and

a processor that executes at least one of the computer-executable components that:

accesses a request comprising a fixed prefix and variable data;

computes a temporal moving average as an average of a number of times the fixed prefix is accessed over a defined time period; and

selectively stores in a cache, one or more key-value vectors generated via processing of the fixed prefix by a large language model (LLM), based on the temporal moving average of the fixed prefix being greater than a first defined threshold, according to a smart encoding policy, wherein the selective storing comprises:

based on the temporal moving average being greater than the first defined threshold, predicting, using a machine learning model that processes the fixed prefix, a first number of tokens generated via processing of the request by the LLM,

determining a ratio of the first number of tokens to a token length of the fixed prefix, and

storing the one or more key-value vectors in the cache based on the ratio being greater than a second defined threshold.

2 . The system of claim 1 , wherein the at least one of the computer-executable components further:

trains the machine learning model to predict the first number of tokens, wherein the training comprises:

accessing pairs of data points collected for a previous request comprising a previous fixed prefix and previous variable data, wherein each pair of data points of the pairs of data points includes a first data point representing a token length of the previous fixed prefix and a second data point representing a second number of tokens generated via processing of the previous request by the LLM; and

training the machine learning model with the previous fixed prefix as input and the second number of tokens as an output, wherein the machine learning model is a regression model built on a sequence embedding model.

3 . The system of claim 1 , wherein the at least one of the computer-executable components further:

queries the fixed prefix against the cache to extract the one or more key-value vectors and that queries the fixed prefix against the LLM to generate the one or more key-value vectors if the one or more key-value vectors are missing from the cache.

4 . The system of claim 1 , wherein the at least one of the computer-executable components further:

inputs the one or more key-value vectors and the variable data into the LLM, wherein the LLM generates a response by processing the one or more key-value vectors and the variable data.

5 . The system of claim 1 , wherein storing the one or more key-value vectors reduces an amount of computation performed by an LLM-based pipeline employing the cache for subsequent requests comprising the fixed prefix and increases an inferencing speed of the LLM-based pipeline in proportion to a ratio of a token length of the fixed prefix to a token length of an output generated by the LLM-based pipeline.

6 . The system of claim 1 , wherein the at least one of the computer-executable components further:

configures the smart encoding policy according to a use case.

7 . The system of claim 1 , wherein storing the one or more key-value vectors in the cache employs positional encoding with the smart encoding policy.

8 . The system of claim 1 , wherein an application programming interface (API) explicitly defines a prefix field in the request to input the fixed prefix.

9 . A computer-implemented method, comprising:

accessing, by a system operatively coupled to a processor, a request comprising a fixed prefix and variable data;

computing, by the system, a temporal moving average as an average of a number of times the fixed prefix is accessed over a defined time period; and

selectively storing, by the system, in a cache, one or more key-value vectors generated via processing of the fixed prefix by an LLM, based on a temporal moving average of the fixed prefix being greater than a first defined threshold, according to a smart encoding policy, wherein the selective storing comprises:

accessing, in response to the temporal moving average being greater than the first defined threshold, a machine learning model that processes the fixed prefix to predict a first number of tokens generated via processing of the request by the LLM,

computing a ratio of the first number of tokens to a token length of the fixed prefix, and

storing the one or more key-value vectors in the cache in response to the ratio being greater than a second defined threshold.

10 . The computer-implemented method of claim 9 , further comprising:

training, by the system, the machine learning model to predict the first number of tokens, wherein the training comprises:

accessing, by the system, pairs of data points collected for a previous request comprising a previous fixed prefix and previous variable data, wherein each pair of data points of the pairs of data points includes a first data point representing a token length of the previous fixed prefix and a second data point representing a second number of tokens generated via processing of the previous request by the LLM; and

training, by the system, the machine learning model with the previous fixed prefix as input and the second number of tokens as an output, wherein the machine learning model is a regression model built on a sequence embedding model.

11 . The computer-implemented method of claim 9 , further comprising:

querying, by the system, the fixed prefix against the cache to extract the one or more key-value vectors and that queries the fixed prefix against the LLM to generate the one or more key-value vectors if the one or more key-value vectors are missing from the cache.

12 . The computer-implemented method of claim 9 , further comprising:

inputting, by the system, the one or more key-value vectors and the variable data into the LLM, wherein the LLM generates a response by processing the one or more key-value vectors and the variable data.

13 . The computer-implemented method of claim 9 , wherein the storing the one or more key-value vectors reduces an amount of computation performed by an LLM-based pipeline employing the cache for subsequent requests comprising the fixed prefix and increases an inferencing speed of the LLM-based pipeline in proportion to a ratio of a token length of the fixed prefix to a token length of an output generated by the LLM-based pipeline.

14 . The computer-implemented method of claim 9 , further comprising:

configuring, by the system, the smart encoding policy according to a use case.

15 . The computer-implemented method of claim 9 , further comprising:

employing, by the system, positional encoding with the smart encoding policy to store the one or more key-value vectors in the cache.

16 . The computer-implemented method of claim 9 , wherein an API explicitly defines a prefix field in the request to input the fixed prefix.

17 . A computer program product for speeding up an LLM service with inference caching, the computer program product comprising a non-transitory computer readable medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:

access a request comprising a fixed prefix and variable data; and

compute a temporal moving average as an average of a number of times the fixed prefix is accessed over a defined time period; and

store, in a cache, one or more key-value vectors generated via processing of the fixed prefix by an LLM, based on a temporal moving average of the fixed prefix being greater than a first defined threshold, according to a smart encoding policy, wherein the selective storing comprises:

accessing, in response to the temporal moving average being greater than the first defined threshold, a machine learning model that processes the fixed prefix to predict a first number of tokens generated via processing of the request by the LLM,

computing a ratio of the first number of tokens to a token length of the fixed prefix, and

storing the one or more key-value vectors in the cache in response to the ratio being greater than a second defined threshold.

18 . The computer program product of claim 17 , wherein the program instructions are further executable by the processor to cause the processor to:

train the machine learning model to predict the first number of tokens, wherein the training comprises:

accessing pairs of data points collected for a previous request comprising a previous fixed prefix and previous variable data, wherein each pair of data points of the pairs of data points includes a first data point representing a token length of the previous fixed prefix and a second data point representing a second number of tokens generated via processing of the previous request by the LLM; and

training the machine learning model with the previous fixed prefix as input and the second number of tokens as an output, wherein the machine learning model is a regression model built on a sequence embedding model.

19 . The computer program product of claim 17 , wherein the program instructions are further executable by the processor to cause the processor to:

query the fixed prefix against the cache to extract the one or more key-value vectors and that queries the fixed prefix against the LLM to generate the one or more key-value vectors if the one or more key-value vectors are missing from the cache.

20 . The computer program product of claim 17 , wherein the program instructions are further executable by the processor to cause the processor to:

input the one or more key-value vectors and the variable data into the LLM, wherein the LLM generates a response by processing the one or more key-value vectors and the variable data.