System and method for cost and carbon aware large language model cache management

1 . A computer-implemented method for cost and carbon aware large language model (LLM) cache management, the computer-implemented method comprising:

receiving, by at least one processor, an input prompt from a user device;

determining, by the at least one processor, whether a prompt matching the input prompt exists in a plurality of prompts stored in a cache memory; and

upon determining that the prompt matching the input prompt exists in the cache memory,

returning, by the at least one processor, a response associated with the prompt from the cache memory;

upon determining that the prompt matching the input prompt is not cached in the cache memory,

invoking, by the at least one processor, the LLM to obtain a response to the input prompt and returning the response generated by the LLM to the user device;

updating, by the at least one processor, the cache memory with the input prompt and the response obtained from the LLM as an entry in the cache memory;

estimating, by the at least one processor, an operational cost and a carbon cost associated with generating the response to the input prompt,

wherein the operational cost in generating the response is computed based upon time taken for generating the response and a cost per second for an instance of computing resources generating the response;

fetching, by the least one processor, data associated with the LLM; and

updating, by the at least one processor, the operational cost, the carbon cost and the data associated with the LLM as metadata of the entry in the cache memory.

2 . The computer-implemented method of claim 1 , updating the cache memory further comprises swapping out an existing entry in the cache memory upon determining that the cache memory is full.

3 . The computer-implemented method of claim 2 , wherein swapping out an existing entry in the cache memory upon determining that the cache memory is full comprises:

computing a frugal cache score for each entry among a plurality of entries stored in the cache memory, wherein the frugal cache score is computed based at least in part upon the metadata of each entry and based at least in part upon a caching policy;

identifying an entry with a least frugal cache score among the plurality of entries cached in the cache memory; and

swapping the entry with the least frugal cache score value in the cache memory to store a new entry and metadata of the new entry.

4 . The computer-implemented method of claim 1 , wherein determining the prompt matching the input prompt exists in the plurality of prompts stored in the cache comprises one or more of:

comparing texts of the input prompt with texts of one or more of the plurality of prompts to identify the prompt matching the input prompt; and

performing semantic similarity analyses between the input prompt and the one or more of the plurality of prompts to identify the prompt matching the input prompt.

5 . The computer-implemented method of claim 1 , wherein the data associated with the LLM comprises top-level hyperparameter sizes.

6 . The computer-implemented method of claim 1 , further comprising:

determining, by the at least one processor, whether the response associated with the prompt for an entry of cached entries in the cache memory has been rejected by a threshold number of times; and

deleting, by the at least one processor, from the cache memory, the response and associated prompt along with associated metadata of the entry of cached entries.

7 . A system for cost and carbon aware large language model (LLM) cache management, the system comprising:

at least one memory storing instructions;

a cache memory; and

at least one processor communicatively coupled with the at least one memory and the cache memory, wherein the at least one processor is configured to execute the instructions to:

receive an input prompt from a user device;

determine whether a prompt matching the input prompt exists in a plurality of prompts stored in a cache memory; and

upon determining that the prompt matching the input prompt exists in the cache memory,

return a response associated with the prompt from the cache memory;

upon determining that the prompt matching the input prompt is not cached in the cache memory,

invoke the LLM to obtain a response to the input prompt and returning the response generated by the LLM to the user device;

update the cache memory with the input prompt and the response obtained from the LLM as an entry in the cache memory;

estimate an operational cost and a carbon cost associated with generating the response to the input prompt,

wherein the operational cost in generating the response is computed based upon time taken for generating the response and a cost per second for an instance of computing resources generating the response;

fetch data associated with the LLM; and

update the operational cost, the carbon cost and the data associated with the LLM as metadata of the entry in the cache memory.

8 . The system of claim 7 , wherein the at least one processor is further configured to update the cache memory by swapping out an existing entry in the cache memory upon determining that the cache memory is full.

9 . The system of claim 8 , wherein swapping out an existing entry in the cache memory upon determining that the cache memory is full comprises:

computing a frugal cache score for each entry among a plurality of entries stored in the cache memory, wherein the frugal cache score is computed based at least in part upon the metadata of each entry and based at least in part upon a caching policy;

identifying an entry with a least frugal cache score among the plurality of entries cached in the cache memory; and

swapping the entry with the least frugal cache score value in the cache memory to store a new entry and metadata of the new entry.

10 . The system of claim 7 , wherein to determine the prompt matching the input prompt exists in the plurality of prompts stored in the cache, the processor is configured to perform one or more of:

comparing texts of the input prompt with texts of one or more of the plurality of prompts to identify the prompt matching the input prompt; and

performing semantic similarity analyses between the input prompt and the one or more of the plurality of prompts to identify the prompt matching the input prompt.

11 . The system of claim 7 , wherein the data associated with the LLM comprises top-level hyperparameter sizes.

12 . The system of claim 7 , wherein the processor is further configured to:

determine whether the response associated with the prompt for an entry of cached entries in the cache memory has been rejected by a threshold number of times; and

delete, from the cache memory, the response and associated prompt along with associated metadata of the entry of cached entries.

13 . A non-transitory computer-readable media (CRM) comprising instructions stored thereon, which, when executed by at least one processor of a computing device configured for cost and carbon aware large language model (LLM) cache management, cause the computing device to:

receive an input prompt from a user device;

determine whether a prompt matching the input prompt exists in a plurality of prompts stored in a cache memory; and

upon determining that the prompt matching the input prompt exists in the cache memory,

return a response associated with the prompt from the cache memory;

upon determining that the prompt matching the input prompt is not cached in the cache memory,

invoke the LLM to obtain a response to the input prompt and returning the response generated by the LLM to the user device;

update the cache memory with the input prompt and the response obtained from the LLM as an entry in the cache memory;

estimate an operational cost and a carbon cost associated with generating the response to the input prompt,

wherein the operational cost in generating the response is computed based upon time taken for generating the response and a cost per second for an instance of computing resources generating the response;

fetch data associated with the LLM; and

update the operational cost, the carbon cost and the data associated with the LLM as metadata of the entry in the cache memory.