Generative artificial intelligence response caching using prompt processing units

1 . A method, comprising:

providing, by a device, a prompt to a large language model to perform a task;

determining, by the device and using a generative artificial intelligence model trained to identify whether one or more tasks in received prompts can be solved using a quasi-idempotent function without executing the one or more tasks, a likelihood of an outcome of the task changing over a period of time based on whether the task qualifies as solvable through the quasi-idempotent function without requiring execution of the task;

storing, by the device, the prompt and the outcome in a cache when the likelihood indicates that the outcome of the task is unlikely to change over the period of time; and

returning, by the device, the outcome from the cache in response to a subsequent prompt asking the large language model to perform the task, in lieu of providing the subsequent prompt to the large language model.

2 . The method of claim 1 , further comprising:

providing the prompt to the large language model without storing the prompt and the outcome in the cache when the likelihood indicates that the outcome of the task is likely to change over the period of time.

3 . The method of claim 1 , wherein the outcome is stored in the cache responsive to determining, by a cache lookup operation, that the outcome is not already stored in the cache.

4 . The method of claim 1 , further comprising:

determining whether the subsequent prompt is asking the large language model to perform the task by performing a cache lookup operation for a cacheable task included in the subsequent prompt.

5 . The method of claim 1 , wherein the period of time corresponds to a phase out time for entries in the cache.

6 . The method of claim 1 , further comprising:

parsing a characterization of the prompt to generate a breakdown of tasks included in the prompt.

7 . The method of claim 6 , further comprising:

tokenizing each of the tasks in the breakdown in a nominal task format; and

OR-ing tokenized tasks from the breakdown.

8 . The method of claim 1 , further comprising:

indexing cache entries utilizing task outcomes as entry identifiers.

9 . The method of claim 1 , further comprising:

parsing the prompt to generate a prompt characterization, wherein the prompt characterization includes one or more of tasks requested in the prompt, sensitive data entailed in completing the tasks, a constraint applicable to completing the tasks, or a targeted output upon completion of the tasks.

10 . An apparatus, comprising:

one or more network interfaces to communicate with a network;

a processor coupled to the one or more network interfaces and configured to execute one or more processes; and

a memory configured to store a process that is executable by the processor, the process, when executed, configured to:

provide a prompt to a large language model to perform a task;

determine, using a generative artificial intelligence model trained to identify whether one or more tasks in received prompts can be solved using a quasi-idempotent function without executing the one or more tasks, a likelihood of an outcome of the task changing over a period of time based on whether the task qualifies as solvable through the quasi-idempotent function without requiring execution of the task;

store the prompt and the outcome in a cache when the likelihood indicates that the outcome of the task is unlikely to change over the period of time; and

return the outcome from the cache in response to a subsequent prompt asking the large language model to perform the task, in lieu of providing the subsequent prompt to the large language model.

11 . The apparatus as in claim 10 , the process further configured to:

provide the prompt to the large language model without storing the prompt and the outcome in the cache when the likelihood indicates that the outcome of the task is likely to change over the period of time.

12 . The apparatus as in claim 10 , wherein the outcome is stored in the cache responsive to determining, by a cache lookup operation, that the outcome is not already stored in the cache.

13 . The apparatus as in claim 10 , the process further configured to:

determine whether the subsequent prompt is asking the large language model to perform the task by performing a cache lookup operation for a cacheable task included in the subsequent prompt.

14 . The apparatus as in claim 10 , wherein the period of time corresponds to a phase out time for entries in the cache.

15 . The apparatus as in claim 10 , the process further configured to:

parse a characterization of the prompt to generate a breakdown of tasks included in the prompt.

16 . The apparatus as in claim 15 , the process further configured to:

tokenize each of the tasks in the breakdown in a nominal task format; and

OR tokenized tasks from the breakdown.

17 . The apparatus as in claim 10 , the process further configured to:

index cache entries utilizing task outcomes as entry identifiers.

18 . A tangible, non-transitory, computer-readable medium having computer-executable instructions stored thereon that, when executed by a processor on a computer, cause the computer to perform a method comprising:

providing a prompt to a large language model to perform a task;

determining, using a generative artificial intelligence model trained to identify whether one or more tasks in received prompts can be solved using a quasi-idempotent function without executing the one or more tasks, a likelihood of an outcome of the task changing over a period of time based on whether the task qualifies as solvable through the quasi-idempotent function without requiring execution of the task;

storing the prompt and the outcome in a cache when the likelihood indicates that the outcome of the task is unlikely to change over the period of time; and

returning the outcome from the cache in response to a subsequent prompt asking the large language model to perform the task, in lieu of providing the subsequent prompt to the large language model.

19 . The method as in claim 1 , wherein:

determining the likelihood of the outcome of the task changing over the period of time comprises parsing a characterization of the prompt to generate a breakdown of tasks included in the prompt;

storing the prompt and the outcome in the cache comprises indexing the prompt and the outcome to enable retrieval for subsequent tasks that are semantically the same but expressed differently or listed in a different order by:

tokenizing the tasks in the breakdown in a nominal task format;

generating task outcomes by OR-ing tokenized tasks from the breakdown; and

indexing cache entries utilizing the task outcomes as entry identifiers.

20 . The method as in claim 1 , wherein determining the likelihood of the outcome of the task changing over the period of time comprises:

parsing a characterization of the prompt to generate a breakdown of tasks included in the prompt; and

evaluating, without executing the tasks, whether each task in the breakdown qualifies as solvable through the quasi-idempotent function.