Generative artificial intelligence (AI) system

1 . A computer implemented method, comprising:

receiving, at a generative artificial intelligence (AI) application programming interface (API), from a calling system, a generative AI request;

determining a generative AI model type configured to execute the generative AI request;

determining a subset of generative AI models from a plurality of different generative AI models that correspond to the generative AI model type:

identifying dedicated graphics processing units (GPUs) associated with the subset of generative AI models, individual dedicated GPUs being dedicated to individual generative AI models prior to receiving the generative AI request;

estimating a plurality of estimated generative loads for executing the generative AI request by individual dedicated GPUs, estimating the plurality of estimated generative loads being based at least in part on a hardware characteristic of the dedicated GPUs;

routing the generative AI request to a selected generative AI model, of the subset of generative AI models, based on the plurality of estimated generative loads;

receiving a response from the selected generative AI model; and

returning the response to the calling system.

2 . The computer implemented method of claim 1 wherein the generative AI request includes a prompt with a linguistic request portion and wherein processing the generative AI request further comprises:

processing the linguistic request portion of the prompt to generate a set of tokens based on words in the linguistic request portion; and

identifying a plurality of calls to be made to the generative AI model based on the set of tokens.

3 . The computer implemented method of claim 1 wherein receiving a generative AI request comprises:

receiving a plurality of different generative AI requests from a plurality of different calling systems.

4 . The computer implemented method of claim 3 and further comprising:

assigning a request priority to each of the plurality of generative AI requests; and

routing the plurality of generative AI requests based on the request priority assigned to each of the plurality of generative AI requests.

5 . The computer implemented method of claim 4 wherein assigning a request priority comprises:

evaluating a set of priority criteria to obtain a priority evaluation result for a selected generative AI request, of the plurality of generative AI requests; and

assigning the request priority to the selected generative AI request based on the priority evaluation result.

6 . The computer implemented method of claim 5 wherein assigning a request priority comprises:

identifying an access pattern corresponding to the selected generative AI request;

identifying a request priority queue, of a plurality of different request priority queues, based on the identified access pattern corresponding to the selected generative AI request; and

generating an entry in the identified request priority queue based on the request priority assigned to the selected generative AI request.

7 . The computer implemented method of claim 6 wherein identifying an access pattern comprises:

identifying whether the selected generative AI request is a synchronous request or an asynchronous request.

8 . The computer implemented method of claim 7 wherein identifying the request priority queue comprises:

identifying the request priority queue as a synchronous request priority queue that stores entries corresponding to synchronous generative AI requests when the selected generative AI request is a synchronous request; and

identifying the request priority queue as an asynchronous request priority queue that stores entries corresponding to asynchronous generative AI requests when the selected generative AI request is an asynchronous request.

9 . The computer implemented method of claim 2 wherein processing the generative AI request further comprises:

processing the prompt to identify data to be extracted and routed to the AI model with the generative AI request.

10 . The computer implemented method of claim 9 wherein processing the prompt comprises:

extracting the identified data to obtain extracted data and wherein routing comprises sending the extracted data to the generative AI model.

11 . The computer implemented method of claim 10 wherein extracting the identified data comprises at least one of:

extracting context data or extracting augmented data.

12 . The computer implemented method of claim 1 wherein the generative AI request includes a prompt and further comprising:

capturing prompt data based on the prompt and response data based on the response; and

sending the prompt data and the response data to a prompt/response data store.

13 . The computer implemented method of claim 1 , wherein the estimated generative load comprises estimated computing system resources required to generate a response to the generative AI request.

14 . The computer implemented method of claim 1 , wherein the estimated generative load comprises an estimated amount of time for the dedicated GPUs to generate a response to the generative AI request.

15 . The computer implemented method of claim 1 , wherein the estimated generative load comprises an estimated amount of memory needed to generate a response to the generative AI request.

16 . A computing system, comprising:

a generative artificial intelligence (AI) application programming interface (API) that receives a generative AI request from a calling system;

a prompt processor configured to:

determine a generative AI model type configured to execute the generative AI request;

determine a subset of generative AI models from a plurality of different generative AI models that correspond to the generative AI model type:

identify dedicated graphics processing units (GPUs) associated with the subset of generative AI models, individual dedicated GPUs being dedicated to individual generative AI models prior to receiving the generative AI request; and

estimate a plurality of estimated generative loads for executing the generative AI request by individual dedicated GPUs, estimating the plurality of estimated generative loads being based at least in part on a hardware characteristic of the dedicated GPUs;

a generative AI request routing system configured to route the generative AI request to a selected generative AI model, of the subset of different generative AI models, based on the plurality of estimated generative loads; and

a response processor configured to receive a response from the selected generative AI model and returns the response to the calling system.

17 . The computing system of claim 16 wherein the generative AI API receives a plurality of different generative AI requests from a plurality of different calling and further comprising:

a request priority processor configured to assign a request priority to each of the plurality of generative AI requests and wherein the generative AI request routing system routes the plurality of generative AI requests based on the request priority assigned to each of the plurality of generative AI requests.

18 . The computing system of claim 17 wherein the request priority processor comprises:

an access pattern identifier that identifies an access pattern corresponding to a selected generative AI request.

19 . A computer system, comprising:

at least one processor;

a memory storing computer executable instructions which, when executed by the at least one processor, cause the at least one processor to perform steps comprising:

receiving a generative AI request;

processing the generative AI request, at a generative AI processing layer, by:

determining a generative AI model type configured to execute the generative AI request;

determining a subset of generative AI models from a plurality of different generative AI models that correspond to the generative AI model type;

identifying dedicated graphics processing units (GPUs) associated with the subset of generative AI models, individual dedicated GPUs being dedicated to individual generative AI models prior to receiving the generative AI request;

estimating a plurality of estimated generative loads for executing the generative AI request by individual dedicated GPUs, estimating the plurality of estimated generative loads being based at least in part on a hardware characteristic of the dedicated GPUs;

routing the generative AI request from the generative AI processing layer to the selected generative AI model.

20 . The computer system of claim 19 wherein processing the generative AI request comprises identifying an estimated length of a requested AI generation, and wherein routing comprises:

routing the generative AI request from the generative AI processing layer to the selected generative AI model, of the plurality of different types of generative AI models based on the estimated length of the requested AI generation.