Granted Patent B2

US 12,681,830 · App. 18/812,913 · Granted Jul 14, 2026

Inventors: Sourabh Deb (Tampa, FL); Jason Engelbrecht (London, GB); Zheyu Wang (Shanghai, CN); Haolin Jin (Shanghai, CN); Payal Jain (London, GB); Tariq Husayn Maonah (London, GB); Mariusz Saternus (Cracow, PL); Daniel Lewandowski (Cracow, PL); Biraj Krushna Rath (London, GB); Stuart Murray (London, GB); Philip Davies (London, GB)

G06F11/3419G06F9/5055

Abstract

Claims (79)

1 . A non-transitory computer-readable storage medium comprising instructions stored thereon, wherein the instructions when executed by at least one data processor of a system, cause the system to:

receive, from a computing device, a set of output generation requests, each comprising a prompt for generation of one or more responses by executing one or more artificial intelligence (AI) models on one or more hardware resources of a set of available hardware resources;

for each output generation request of the set of output generation requests: using the prompt of the output generation request, generate a set of output attributes of the output generation request, wherein the generated set of output attributes of the output generation request indicate: (1) a type of the output generated from the prompt and (2) a threshold response time of the generation of the output; using the set of output attributes, map the output generation request to a set of requested hardware resources by: identifying one or more dependencies associated with processing the output generation request using the one or more AI models, and using the identified dependencies, determining an estimated hardware resource usage associated with processing the output generation request using the one or more AI models;

dynamically partition the set of available hardware resources to determine, for the each output generation request, a set of selected hardware resources within the set of available hardware resources using: (1) a compatibility between one or more hardware resources of the set of available hardware resources and the set of requested hardware resources and (2) one or more sets of requested hardware resources of other output generation requests in the set of output generation requests;

provide the prompt of each output generation request to a corresponding set of selected hardware resources to generate a set of outputs by processing the prompt included in the output generation request using the one or more AI models; and

responsive to the generated set of outputs, transmit, to the computing device, the output within the threshold response time.

2 . The non-transitory computer-readable storage medium of claim 1 , wherein the instructions further cause the system to:

obtain a set of operation boundaries of the one or more AI models; and

map the output generation request to the set of requested hardware resources in accordance with the set of operation boundaries.

3 . The non-transitory computer-readable storage medium of claim 1 ,

wherein each output generation request includes a predefined query context corresponding to a user of the computing device, and

wherein the predefined query context includes a vector representation of one or more output attributes.

4 . The non-transitory computer-readable storage medium of claim 1 , wherein the instructions further cause the system to:

generate for display at the computing device, a layout indicating the set of outputs, wherein the layout includes one or more of:

a first representation of each AI model,

a second representation of the corresponding set of selected hardware resources, or

a third representation of the set of outputs.

5 . The non-transitory computer-readable storage medium of claim 1 , wherein the set of available hardware resources includes (1) a set of shared hardware and (2) a set of dedicated hardware, wherein generating the set of output further causes the system to:

using the set of shared hardware, process the prompt included in the output generation request using the one or more AI models for a predetermined time period; and

upon expiration of the predetermined time period, continue processing the prompt using the one or more AI models on the set of dedicated hardware.

6 . The non-transitory computer-readable storage medium of claim 1 , wherein the instructions further cause the system to:

receive a set of user feedback associated with the generated set of outputs, wherein the set of user feedback indicates an expected set of outputs; and

using the received user feedback, adjust the dynamic partitioning of the set of available hardware resources to satisfy the expected set of outputs.

7 . The non-transitory computer-readable storage medium of claim 1 , wherein determining the estimated hardware resource usage indicates usage of one or more of:

central processing unit (CPU) resources,

graphical processing unit (GPU) resources, or

memory resources.

8 . A method for dynamically selecting hardware resources based on resource usage, the method comprising:

obtaining, from a computing device, a set of output generation requests, each comprising an input for generation of one or more results by executing one or more artificial intelligence (AI) models on one or more hardware resources of a set of available hardware resources;

for each output generation request of the set of output generation requests, associating the output generation request to a set of requested hardware resources by determining an estimated hardware resource usage associated with processing the output generation request using the one or more AI models;

dynamically partitioning the set of available hardware resources to determine, for the each output generation request, a set of selected hardware resources within the set of available hardware resources using one or more of: (1) a compatibility between one or more hardware resources of the set of available hardware resources and the set of requested hardware resources, or (2) one or more sets of requested hardware resources of other output generation requests in the set of output generation requests; and

providing the input of each output generation request to a corresponding set of selected hardware resources to generate a set of outputs by processing the input included in the output generation request using the one or more AI models.

9 . The method of claim 8 , further comprising:

for each output generation request, assigning a set of weights to each hardware resource of the set of available hardware resources.

10 . The method of claim 8 ,

wherein the one or more AI models is a first set of AI models,

wherein dynamically partitioning the set of available hardware resources uses a second set of AI models, and

wherein the first set of AI models is different from the second set of AI models.

11 . The method of claim 8 , further comprising, for one or more hardware resources of the set of available hardware resources:

determining a current resource usage value for the available hardware resource;

determining a maximum usage value for the available hardware resource; and

calculating an allowance value corresponding to the available hardware resource for the set of output generation requests using a difference between a corresponding maximum usage value and a corresponding current resource usage value.

12 . The method of claim 8 , further comprising:

partitioning one or more output generation requests of the set of output generation requests into a set of segments; and

routing each segment of the set of segments to the set of selected hardware resources,

wherein at least one segment is routed to a different selected hardware resource than another segment in the set of segments.

13 . The method of claim 8 , further comprising:

generating for display at the computing device, a layout indicating the set of outputs, wherein the layout includes one or more of:

a first representation of each AI model,

a second representation of the corresponding set of selected hardware resources, or

a third representation of the set of outputs.

14 . The method of claim 8 , further comprising:

receiving a set of user feedback on the generated set of outputs; and

using the received user feedback, adjusting the dynamic partitioning of the set of available hardware resources.

15 . A system comprising: at least one hardware processor; and at least one non-transitory memory storing instructions, which, when executed by the at least one hardware processor, cause the system to:

obtain, from a computing device, a set of output generation requests, each comprising an input for generation of one or more results by executing one or more software applications on one or more hardware resources of a set of available hardware resources;

for one or more output generation requests of the set of output generation requests, associating the output generation request to a set of requested hardware resources using an estimated hardware resource usage associated with processing the output generation request by the one or more software applications;

dynamically partition the set of available hardware resources to determine, for the each output generation request of the one or more output generation requests, a set of selected hardware resources within the set of available hardware resources using one or more of: (1) a compatibility between one or more hardware resources of the set of available hardware resources and the set of requested hardware resources, or (2) one or more sets of requested hardware resources of other output generation requests in the set of output generation requests; and

provide the input of the one or more output generation requests to a corresponding set of selected hardware resources to generate a set of outputs by processing the input included in the output generation request using the one or more software applications.

16 . The system of claim 15 , wherein the system is further caused to:

wherein the one or more software applications is a first set of software applications,

wherein dynamically partitioning the set of available hardware resources uses a second set of software applications, and

wherein the first set of software applications is different from the second set of software applications.

17 . The system of claim 15 , wherein the system is further caused to, for one or more hardware resources of the set of available hardware resources:

determine a current resource usage value for the available hardware resource;

determine a maximum usage value for the available hardware resource; and

calculate an allowance value corresponding to the available hardware resource for the set of output generation requests using a difference between a corresponding maximum usage value and a corresponding current resource usage value.

18 . The system of claim 15 , wherein the system is further caused to:

partition one or more output generation requests of the set of output generation requests into a set of segments; and

route each segment of the set of segments to the set of selected hardware resources,

wherein at least one segment is routed to a different selected hardware resource than another segment in the set of segments.

19 . The system of claim 15 , wherein the system is further caused to:

generate for display at the computing device, a layout indicating the set of outputs, wherein the layout includes one or more of:

a first representation of each software application,

a second representation of the corresponding set of selected hardware resources, or

a third representation of the set of outputs.

20 . The system of claim 15 , wherein the system is further caused to:

receive a set of user feedback on the generated set of outputs; and

using the received user feedback, adjust the dynamic partitioning of the set of available hardware resources.

Assignments (2)

Continuity (4)

References Cited (305)