Method and system for multi-level artificial intelligence supercomputer design
A system for processing LLM operations including a memory module, a data receiver to receive an input data stream and break the input data stream into batches of input data, a processing layer to process the batches of input data in the one or more memory modules and having LLMs to process the batches of input data and generate corresponding batches of processed output data and an output broker to cache results from the plurality of LLMs in a results cache, assign priority scores to the cached results, rank the cached results based on their priority scores, and provide ranked results responsive to subsequent similar input data received by the data receiver from the results cache.
1 . A system for processing large language model (LLM) operations, comprising:
one or more memory modules;
a data receiver configured to:
receive an input data stream; and
break the input data stream into batches of input data;
a processing layer comprising:
one or more in-memory processing components positioned in communication with the one or more memory modules and configured to process the batches of input data in the one or more memory modules;
a plurality of LLMs configured to process the batches of input data and generate corresponding batches of processed output data; and
an output broker configured to:
cache results from the plurality of LLMs in a results cache;
assign priority scores to the cached results;
rank the cached results based on their priority scores; and
provide ranked results responsive to subsequent similar input data received by the data receiver from the results cache.
2 . The system of claim 1 wherein the one or more memory modules comprises at least one of random-access memory (RAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), magnetoresistive random-access memory (MRAM), or non-volatile random-access memory (NVRAM).
3 . The system of claim 1 wherein the input data stream is received via an application programming interface (API).
4 . The system of claim 3 wherein the data receiver is implemented in a cloud container environment and is accessible via the API.
5 . The system of claim 1 wherein the output broker is implemented in a cloud container environment and is accessible via an API.
6 . The system of claim 1 wherein the one or more in-memory processing components are configured to process the batches of input data entirely in the one or more memory modules.
7 . A method for in-memory processing of large language model (LLM) operations, comprising:
receiving an input data stream at a data receiver;
determining an aggregation duration for collecting data from the input data stream;
aggregating input data from the input data stream over the determined aggregation duration;
breaking the aggregated input data into data batches sized for in-memory processing;
processing the data batches in one or more memory modules using multiple LLMs to generate processed output data;
caching the processed output data as cached results in a results cache by an output broker;
assigning priority scores by the output broker to the cached processed output data based at least partially on a:
computational cost of generating the output data;
providing the cached processed output data to subsequently received similar input data based on the assigned priority scores.
8 . The method of claim 7 wherein the data batches are processed entirely in the one or more memory modules.
9 . The method of claim 7 wherein processing the data batches in one or more memory modules comprises processing the data batches in one or more modules of random-access memory (RAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), magnetoresistive random-access memory (MRAM), or non-volatile random-access memory (NVRAM).
10 . The method of claim 7 wherein receiving the input data stream at the data receiver comprises receiving the input data stream at the data receiver via an application programming interface.
11 . The method of claim 10 further comprising implementing the data receiving in a cloud container environment.
12 . The method of claim 7 further comprising implementing the output broker in a cloud container environment.
13 . A system for processing large language model (LLM) operations, comprising:
one or more memory modules;
a processor coupled to the one or more memory modules; and
a non-transitory computer-readable storage medium coupled to the processor and having software stored thereon that is executable by the processor to:
operate a data receiver configured to:
receive an input data stream; and
break the input data stream into batches of input data;
operate a processing layer comprising:
one or more in-memory processing components positioned in communication with the one or more memory modules and configured to process the batches of input data in the one or more memory modules;
a plurality of LLMs configured to process the batches of input data and generate corresponding batches of processed output data; and
an output broker configured to:
cache results from the plurality of LLMs in a results cache;
assign priority scores to the cached results;
rank the cached results based on their priority scores; and
provide ranked results responsive to subsequent similar input data received by the data receiver from the results cache; and
a communication device operable to send and receive data to and from the processor across a network.
14 . The system of claim 13 wherein the one or more memory modules comprises at least one of random-access memory (RAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), magnetoresistive random-access memory (MRAM), or non-volatile random-access memory (NVRAM).
15 . The system of claim 13 wherein the software is further configured to, when executed, implement an application programming interface (API) to receive the input data stream.
16 . The system of claim 15 wherein the software is further configured to, when executed, implement the data receiver in a cloud container environment and to be accessible via the API.
17 . The system of claim 13 wherein the output broker is implemented in a cloud container environment and is accessible via an API.