Method and system for multi-level artificial intelligence supercomputer design featuring sequencing of large language models
A system and method for creating a merged large language model (h-LLM) using a bagging approach including receiving input data at a computer system, creating a plurality of data subsets from the input data, training a plurality of h-LLMs, each h-LLM of the plurality of h-LLMs being trained on a respective data subset of the plurality of data subsets, creating a merged h-LLM by merging the plurality of h-LLMs, and outputting the merged h-LLM.
1 . A method for creating a merged family of large language models (h-LLM) using a bagging approach comprising:
receiving input data at a computer system comprising a processor, non-transitory storage medium, and software stored on the non-transitory storage medium;
creating a plurality of data subsets from the input data;
training a plurality of h-LLMs, each h-LLM of the plurality of h-LLMs being trained on a respective data subset of the plurality of data subsets;
creating a merged h-LLM by merging the plurality of h-LLMs; and
outputting the merged h-LLM.
2 . The method of claim 1 wherein creating the plurality of data subsets comprises dividing the input data to create multiple data subsets.
3 . The method of claim 1 wherein merging the plurality of h-LLMs comprises combining model parameters from each h-LLM through a merging or fusing process.
4 . The method of claim 1 wherein the h-LLMs of the plurality of h-LLMs are trained on different computational resources within a distributed computing environment.
5 . The method of claim 1 wherein the h-LLMs of the plurality of h-LLMs are trained concurrently.
6 . The method of claim 1 wherein the merged h-LLM has at least one of a higher precision, a higher accuracy, and an improved stability than each h-LLM of the plurality of h-LLMs.
7 . A method for creating an enhanced family of large language models (h-LLM) using a boosting approach comprising:
training a first h-LLM using original input data;
testing the first h-LLM to generate first output results;
generating a first weighted data by assigning one or more weights to the first output results;
training a second h-LLM using the first weighted data;
generating a sequence of h-LLMs with increasing levels of at least one of precision and accuracy by iteratively testing the second h-LLM and subsequent h-LLMs, generating subsequent output results, generating subsequent weighted data from the subsequent output results, and training subsequent h-LLMs on the subsequent weighted data;
merging the sequence of h-LLMs to create a merged enhanced h-LLM; and
outputting the merged enhanced h-LLM for use in processing language tasks.
8 . The method of claim 7 wherein the sequence of h-LLMs comprises at least three h-LLMs, each subsequent h-LLM in the sequence having higher accuracy than an immediately preceding h-LLM in the sequence.
9 . A method for creating a specialized family of large language models (h-LLM) through extraction comprising:
receiving a general purpose h-LLM;
identifying a specialized task from a group of tasks;
extracting task-specific knowledge from the general purpose h-LLM corresponding to the specialized task;
creating the specialized h-LLM having reduced computational requirements compared to the general purpose h-LLM while maintaining performance for the specialized task; and
configuring the specialized h-LLM to process prompts related to the specialized task.
10 . The method of claim 9 wherein extracting task-specific knowledge comprises identifying and utilizing one or more neural network components relevant to the specialized task.
11 . The method of claim 9 wherein the specialized h-LLM comprises fewer parameters than the general purpose h-LLM while achieving comparable accuracy for the specialized task.
12 . The method of claim 9 wherein creating the specialized h-LLM comprises using the general purpose h-LLM to guide the creation process.
13 . The method of claim 9 wherein the general purpose h-LLM is trained on broad domain data.
14 . The method of claim 9 wherein the group of tasks consists of sentiment analysis, question answering, information extraction, image captioning, object recognition, instruction following, classification, inferencing, and sentence similarity.
15 . A system for creating specialized families of large language models (h-LLMs) comprising:
a processor;
a non-transitory computer-readable storage medium positioned in communication with the processor; and
software stored on the storage medium that, when executed by the processor, is operable to:
receive a general purpose h-LLM;
identify a specialized task;
extract task-specific knowledge from the general purpose h-LLM;
generate a specialized h-LLM with reduced computational requirements while maintaining task-specific performance related to the specialized task; and
deploy the specialized h-LLM for processing prompts related to the specialized task.