Method and system for multi-level artificial intelligence supercomputer design
A method for generating a merged large language model (LLM) from input data including receiving derived prompts generated from a user prompt and relevant contexts, receiving the relevant contexts related to the generation of the plurality of derived prompts, providing derived prompts and the relevant contexts to one or more merged LLMs, generating a plurality of results, and sending the plurality of results to an output broker.
1. A method for generating a merged large language model (LLM) from input data comprising:
receiving a plurality of derived prompts generated from a user prompt and one or more relevant contexts;
sending the plurality of derived prompts to one or more prompt embedding models;
receiving one or more knowledge documents responsive to sending the plurality of derived prompts to the one or more prompt embedding models;
receiving a plurality of updated derived prompts that were generated based on the plurality of derived prompts and the one or more knowledge documents;
providing the plurality of updated derived prompts to one or more merged LLMs, generating a plurality of results; and
sending the plurality of results to an output broker.
2. The method of claim 1 wherein the one or more merged LLMs comprises a plurality of merged LLMs.
3. The method of claim 1 wherein:
the one or more merged LLMs are the result of merging a first LLM trained on a first subset of input data for a first specialized task and a second LLM trained on a second subset of input data for a second specialized task;
the first subset of the input data has a first data type; and
the second subset of the input data has a second data type.
4. The method of claim 3 wherein:
the first data type is one of text data, image data, audio data, video data, and code data; and
the second data type is one of text data, image data, audio data, video data, and code data and is different from the first data type.
5. The method of claim 3 wherein the one or more merged LLMs has at least one of an improved precision and an improved accuracy relative to at least one of the first LLM and the second LLM.
6. The method of claim 4 wherein the one or more merged LLMs has at least one of an improved precision and an improved accuracy relative to each of the first LLM and the second LLM.
7. The method of claim 3 wherein the first LLM is trained in parallel with the second LLM.
8. The method of claim 3 wherein:
the first LLM is a first family of LLMs (h-LLM);
the second LLM is a second h-LLM; and
the one or more merged LLMs is one or more merged h-LLMs.
9. The method of claim 3 wherein the one or more merged LLMs are produced by a merging process that is configured to produce a merged LLM that is specialized to perform a specific task.
10. The method of claim 1 further comprising filtering the plurality of results for at least one of accuracy or service level assurance.
11. The method of claim 1 wherein:
the one or more merged LLMs is added to a plurality of merged LLMs; and
providing the plurality of updated derived prompts to the one or more merged LLMs comprises sending the plurality of updated derived prompts to the plurality of merged LLMs, generating the plurality of results.
12. A method for generating a merged large language model (LLM) from input data comprising:
training a first LLM on a first subset of the input data for a first specialized task;
training a second LLM on a second subset of the input data for a second specialized task;
performing a merging process on the first LLM and the second LLM to produce one or more merged LLMs, the one or more merged LLMs being operable to:
receive a plurality of updated derived prompts generated from a plurality of derived prompts and one or more relevant contexts; and
generate a plurality of results based on the plurality of updated derived prompts.
13. The method of claim 12 wherein the one or more merged LLMs comprises a plurality of merged LLMs.
14. The method of claim 12 wherein:
the first subset of the input data has a first data type; and
the second subset of the input data has a second data type.
15. The method of claim 14 wherein:
the first data type is one of text data, image data, audio data, video data, and code data; and
the second data type is one of text data, image data, audio data, video data, and code data and is different from the first data type.
16. The method of claim 12 wherein the one or more merged LLMs has at least one of an improved precision and an improved accuracy relative to at least one of the first LLM and the second LLM.
17. The method of claim 12 wherein the first LLM is trained in parallel with the second LLM.
18. The method of claim 12 wherein:
the first LLM is a first family of LLMs (h-LLM);
the second LLM is a second h-LLM; and
the one or more merged LLMs is one or more merged h-LLMs.
19. The method of claim 12 wherein the one or more merged LLMs are produced by a merging process that is configured to produce a merged LLM that is specialized to perform a specific task.
20. The method of claim 12 further comprising filtering the plurality of results for at least one of accuracy or service level assurance.
21. The method of claim 12 wherein:
the one or more merged LLMs is added to a plurality of merged LLMs; and
providing the plurality of updated derived prompts to the one or more merged LLMs comprises sending the plurality of updated derived prompts to the plurality of merged LLMs, generating the plurality of results.
22. A system for generating a merged large language model (LLM) from input data comprising:
a processor;
a network communication device positioned in communication with the processor and operable to communicate across a computer network; and
a non-transitory computer-readable storage medium positioned in communication with the processor and having stored thereon software that, when executed by the processor is operable to:
receive a plurality of derived prompts generated from a user prompt and one or more relevant contexts;
receive the one or more relevant contexts;
receive a plurality of updated derived prompts from the plurality of derived prompts and one or more relevant contexts;
provide the plurality of updated derived prompts to one or more merged LLMs, generating a plurality of results; and
send the plurality of results to an output broker.