Method and system for multi-level artificial intelligence supercomputer design
A method of generating responses responsive to input prompts including receiving a multimodal input prompt, the multi-modal input prompt comprising a first input prompt mode and a second input prompt mode, generating first and second derived prompts responsive to the first and second input prompt modes, transmitting the first and second derived prompts to first and second h-LLMs that are trained using data having a mode that is the same of the respective input prompt modes, receiving one or more first and second results from the first and second h-LLMs, generating at least one combined result from the one or more first and second results, and transmitting the combined result to the user.
1. A method of generating responses responsive to input prompts using one or more families of large language models (h-LLMs) by a computer comprising a processor, a non-transitory storage medium, and software on the storage medium, the method comprising:
receiving a multimodal input prompt at an input broker from a user, the multimodal input prompt comprising a first input prompt mode and a second input prompt mode;
generating a first derived prompt responsive to the first input prompt mode;
generating a second derived prompt responsive to the second input prompt mode;
transmitting the first derived prompt to a first h-LLM that is trained using data having a mode that is the same of the first input prompt mode;
transmitting the second derived prompt to a second h-LLM that is trained using data having a mode that is the same of the second input prompt mode;
receiving one or more first results from the first h-LLM at an output broker;
receiving one or more second results from the second h-LLM at the output broker;
generating at least one combined result from the one or more first results and the one or more second results; and
transmitting the combined result to the user.
2. The method of claim 1 wherein each of the first input prompt mode and the second input prompt mode are one of a text prompt, an image prompt, an audio prompt, or a video prompt.
3. The method of claim 1 wherein the combined result is multimodal.
4. The method of claim 1 , wherein:
the one or more first results are the same mode as the first input prompt mode; and
the one or more second results are the same mode as the second input prompt mode.
5. The method of claim 1 , wherein at least one of the first h-LLM and second h-LLM are fine-tuned to perform one or more specific tasks.
6. The method of claim 5 wherein the one or more specific tasks are selected from the group consisting of question answering, information extraction, sentiment analysis, image captioning, object recognition, instruction following, classification, inferencing, and sentence similarity.
7. The method of claim 1 wherein:
the first h-LLM is trained exclusively using data that is the same mode of the first input prompt mode; and
the second h-LLM is trained exclusively using data that is the same mode of the second input prompt mode.
8. The method of claim 1 wherein the multimodal input prompt further comprises a third input prompt mode, the method further comprising:
generating a third derived prompt responsive to the third input prompt mode;
transmitting the third derived prompt to a third h-LLM that is trained using data having a mode that is the same of the third input prompt mode; and
receiving one or more third results from the third h-LLM at the output broker;
wherein the at least one combined result is generated result from the one or more first results, the one or more second results, and the one or more third results.
9. The method of claim 1 wherein the first h-LLM and the second h-LLM are in parallel with each other.
10. The method of claim 1 wherein a subset of at least one of the one or more first results and the one or more second results are used in generating the at least one combined result.
11. A system for generating responses responsive to input prompts using one or more families of large language models (h-LLMs) comprising:
a processor configured to:
operate an input broker operable to:
receive a multimodal input prompt from a user, the multimodal input prompt comprising a first input prompt mode and a second input prompt mode;
generate a first derived prompt responsive to the first input prompt mode;
generate a second derived prompt responsive to the second input prompt mode;
transmit the first derived prompt to a first h-LLM that is trained using data having a mode that is the same of the first input prompt mode; and
transmit the second derived prompt to a second h-LLM that is trained using data having a mode that is the same of the second input prompt mode;
operate an output broker operable to:
receive one or more first results from the first h-LLM;
receive one or more second results from the second h-LLM;
generate at least one combined result from the one or more first results and the one or more second results; and
transmit the combined result to the user
a non-transitory storage medium positioned in communication with the processor and having stored thereon software that is executable to operate the input broker and the output broker; and
a communication device operable to receive the multimodal input prompt from the user and transmit the combined result to the user.
12. The system of claim 11 wherein each of the first input prompt mode and the second input prompt mode are one of a text prompt, an image prompt, an audio prompt, or a video prompt.
13. The system of claim 11 wherein the combined result is multimodal.
14. The system of claim 11 , wherein:
the one or more first results are the same mode as the first input prompt mode; and
the one or more second results are the same mode as the second input prompt mode.
15. The system of claim 11 , wherein at least one of the first h-LLM and second h-LLM are fine-tuned to perform one or more specific tasks.
16. The system of claim 15 wherein the one or more specific tasks are selected from the group consisting of question answering, information extraction, sentiment analysis, image captioning, object recognition, instruction following, classification, inferencing, and sentence similarity.
17. The system of claim 11 wherein:
the first h-LLM is trained exclusively using data that is the same mode of the first input prompt mode; and
the second h-LLM is trained exclusively using data that is the same mode of the second input prompt mode.
18. The system of claim 11 wherein:
the multimodal input prompt further comprises a third input prompt mode;
the processor is further configured to:
operate the input broker to:
generate a third derived prompt responsive to the third input prompt mode; and
transmit the third derived prompt to a third h-LLM that is trained using data having a mode that is the same of the third input prompt mode; and
operate the output broker to receive one or more third results from the third h-LLM; and
the at least one combined result is generated from the one or more first results, the one or more second results, and the one or more third results.
19. The system of claim 11 wherein the first h-LLM and the second h-LLM are in parallel with each other.
20. The system of claim 11 wherein a subset of at least one of the one or more first results and the one or more second results are used in generating the at least one combined result.