Method and system for multi-level artificial intelligence supercomputer design
A method for utilizing hierarchical tokens (h-tokens) in large language models (LLMs) including receiving and processing an input prompt to generate tokens, identifying functional components derived from a subset of tokens, associating or generating h-tokens for at least one of the functional components by compressing the subset of tokens into an h-token for each functional component, associating or generating implementation characteristics including a defined function, an identified event, and/or an implementation method for each h-token, performing a retrieval augmented generation (RAG) processing function on each h-token, and generating a response from an output of the RAG processing function.
1 . A method for utilizing hierarchical tokens (h-tokens) in large language models (LLMs) comprising:
receiving an input prompt;
performing input processing on the input prompt comprising generating a plurality of tokens from the input prompt;
identifying one or more functional components from the input prompt, each functional component of the one or more functional components being derived from a subset of tokens of the plurality of tokens;
at least one of associating or generating one or more h-tokens for at least one functional component of the one or more functional components, comprising compressing the subset of tokens of the plurality of tokens from which a functional component of the one or more functional components was derived into an h-token of the one or more h-tokens, wherein each h-token of the one or more h-tokens is associated or generated for a respective functional component of the one or more functional components and preserves a semantic meaning of the subset of tokens of the plurality of tokens from which the respective functional component of the one or more functional components was derived;
at least one of associating or generating one or more implementation characteristics comprising at least one of a defined function, an identified event, and an implementation method for each h-token of the one or more h-tokens, each implementation characteristic to be comprised by the h-token of the one or more h-tokens for which it was generated or associated;
performing a retrieval augmented generation (RAG) processing function on each h-token of the one or more h-tokens responsive to the one or more implementation characteristics of the h-token; and
generating a response from an output of the RAG processing function.
2 . The method of claim 1 wherein the one or more implementation characteristics for each h-token comprises at least one of:
a defined function describing an operation to be performed;
an identified event describing an outcome of the defined function; and
an implementation method describing how to execute the defined function.
3 . The method of claim 1 wherein input processing further comprises identifying one or more domains indicated by the input prompt.
4 . The method of claim 3 further comprising performing a domain-specific analysis for each domain of the one or more domains.
5 . The method of claim 1 wherein identifying the one or more functional components comprises:
analyzing the input prompt to identify at least one main function; and
decomposing the at least one main function into one or more sub-functions;
wherein each sub-function of the one or more sub-functions defines a functional component of the one or more functional components.
6 . The method of claim 5 wherein identifying the one or more functional components further comprises:
performing a hierarchical mapping of each sub-function of the one or more sub-functions; and
performing a functional relationship analysis of the one or more sub-functions responsive to the hierarchical mapping.
7 . The method of claim 1 wherein at least one functional component of the one or more functional components is associated with a microservice implemented on a cloud-based container that is accessible via an application programming interface.
8 . The method of claim 1 wherein generating the one or more h-tokens comprises:
analyzing each functional component of the one or more functional components to identify a functional area of each functional component of the one or more functional components;
organizing the one or more h-tokens responsive to a hierarchy of the one or more functional components; and
mapping a relationship of the one or more h-tokens.
9 . The method of claim 8 wherein organizing the one or more h-tokens responsive to a hierarchy comprises:
creating a plurality of levels of abstraction, with higher levels representing broader functional concepts and lower levels representing specific implementations;
associating each h-token of the one or more h-tokens with a level of the plurality of levels of abstraction; and
organizing each h-token of the one or more h-tokens responsive to its associated level of abstraction.
10 . The method of claim 1 wherein performing the RAG processing function comprises:
identifying a context for each h-token of the one or more h-tokens;
selecting a processing modality for each h-token of the one or more h-tokens;
executing the selected processing modality for each h-token of the one or more h-tokens to generate one or more processed h-token outputs; and
generating one or more coherent outputs by combining each processed h-token output of the one or more processed h-token outputs with the context for the h-token from which the processed h-token output was produced;
wherein the output of the RAG processing function comprises the one or more coherent outputs.
11 . The method of claim 10 wherein generating a response from an output of the RAG processing function comprises:
assembling a response data content comprising the one or more coherent outputs;
forming a response from the response data content; and
performing a quality verification on the response.
12 . The method of claim 1 wherein a functional component of the one or more functional components is associated with at least one of a superchunk and a chunk.
13 . A system for utilizing hierarchical tokens (h-tokens) in large language models (LLMs) comprising:
a processor;
a communication device operable to communicate with a remote computerized device and operably coupled to the processor; and
a non-transitory computer-readable medium having stored thereon software that, when executed by the processor, is operable to:
receive an input prompt;
perform input processing on the input prompt comprising generating a plurality of tokens from the input prompt;
identify one or more functional components from the input prompt, each functional component of the one or more functional components being derived from a subset of tokens of the plurality of tokens;
at least one of associate or generate one or more h-tokens for at least one functional component of the one or more functional components, comprising compressing the subset of tokens of the plurality of tokens from which a functional component of the one or more functional components was wherein each h-token of the one or more h-tokens is associated or generated for a respective functional component of the one or more functional components and preserves a semantic meaning of the subset of tokens of the plurality of tokens from which the respective functional component of the one or more functional components was derived;
at least one of associate or generate one or more implementation characteristics comprising at least one of a defined function, an identified event, and an implementation method for each h-token of the one or more h-tokens, each implementation characteristic to be comprised by the h-token of the one or more h-tokens for which it was generated or associated;
perform a retrieval augmented generation (RAG) processing function on each h-token of the one or more h-tokens responsive to the one or more implementation characteristics of the h-token; and
generate a response from an output of the RAG processing function.
14 . The system of claim 13 wherein the one or more implementation characteristics for each h-token comprises at least one of:
a defined function describing an operation to be performed;
an identified event describing an outcome of the defined function; and
an implementation method describing how to execute the defined function.
15 . The system of claim 13 wherein input processing further comprises identifying one or more domains indicated by the input prompt.
16 . The system of claim 15 wherein the software is further configured to, when executed by the processor, perform a domain-specific analysis for each domain of the one or more domains.
17 . The system of claim 13 wherein the software is configured to, when executed by the processor, identify the one or more functional components by:
analyzing the input prompt to identify at least one main function; and
decomposing the at least one main function into one or more sub-functions;
wherein each sub-function of the one or more sub-functions defines a functional component of the one or more functional components.
18 . The system of claim 17 wherein the software is configured to, when executed by the processor, identify the one or more functional components by:
performing a hierarchical mapping of each sub-function of the one or more sub-functions; and
performing a functional relationship analysis of the one or more sub-functions responsive to the hierarchical mapping.
19 . The system of claim 13 wherein at least one functional component of the one or more functional components is associated with a microservice implemented on a cloud-based container that is accessible via an application programming interface.
20 . The system of claim 13 wherein the software is further configured to, when executed by the processor, generate the one or more h-tokens by:
analyzing each functional component of the one or more functional components to identify a functional area of each functional component of the one or more functional components;
organizing the one or more h-tokens responsive to a hierarchy of the one or more functional components; and
mapping a relationship of the one or more h-tokens.
21 . The system of claim 20 wherein the software is further configured to, when executed by the processor, organize the one or more h-tokens responsive to a hierarchy by:
creating a plurality of levels of abstraction, with higher levels representing broader functional concepts and lower levels representing specific implementations;
associating each h-token of the one or more h-tokens with a level of the plurality of levels of abstraction; and
organizing each h-token of the one or more h-tokens responsive to its associated level of abstraction.
22 . The system of claim 13 wherein the software is further configured to, when executed by the processor, perform the RAG processing function by:
identifying a context for each h-token of the one or more h-tokens;
selecting a processing modality for each h-token of the one or more h-tokens;
executing the selected processing modality for each h-token of the one or more h-tokens to generate one or more processed h-token outputs; and
generating one or more coherent outputs by combining each processed h-token output of the one or more processed h-token outputs with the context for the h-token from which the processed h-token output was produced;
wherein the output of the RAG processing function comprises the one or more coherent outputs.
23 . The system of claim 22 wherein the software is further configured to, when executed by the processor, generate a response from an output of the RAG processing function by:
assembling a response data content comprising the one or more coherent outputs;
forming a response from the response data content; and
performing a quality verification on the response.
24 . The system of claim 13 wherein a functional component of the one or more functional components is associated with at least one of a superchunk and a chunk.