Orchestrating dynamic actions through multi-modality inputs
Certain aspects of the disclosure provide for processing multi-modality inputs to trigger dynamic actions. In examples, a method may include: receiving user input through one or more input modalities; converting the user input and one or more supported actions obtained from an actions repository into a unified schema; applying a set of moderation rules to the unified schema to generate a moderated input; generating a prompt based on the moderated input and one or more influencing examples; processing the prompt using one or more Large Language Models (LLMs) to obtain one or more matched actions and populated data models; augmenting the one or more matched actions with additional data; initiating one or more supported actions based on the one or more matched actions and populated data models; and generating a response to the user input based on results of executing the one or more supported actions.
1 . A method for processing multi-modality inputs to trigger dynamic actions, the method comprising:
receiving user input through one or more input modalities, wherein the one or more input modalities include at least one of voice, image, or text;
converting the user input and one or more supported actions obtained from an actions repository into a unified schema, wherein the unified schema represents the user input and the one or more supported actions in a standardized format;
applying a set of moderation rules to the unified schema to generate a moderated input, including:
filtering irrelevant or unnecessary information from the unified schema; and
constraining the unified schema to adhere to specific formatting or structural requirements;
generating a prompt based on the moderated input and one or more influencing examples;
processing the prompt using one or more Large Language Models (LLMs) to obtain one or more matched actions and populated data models, wherein the one or more matched actions represents a mapping between the user input and the one or more supported actions;
augmenting the one or more matched actions with additional data, wherein the additional data provides context for executing the one or more matched actions;
initiating one or more supported actions based on the one or more matched actions and populated data models; and
generating a response to the user input based on results of executing the one or more supported actions.
2 . The method of claim 1 , wherein the one or more influencing examples includes at least one of: examples of desired response formats or structures; guidance on a tone, style, or level of formality to be used in the generated responses; or domain-specific terminology or conventions.
3 . The method of claim 1 , wherein generating the prompt based on the moderated input and the one or more influencing examples comprises:
combining the moderated input and the one or more influencing examples into a single input format; and
providing the combined moderated input to the one or more LLMs for processing.
4 . The method of claim 1 , wherein augmenting the one or more matched actions with additional data comprises:
identifying missing or incomplete information in the one or more matched actions; and
retrieving the missing or incomplete information from a data service.
5 . The method of claim 1 , wherein executing the one or more of the supported actions comprises executing the one or more of the supported actions through an adaptive user interface that dynamically adapts to specific requirements of a respective action of the one or more of the supported actions.
6 . The method of claim 5 , wherein the adaptive user interface comprises a set of reusable user interface components dynamically assembled based on requirements of each action.
7 . The method of claim 1 , wherein generating the response to the user input comprises presenting the response using an adaptive user interface.
8 . The method of claim 7 , wherein generating the response to the user input further comprises:
aggregating results of the one or more of the supported actions;
formatting the aggregated results into a user-friendly format; and
presenting the formatted aggregated results to a user through the adaptive user interface.
9 . The method of claim 8 , wherein the user-friendly format includes at least one of text, speech, visual elements, or interactive components that is specific to a respective action of the one or more of the supported actions.
10 . The method of claim 1 , wherein converting the user input and the one or more supported actions into the unified schema comprises:
identifying entities and intents associated with the user input; and
mapping the identified entities and intents to corresponding elements in the unified schema.
11 . The method of claim 1 , wherein initiating the one or more supported actions comprises:
prioritizing the one or more matched actions based on a set of criteria to obtain prioritized one or more matched actions; and
executing the prioritized one or more matched actions in a sequential order.
12 . The method of claim 11 , wherein the set of predefined criteria includes at least one of user preferences, action relevance, or action complexity.
13 . A processing system, comprising:
a memory comprising computer-executable instructions; and
a processor configured to execute the computer-executable instructions and cause the processing system to:
receive user input through one or more input modalities, wherein the one or more input modalities include at least one of voice, image, or text;
the user input and one or more supported actions obtained from an actions repository into a unified schema, wherein the unified schema represents the user input and the one or more supported actions in a standardized format;
apply a set of moderation rules to the unified schema to generate a moderated input, wherein to apply the set of moderation rules to the unified schema comprises:
to filter out irrelevant or unnecessary information from the unified schema, and
to constrain the unified schema to adhere to specific formatting or structural requirements;
generate a prompt based on the moderated input and one or more influencing examples;
process the prompt with one or more Large Language Models (LLMs) to obtain one or more matched actions and populated data models, wherein the one or more matched actions represents a mapping between the user input and the one or more supported actions;
augment the one or more matched actions with additional data, wherein the additional data provides context for executing the one or more matched actions;
initiate one or more of the one or more supported actions based on the one or more matched actions and populated data models; and
generate a response to the user input based on results of executing the one or more supported actions.
14 . The processing system of claim 13 , wherein to generate the prompt based on the moderated input and the one or more influencing examples, the computer-executable instructions are further executable by the processor to cause the processing system to:
combine the moderated input and the one or more influencing examples into a single input format; and
provide the combined moderated input to the one or more LLMs for processing.
15 . The processing system of claim 13 , wherein to augment the one or more matched actions with additional data, the computer-executable instructions are further executable by the processor to cause the processing system to:
identify missing or incomplete information in the one or more matched actions; and
retrieve the missing or incomplete information from a data service.
16 . The processing system of claim 13 , wherein to execute one or more of the supported actions, the computer-executable instructions are further executable by the processor to cause the processing system to execute one or more of the supported actions through an adaptive user interface that dynamically adapts to specific requirements of a respective action of the one or more of the supported actions.
17 . The processing system of claim 13 , wherein to generate the response to the user input, the computer-executable instructions are further executable by the processor to cause the processing system to present the response using an adaptive user interface.
18 . The processing system of claim 17 , wherein to generate the response to the user input, the computer-executable instructions are further executable by the processor to cause the processing system to:
aggregate results of the one or more of the supported actions;
format the aggregated results into a user-friendly format; and
present the formatted aggregated results to a user through the adaptive user interface.
19 . The processing system of claim 13 , wherein to convert the user input and the one or more supported actions into the unified schema, the computer-executable instructions are further executable by the processor to cause the processing system to:
identify entities and intents associated with the user input; and
map the identified entities and intents to corresponding elements in the unified schema.
20 . The processing system of claim 13 , wherein to initiate the one or more supported actions, the computer-executable instructions are further executable by the processor to cause the processing system to:
prioritize the one or more matched actions based on a set of criteria to obtain prioritized one or more matched actions; and
execute the prioritized one or more matched actions in a sequential order.