Multi-stage multi-hop natural language and model execution plan generation
Machine learning models are used to invoke a sequence of tools in response to a user request. A list of available tools is culled before a machine learning model selects from the remaining tools to generate a plan. The plan, which describes tool invocations in natural language, is then converted to code that can programmatically invoke the tools. In some configurations, the list of available tools is first culled by removing tools that do not appear in similarly described plans. The list may further be culled by removing tools that do not appear in plans generated by machine learning models. Then, a machine learning model is prompted to generate a final plan from the culled list and the user request. In some configurations, the final plan is converted to code by prompting a machine learning model to extrapolate from the code of similar plans.
1 . A method comprising:
receiving a user request;
identifying a plurality of pre-generated plans that have embeddings within a defined distance of an embedding of the user request, wherein each of the plurality of pre-generated plans references one or more tools of a plurality of available tools;
identifying tools referenced by at least one of the identified plurality of pre-generated plans as a first plurality of tools;
generating a plurality of model-generated plans that respond to the user request and that are limited to using at least one of the first plurality of tools;
identifying tools referenced by at least one of the plurality of model-generated plans as a second plurality of tools; and
providing a machine learning model with a prompt that instructs the machine learning model to generate a plan that responds to the user request and that is limited to using at least one of the second list of tools; and
invoking the plan.
2 . The method of claim 1 , wherein the plan comprises a natural language description of invoking the at least one of the second plurality of tools, the method further comprising:
converting the natural language description of invoking the at least one of the second plurality of tools to source code capable of programmatically invoking the at least one of the second plurality of tools, wherein invoking the plan comprises invoking the source code capable of programmatically invoking the at least one of the second plurality of tools.
3 . The method of claim 2 , wherein the natural language description of invoking the at least one of the second plurality of tools is converted to the source code capable of programmatically invoking the at least one of the second plurality of tools by:
providing a second machine learning model with a prompt that includes an example natural language description of invoking the at least one of the second plurality of tools and corresponding example source code capable of programmatically invoking the tool.
4 . The method of claim 1 , wherein an individual pre-generated plan of the plurality of pre-generated plans is constructed by randomly selecting tools from a list of available tools.
5 . The method of claim 4 , wherein the tools of the individual pre-generated plan of the plurality of pre-generated plans are selected such that values passed between the tools have compatible data types.
6 . The method of claim 1 , wherein the plurality of model-generated plans are generated by providing a model generation prompt to a third machine learning model, wherein the model generation prompt includes a list of the first plurality of tools, the user prompt, and a request to generate the plurality of model-generated plans using at least one of the first list of tools.
7 . A system comprising:
a processing unit; and
a computer-readable storage medium having computer-executable instructions stored thereupon, which, when executed by the processing unit, cause the processing unit to:
receive a user request;
identify a plurality of pre-generated plans that have embeddings within a defined distance of an embedding of the user request, wherein each of the plurality of pre-generated plans references one or more tools of a plurality of available tools;
identify tools referenced by at least one of the identified plurality of pre-generated plans as a first plurality of tools;
generate a plurality of model-generated plans that respond to the user request and that are limited to using at least one of the first plurality of tools;
identify tools referenced by at least one of the plurality of model-generated plans as a second plurality of tools;
provide a machine learning model with a prompt that instructs the machine learning model to generate a plan that responds to the user request and that is limited to using at least one of the second plurality of tools;
convert a natural language description included in the plan of invoking the one of the second plurality of tools to source code capable of programmatically invoking the one of the second plurality of tools; and
invoke the plan.
8 . The system of claim 7 , wherein the user request comprises a natural language description of a task, and wherein invoking the plan invokes source code that performs the task.
9 . The system of claim 7 , wherein an embedding of one of the plurality of pre-generated plans is computed of a description of the one of the plurality of pre-generated plans, wherein the description is generated for an individual pre-generated plan of the plurality of pre-generated plans by submitting a prompt to an individual machine learning model that includes the one or more tools referenced by the individual pre-generated plan and instructions to generate a description for the pre-generated plan.
10 . The system of claim 7 , wherein the pre-generated plans are constructed by systematically combining all permutations of tools.
11 . The system of claim 7 , wherein the plan comprises a third plurality of tools selected from the second plurality of tools.
12 . The system of claim 11 , wherein the plan receives an output from a first of the third plurality of tools and provides the output as input to a second of the third plurality of tools.
13 . A computer-readable storage medium having encoded thereon computer-readable instructions that when executed by a processing unit causes a system to:
receive a user request;
identify a plurality of pre-generated plans that have embeddings within a defined distance of an embedding of the user request, wherein each of the plurality of pre-generated plans references one or more tools of a plurality of available tools;
identify tools referenced by at least one of the identified plurality of pre-generated plans as a first plurality of tools;
generate a plurality of model-generated plans that respond to the user request using tools selected from the first plurality of tools;
identify tools referenced by at least one of the plurality of model-generated plans as a second plurality of tools;
generate a plan that, when executed, responds to the user request, wherein the plan comprises at least one natural language description of a tool invocation, wherein the plan is generated by providing a first machine learning model with instructions to generate the plan using tools selected from the second plurality of tools;
convert the at least one natural language description of the tool invocation to tool invocation code by providing a second machine learning model with a prompt that includes an example natural language description of invoking the tool and corresponding example source code capable of programmatically invoking the tool; and
invoke the plan by invoking at least the tool invocation code.
14 . The computer-readable storage medium of claim 13 , wherein the prompt includes a list of input variables provided to a tool referenced by the example natural language description and a list of output variables received from the tool referenced by the example natural language description.
15 . The computer-readable storage medium of claim 13 , wherein the prompt includes example source code invocations of at least one tool referenced by the example natural language description.
16 . The computer-readable storage medium of claim 13 , wherein the example natural language description of invoking the tool and corresponding example source code capable of programmatically invoking the tools are part of an example plan, wherein the example plan is selected for inclusion in the prompt based on having a tool in common with the plan.