Techniques for generating workflows using an LLM-based system

1 . A method performed by a computing system of workflow generation, the method comprising:

receiving, from a user, a first instruction to create a workflow to perform a task;

generating one or more prompts based on the first instruction and a predefined workflow schema, including generating a first prompt requesting a set of steps of the workflow to perform the task;

querying a large language model (LLM) with the one or more prompts, including querying the LLM with the first prompt;

in response to querying the LLM, receiving a workflow in a format consistent with the predefined workflow schema from the LLM;

in response to the LLM responding to the first prompt, generating a second prompt requesting a set of entity properties for entities used to accomplish performance of the set of steps, wherein generating the second prompt includes reference to a predefined entity type schema;

querying the LLM with the second prompt;

receiving the set of entity properties in response to the second prompt;

receiving a second instruction from the user to create workpieces to be used in conjunction with the received workflow;

generating a third prompt based on the second instruction, the received workflow, and the received set of entity properties;

querying the LLM with the third prompt;

in response to querying the LLM with the third prompt, receiving a set of workpiece data structures in a format consistent with the predefined entity type schema from the LLM;

displaying a workpiece visualization of the received set of workpiece data structures to the user; and

displaying a visualization of the received workflow to the user, including:

displaying a step directing the user to photograph one or more workpiece corresponding to one or more workpiece data structures of the set of workpiece data structures;

in response to receiving a picture of the one or more workpiece, comparing the one or more workpiece to the corresponding one or more workpiece data structures and determining that there is an error with the one or more workpiece; and

in response to determining that there is the error, pausing displaying the visualization of the received workflow to the user.

2 . The method of claim 1 wherein the first and second instructions are natural language instructions.

3 . The method of claim 1 wherein the first instruction is a document containing a description of how to perform the task.

4 . The method of claim 1 wherein:

generating the one or more prompts further includes:

in response to the LLM responding to the first prompt with the set of steps, for at least one step of the set of steps, generating a fourth prompt requesting a set of substeps to accomplish performance of that step; and

querying the LLM with the one or more prompts further includes:

in response to generating the fourth prompt, querying the LLM with the fourth prompt.

5 . The method of claim 4 wherein:

generating the one or more prompts further includes, in response to the LLM responding to the first prompt:

displaying a preliminary visualization of the set of steps to the user;

receiving a natural language instruction from the user to modify the set of steps;

generating a fifth prompt based on the natural language instruction and the set of steps;

querying the LLM with the fifth prompt; and

in response to querying the LLM with the fifth prompt, receiving an updated set of steps; and

generating the fourth prompt is performed in response to receiving the updated set of steps.

6 . The method of claim 4 wherein displaying the visualization of the received workflow to the user includes:

for the at least one step of the set of steps, displaying a description of that step, including the substeps for that step, as a checklist; and

in response to the user checking off each substep, marking the at least one step as completed and displaying a description of a next step of the set of steps.

7 . A computer program product comprising a non-transitory computer-readable storage medium storing instructions, which, when performed by processing circuitry of a computing system, cause the computing system to:

receive, from a user, a first instruction to create a workflow to perform a task;

generate one or more prompts based on the first instruction and a predefined workflow schema, including generating a first prompt requesting a set of steps of the workflow to perform the task;

query a large language model (LLM) with the one or more prompts, including querying the LLM with the first prompt;

in response to querying the LLM, receive a workflow in a format consistent with the predefined workflow schema from the LLM;

in response to the LLM responding to the first prompt, generate a second prompt requesting a set of entity properties for entities used to accomplish performance of the set of steps, wherein generating the second prompt includes reference to a predefined entity type schema;

query the LLM with the second prompt;

receive the set of entity properties in response to the second prompt;

receive a second instruction from the user to create workpieces to be used in conjunction with the received workflow;

generate a third prompt based on the second instruction, the received workflow, and the received set of entity properties;

query the LLM with the third prompt;

in response to querying the LLM with the third prompt, receive a set of workpiece data structures in a format consistent with the predefined entity type schema from the LLM;

display a workpiece visualization of the received set of workpiece data structures to the user; and

display a visualization of the received workflow to the user, including:

displaying a step directing the user to photograph one or more workpiece corresponding to one or more workpiece data structures of the set of workpiece data structures;

in response to receiving a picture of the one or more workpiece, comparing the one or more workpiece to the corresponding one or more workpiece data structures and determining that there is an error with the one or more workpiece; and

in response to determining that there is the error, pausing displaying the visualization of the received workflow to the user.

8 . The computer program product of claim 7 wherein the first and second instructions are natural language instructions.

9 . The computer program product of claim 7 wherein the first instruction is a document containing a description of how to perform the task.

10 . The computer program product of claim 7 wherein:

generating the one or more prompts further includes:

in response to the LLM responding to the first prompt with the set of steps, for at least one step of the set of steps, generating a fourth prompt requesting a set of substeps to accomplish performance of that step; and

querying the LLM with the one or more prompts further includes:

in response to generating the fourth prompt, querying the LLM with the fourth prompt.

11 . The computer program product of claim 10 wherein:

generating the one or more prompts further includes, in response to the LLM responding to the first prompt:

displaying a preliminary visualization of the set of steps to the user;

receiving a natural language instruction from the user to modify the set of steps;

generating a fifth prompt based on the natural language instruction and the set of steps;

querying the LLM with the fifth prompt; and

in response to querying the LLM with the fifth prompt, receiving an updated set of steps; and

generating the fourth prompt is performed in response to receiving the updated set of steps.

12 . The computer program product of claim 10 wherein displaying the visualization of the received workflow to the user includes:

for the at least one step of the set of steps, displaying a description of that step, including the substeps for that step, as a checklist; and

in response to the user checking off each substep, marking the at least one step as completed and displaying a description of a next step of the set of steps.

13 . A system comprising:

user interface circuitry configured to:

receive, from a user, a first instruction to create a workflow to perform a task; and

receive a second instruction from the user to create workpieces to be used in conjunction with the workflow; and

processing circuitry coupled to memory configured to:

generate one or more prompts based on the first instruction and a predefined workflow schema, including generating a first prompt requesting a set of steps of the workflow to perform the task;

query a large language model (LLM) with the one or more prompts, including querying the LLM with the first prompt;

in response to querying the LLM, receive a workflow in a format consistent with the predefined workflow schema from the LLM;

in response to the LLM responding to the first prompt, generate a second prompt requesting a set of entity properties for entities used to accomplish performance of the set of steps, wherein generating the second prompt includes reference to a predefined entity type schema;

query the LLM with the second prompt;

receive the set of entity properties in response to the second prompt;

generate a third prompt based on the second instruction, the received workflow, and the received set of entity properties;

query the LLM with the third prompt; and

in response to querying the LLM with the third prompt, receive a set of workpiece data structures in a format consistent with the predefined entity type schema from the LLM;

wherein the user interface circuitry is further configured to display a workpiece visualization of the received set of workpiece data structures to the user and display a visualization of the received workflow to the user, including:

displaying a step directing the user to photograph one or more workpiece corresponding to one or more workpiece data structures of the set of workpiece data structures;

in response to receiving a picture of the one or more workpiece, comparing the one or more workpiece to the corresponding one or more workpiece data structures and determining that there is an error with the one or more workpiece; and

in response to determining that there is the error, pausing displaying the visualization of the received workflow to the user.

14 . The system of claim 13 wherein:

generating the one or more prompts further includes:

in response to the LLM responding to the first prompt with the set of steps, for at least one step of the set of steps, generating a fourth prompt requesting a set of substeps to accomplish performance of that step; and

querying the LLM with the one or more prompts further includes:

in response to generating the fourth prompt, querying the LLM with the fourth prompt.

15 . The system of claim 14 wherein:

generating the one or more prompts further includes, in response to the LLM responding to the first prompt:

displaying a preliminary visualization of the set of steps to the user;

receiving a natural language instruction from the user to modify the set of steps;

generating a fifth prompt based on the natural language instruction and the set of steps;

querying the LLM with the fifth prompt; and

in response to querying the LLM with the fifth prompt, receiving an updated set of steps; and

generating the fourth prompt is performed in response to receiving the updated set of steps.