Multi-dimensional entity generation from natural language input
Aspects of the present disclosure relate to systems and methods for creating a multi-dimensional entity (MDE) based on natural language (NL) input. A user may provide NL input into an application. One or more skills may be identified for the NL input, each of which has an associated prompt template. For example, a skill is associated with a computer-aided design and/or three-dimensional manufacturing application and/or file format, thereby enabling the generation of output associated with such applications and/or file formats. In examples, a skill chain may be generated that includes one or more skills with which to generate MDE output accordingly.
1 . A system comprising:
at least one processor; and
memory storing instructions that, when executed by the at least one processor, cause the system to perform a set of operations, the set of operations comprising:
receiving, from a computing device, a natural language input that includes a description of a multi-dimensional entity;
generating, using a machine learning model, multi-dimensional entity output responsive to the natural language input, wherein the multi-dimensional entity output defines a representation of the multi-dimensional entity, the generating comprising:
populating, for each skill of a skill chain, a prompt template based on the received natural language input;
processing the respective populated prompt template for each skill to generate machine learning model output for each skill; and
combining the machine learning model output for each skill of the skill chain to generate the multi-dimensional entity output; and
providing, to the computing device, the generated multi-dimensional entity output.
2 . The system of claim 1 , wherein generating the model output comprises:
generating, based on the natural language input, the skill chain to generate the indicated multi-dimensional entity, wherein each skill of the skill chain is associated with at least a portion of the user input.
3 . The system of claim 2 , wherein the natural language input includes a target output indication of at least one of a target application or a target data format for the multi-dimensional entity output.
4 . The system of claim 3 , wherein a skill of the skill chain is associated with the target output indication, thereby generating the multi-dimensional entity output according to the target output indication.
5 . The system of claim 2 , wherein:
a first skill of the skill chain is associated with a first subpart of the multi-dimensional entity; and
a second skill of the skill chain is associated with a second subpart of the multi-dimensional entity.
6 . The system of claim 5 , wherein a third skill of the skill chain processes model output of the first skill and model output of the second skill to generate the multi-dimensional entity output.
7 . The system of claim 1 , wherein the generated multi-dimensional entity output includes at least one of:
instructions to render the multi-dimensional entity in a virtual environment; or
instructions to fabricate a physical representation of the multi-dimensional entity.
8 . A computer-implemented method, comprising:
obtaining user input corresponding to a multi-dimensional entity, wherein the user input includes a target output indication;
generating a skill chain to generate the multi-dimensional entity output using a machine learning model based on the target output indication, the generating comprising generating a prompt based at least in part on the received natural language input, thereby causing the machine learning model to generate the skill chain based on the prompt;
for each skill in the skill chain:
populating a prompt template corresponding to the skill; and
processing, using a machine learning model, the populated prompt template to generate model output for the skill;
combining the model output for each skill of the skill chain to generate the multi-dimensional entity output; and
generating, based on the multi-dimensional entity output, a display of the multi-dimensional entity.
9 . The computer-implemented method of claim 8 , wherein:
the user input corresponding to the multi-dimensional entity comprises an indication of the multi-dimensional entity in a first format; and
the target output indication corresponds to a second format that is different than the first format.
10 . The computer-implemented method of claim 9 , wherein the prompt further comprises an indication of the first format.
11 . The computer-implemented method of claim 8 , wherein the target output indication indicates at least one of a target application or a target data format for the multi-dimensional entity output.
12 . The computer-implemented method of claim 8 , wherein each skill of the skill chain is associated with at least a portion of the user input.
13 . The computer-implemented method of claim 8 , wherein a skill of the skill chain is associated with the target output indication, thereby generating the multi-dimensional entity output according to the target output indication.
14 . A computer-implemented method, comprising:
receiving, from a computing device, a natural language input that includes an indication of a multi-dimensional entity;
generating, based on the natural language input, a skill chain to generate the indicated multi-dimensional entity, wherein each skill of the skill chain is associated with at least a portion of the user input;
for each skill in the skill chain:
populating a prompt template corresponding to each skill;
processing, using a machine learning model, the prompt template for each skill to generate model output for the skill;
combining the model output for each skill of the skill chain to generate multi-dimensional entity output that is responsive to the natural language input; and
providing, to the computing device, the generated multi-dimensional entity output.
15 . The computer-implemented method of claim 14 , wherein the natural language input includes a target output indication of at least one of a target application or a target data format for the multi-dimensional entity output.
16 . The computer-implemented method of claim 15 , wherein a skill of the skill chain is associated with the target output indication, thereby generating the multi-dimensional entity output according to the target output indication.
17 . The computer-implemented method of claim 14 , wherein:
a first skill of the skill chain is associated with a first subpart of the multi-dimensional entity; and
a second skill of the skill chain is associated with a second subpart of the multi-dimensional entity.
18 . The computer-implemented method of claim 17 , wherein a third skill of the skill chain processes model output of the first skill and model output of the second skill to generate the multi-dimensional entity output.
19 . The computer-implemented method of claim 14 , wherein the generated multi-dimensional entity output includes at least one of:
instructions to render the multi-dimensional entity in a virtual environment; or
instructions to fabricate a physical representation of the multi-dimensional entity.
20 . The computer-implemented method of claim 14 , wherein the natural language input comprises at least one of a speech input or text input obtained from a user of the computing device.