Deep compositional robotic planners that follow natural language commands
The present approach similarly combines task and motion planning, but does so without symbolic representations and begins with simpler tasks than other models in such domains can handle. Unlike prior approaches, the present approach does so in continuous action and state spaces which require many precise steps in the configuration space to execute what otherwise is a single output token such as “pick up” for discrete problems.
1. A method comprising:
determining, by a planner, a first point in a selected neighborhood of a configuration space;
receiving a natural language input, uttered by a human or generated by a computer;
parsing the received natural language input to determine (i) linguistic structure of the natural language input and (ii) a plurality of words in the natural language input;
modifying a first neural network (NN) to encode the determined linguistic structure of the natural language input by arranging a plurality of component neural networks of the first NN in an order corresponding to the determined linguistic structure of the natural language input, wherein each component neural network corresponds to one or more respective words of the determined plurality of words;
determining, by the modified first NN, a second point in the selected neighborhood of the configuration space;
choosing among the first point and second point to generate an additional node to add to a search tree; and
adding the additional node to the search tree by connecting the additional node to a node associated with the selected neighborhood;
wherein a second NN controls behavior of an agent corresponding to the search tree having the additional node.
2. The method of claim 1 , further comprising:
determining the selected neighborhood by:
determining a first neighborhood to add the additional node to the search tree by evaluating one or more selected nodes of the search tree with the planner, each selected node representing a coordinate in the configuration space;
determining a second neighborhood to add the additional node to the search tree by evaluating the one or more selected nodes of the search tree with the modified first NN; and
choosing, as the selected neighborhood, one neighborhood among the first neighborhood and second neighborhood based on at least one of: a respective level of confidence determined for the first neighborhood and second neighborhood, at least one extrinsic factor, and an impossibility factor indicative of whether a path to each node of the selected neighborhood is possible.
3. The method of claim 1 , wherein choosing includes producing hypotheses for the planner and for the modified first NN, the hypotheses including at least one of confidence weights and features.
4. The method of claim 1 , wherein choosing includes evaluating one or more nodes of the search tree with the planner or the modified first NN, said evaluating being based on observations.
5. The method of claim 1 , wherein the second NN includes component NNs.
6. The method of claim 1 , wherein the second NN outputs to the agent at least one of a path, destination, and stopping point.
7. A system comprising:
a processor; and
a memory with computer code instructions stored thereon, the processor and the memory, with the computer code instructions, being configured to cause the processor to:
determine, by a planner, a first point in a selected neighborhood of a configuration space;
receive a natural language input, uttered by a human or generated by a computer;
parse the received natural language input to determine (i) linguistic structure of the natural language input and (ii) a plurality of words in the natural language input;
modify a first neural network (NN) to encode the determined linguistic structure of the natural language input by arranging a plurality of component neural networks of the first NN in an order corresponding to the determined linguistic structure of the natural language input, wherein each component neural network corresponds to one or more respective words of the determined plurality of words,
determine, by the modified first NN, a second point in the selected neighborhood of the configuration space;
choose among the first point and second point to generate an additional node to add to a search tree; and
add the additional node to the search tree by connecting the additional node to a node associated with the selected neighborhood;
wherein a second NN controls behavior of an agent corresponding to the search tree having the additional node.
8. The system of claim 7 , wherein choosing includes producing hypotheses for the planner and for the modified first NN, the hypotheses including at least one of confidence weights and features.
9. The system of claim 7 , wherein the instructions further configure the processor to:
determine the selected neighborhood by:
determining a first neighborhood to add the additional node to the search tree by evaluating one or more selected nodes of the search tree with the planner, each selected node representing a coordinate in the configuration space;
determining a second neighborhood to add the additional node to the search tree by evaluating the one or more selected nodes of the search tree with the modified first NN; and
choosing as the selected neighborhood, one neighborhood among the first neighborhood and second neighborhood based on at least one of: a respective level of confidence determined for the first neighborhood and second neighborhood, at least one extrinsic factor, and an impossibility factor indicative of whether a path to each node of the selected neighborhood is possible.
10. A method comprising:
determining, by a planner, a first neighborhood of a configuration space and a first point in the first neighborhood;
receiving a natural language input, uttered by a human or generated by a computer;
parsing the received natural language input to determine (i) linguistic structure of the natural language input and (ii) a plurality of words in the natural language input;
modifying a first neural network (NN) to encode the determined linguistic structure of the natural language input by arranging a plurality of component neural networks of the first NN in an order corresponding to the determined linguistic structure of the natural language input, wherein each component neural network corresponds to one or more respective words of the determined plurality of words;
determining, by the modified first NN, a second neighborhood of a configuration space and a second point in the second neighborhood;
choosing as a selected neighborhood, one neighborhood among the first neighborhood and second neighborhood based on at least one of: a respective level of confidence determined for the first neighborhood and second neighborhood, at least one extrinsic factor, and an impossibility factor indicative of whether a path to each node of the selected neighborhood is possible;
choosing among the first point and second point to generate an additional node to add to a search tree; and
adding the additional node to the search tree by connecting the additional node to a node associated with the selected neighborhood;
wherein a second NN controls behavior of an agent corresponding to the search tree having the additional node.
11. The method of claim 1 , wherein the first neural network and the second neural network are component neural networks of a given neural network.
12. The system of claim 7 , wherein the first neural network and the second neural network are component neural networks of a given neural network.
13. The method of claim 10 , wherein the first neural network and the second neural network are component neural networks of a given neural network.