Systems and methods for automated response to natural language instructions
View Patent ↗Systems and methods for automated response to natural language instructions in accordance with embodiments of the invention are illustrated. One embodiment includes a method for training an agent, the method including sampling an instruction and observation pair from a dataset, predicting a skill code, using a skill predictor, based on the instruction and observation pair, predicting, for each of a plurality of timesteps, a set of one or more actions based on the predicted skill code and a state history using a policy, and updating the skill predictor and the policy based on a comparison of the predicted set of actions and the observation. In many embodiments, the trained agent can then be used to carry out natural language instructions.
1 . A method for enabling a machine to act upon natural language instructions, comprising:
obtaining a plurality of instruction and observation pairs;
generating language embeddings for each instruction in the plurality of instruction and observation pairs using a language encoder;
generating observation embeddings for each observation in the plurality of instruction and observation pairs;
predicting a set of skill codes for each given pair in the plurality of instruction and observation pairs based on a given language embedding and a given observation embedding generated from the given pair using a skill predictor;
predicting an action to correctly resolve the instruction of the given pair using a policy based on the set of skill codes; and
controlling a device to perform the predicted action.
2 . The method of claim 1 , wherein the set of skill codes are human interpretable.
3 . The method of claim 1 , wherein vector quantization is used to translate the predicted set of skill codes into enumerated skill codes from a digital codebook.
4 . The method of claim 1 , wherein the controlled device is a robot.
5 . The method of claim 1 , wherein the controlled device is an autonomous vehicle.
6 . The method of claim 1 , wherein the controlled device is a virtual avatar.
7 . A system for enabling a machine to act upon natural language instructions, comprising:
a processor;
a controllable device; and
a memory, comprising a natural language processing application that configures the processor to:
obtain a plurality of instruction and observation pairs;
generate language embeddings for each instruction in the plurality of instruction and observation pairs using a language encoder;
generate observation embeddings for each observation in the plurality of instruction and observation pairs;
predict a set of skill codes for each given pair in the plurality of instruction and observation pairs based on a given language embedding and a given observation embedding generated from the given pair using a skill predictor;
predict an action to correctly resolve the instruction of the given pair using a policy based on the set of skill codes; and
control the controllable device to perform the predicted action.
8 . The system of claim 7 , wherein the set of skill codes are human interpretable.
9 . The system of claim 7 , wherein vector quantization is used to translate the predicted set of skill codes into enumerated skill codes from a digital codebook.
10 . The system of claim 7 , wherein the controlled device is a robot.
11 . The system of claim 7 , wherein the controlled device is an autonomous vehicle.
12 . The system of claim 7 , wherein the controlled device is a virtual avatar.