Update of local features model based on correction to robot action
Methods, apparatus, and computer-readable media for determining and utilizing corrections to robot actions. Some implementations are directed to updating a local features model of a robot in response to determining a human correction of an action performed by the robot. The local features model is used to determine, based on an embedding generated over a corresponding neural network model, one or more features that are most similar to the generated embedding. Updating the local features model in response to a human correction can include updating a feature embedding, of the local features model, that corresponds to the human correction. Adjustment(s) to the features model can immediately improve robot performance without necessitating retraining of the corresponding neural network model.
1. A method implemented by one or more processors of a robot, comprising:
determining, during performance of an action by the robot, a particular natural language classification descriptor for an object that is in an environment of the robot and that is being acted upon, or is to be acted upon, by the robot during performance of the action, wherein determining the natural language classification descriptor for the object comprises:
applying, as input to a neural network model locally stored on one or more computer readable media of the robot, vision sensor data that captures the object and that is generated by a vision sensor of the robot,
generating, over the neural network model based on the applying, a visual embedding of the vision sensor data,
comparing the visual embedding to a plurality of classification embeddings, each of the classification embeddings being mapped, in a classification model locally stored on one or more of the computer readable media of the robot, to a corresponding natural language classification descriptor,
determining, based on the comparing, that the visual embedding is most similar to a particular classification embedding of the classification embeddings, and
identifying the particular natural language classification descriptor for the object based on the particular natural language classification descriptor being the corresponding natural language classification descriptor that is mapped to the particular classification embedding;
in response to determining the particular natural language classification descriptor for the object, and in response to the object being acted upon, or to be acted upon, by the robot during performance of the action:
providing, via a speaker of the robot, audible output that is perceivable by a human in the environment and that speaks the particular natural language classification descriptor of the object.
2. The method of claim 1 , wherein comparing the visual embedding to the plurality of classification embeddings comprises generating corresponding distance measures that are each between the visual embedding and a corresponding one of the classification embeddings.
3. The method of claim 2 , wherein determining, based on the comparing, that the visual embedding is most similar to the particular classification embedding of the classification embeddings comprises:
determining that the corresponding distance measure, that is between the visual embedding and the particular classification embedding, is the smallest of the corresponding distance measures.
4. The method of claim 3 , further comprising:
in response to determining the particular natural language classification descriptor for the object, and in response to the object being acted upon, or to be acted upon, by the robot during performance of the action:
providing, via the robot, visual output that is perceivable by a human in the environment and that indicates the particular natural language classification descriptor of the object.
5. The method of claim 3 , further comprising:
receiving user interface input after providing the audible output;
determining, based on the user interface input and based on the user interface input being received after providing the audible output, that the user interface input indicates the particular natural language descriptor of the object is incorrect.
6. The method of claim 5 , further comprising:
in response to determining that the user interface input indicates the classification of the object is incorrect:
updating the locally stored classification model, wherein updating the locally stored classification model occurs without retraining of the neural network model.
7. The method of claim 1 , further comprising:
in response to determining the particular natural language classification descriptor for the object, and in response to the object being acted upon, or to be acted upon, by the robot during performance of the action:
providing, via the robot, visual output that is perceivable by a human in the environment and that indicates the particular natural language classification descriptor of the object.
8. The method of claim 7 , further comprising:
receiving user interface input after providing the audible output; and
determining, based on the user interface input and based on the user interface input being received after providing the audible output, that the user interface input indicates the particular natural language descriptor of the object is incorrect.
9. The method of claim 8 , further comprising:
in response to determining that the user interface input indicates the classification of the object is incorrect:
updating the locally stored classification model, wherein updating the locally stored classification model occurs without retraining of the neural network model.
10. A robot comprising:
a speaker;
a vision sensor;
memory storing instructions;
one or more processors executing the instructions, stored in the memory, to cause one or more of the processors to:
determine, during performance of an action by the robot, a particular natural language classification descriptor for an object that is in an environment of the robot and that is being acted upon, or is to be acted upon, by the robot during performance of the action, wherein in determining the natural language classification descriptor for the object one or more of the processors are to:
apply, as input to a neural network model locally stored at the robot, vision sensor data that captures the object and that is generated by the vision sensor of the robot,
generate, over the neural network model based on the applying, a visual embedding of the vision sensor data,
compare the visual embedding to a plurality of classification embeddings, each of the classification embeddings being mapped, in a classification model locally stored at the robot, to a corresponding natural language classification descriptor,
determine, based on the comparing, that the visual embedding is most similar to a particular classification embedding of the classification embeddings, and
identify the particular natural language classification descriptor for the object based on the particular natural language classification descriptor being the corresponding natural language classification descriptor that is mapped to the particular classification embedding;
in response to determining the particular natural language classification descriptor for the object, and in response to the object being acted upon, or to be acted upon, by the robot during performance of the action:
provide, via the speaker, audible output that is perceivable by a human in the environment and that speaks the particular natural language classification descriptor of the object.
11. The robot of claim 10 , wherein in comparing the visual embedding to the plurality of classification embeddings one or more of the processors are to generate corresponding distance measures that are each between the visual embedding and a corresponding one of the classification embeddings.
12. The robot of claim 11 , wherein in determining, based on the comparing, that the visual embedding is most similar to the particular classification embedding of the classification embeddings one or more of the processors are to:
determine that the corresponding distance measure, that is between the visual embedding and the particular classification embedding, is the smallest of the corresponding distance measures.
13. The robot of claim 12 , wherein one or more of the processors, in executing the instructions, are further to:
in response to determining the particular natural language classification descriptor for the object, and in response to the object being acted upon, or to be acted upon, by the robot during performance of the action:
provide, via the robot, visual output that is perceivable by a human in the environment and that indicates the particular natural language classification descriptor of the object.
14. The robot of claim 13 , wherein one or more of the processors, in executing the instructions, are further to:
receive user interface input after providing the audible output; and
determine, based on the user interface input and based on the user interface input being received after providing the audible output, that the user interface input indicates the particular natural language descriptor of the object is incorrect.
15. The robot of claim 14 , wherein one or more of the processors, in executing the instructions, are further to:
in response to determining that the user interface input indicates the classification of the object is incorrect:
update the locally stored classification model, wherein updating the locally stored classification model occurs without retraining of the neural network model.
16. The robot of claim 10 , wherein one or more of the processors, in executing the instructions, are further to:
in response to determining the particular natural language classification descriptor for the object, and in response to the object being acted upon, or to be acted upon, by the robot during performance of the action:
provide, via the robot, visual output that is perceivable by a human in the environment and that indicates the particular natural language classification descriptor of the object.
17. The robot of claim 16 , wherein one or more of the processors, in executing the instructions, are further to:
receive user interface input after providing the audible output; and
determine, based on the user interface input and based on the user interface input being received after providing the audible output, that the user interface input indicates the particular natural language descriptor of the object is incorrect.
18. The robot of claim 17 , wherein one or more of the processors, in executing the instructions, are further to:
in response to determining that the user interface input indicates the classification of the object is incorrect:
update the locally stored classification model, wherein updating the locally stored classification model occurs without retraining of the neural network model.