Learning and applying empirical knowledge of environments by robots
Techniques described herein relate to generating a posteriori knowledge about where objects are typically located within environments to improve object location. In various implementations, output from vision sensor(s) of a robot may include visual frame(s) that capture at least a portion of an environment in which a robot operates/will operate. The visual frame(s) may be applied as input across a machine learning model to generate output that identifies potential location(s) of an object of interest. The robot's position/pose may be altered based on the output to relocate one or more of the vision sensors. One or more subsequent visual frames that capture at least a not-previously-captured portion of the environment may be applied as input across the machine learning model to generate subsequent output identifying the object of interest. The robot may perform task(s) that relate to the object of interest.
1. A method implemented by one or more processors, comprising:
determining an object of interest;
receiving vision data, the vision data generated based on vision sensor output from one or more vision sensors of a vision component of a robot, the vision data including one or more visual frames that capture at least a portion of an environment in which a robot operates or will operate;
applying one or more of the visual frames as input across a machine learning model to generate output, wherein the output identifies one or more probabilities that a plurality of pixels representing one or more surfaces in the portion of the environment captured in the one or more visual frames that conceal, from a vantage point of the one or more vision sensors, an instance of the object of interest; and
altering a position or pose of the robot based on the output to relocate one or more of the vision sensors to have a direct view behind one or more of the surfaces;
wherein the machine learning model was trained using one or more annotated vision frames in which a portion of a surface of the one or more annotated vision frames is annotated as concealing another object of interest of a same type as the object of interest.
2. The method of claim 1 , wherein the machine learning model comprises a convolutional neural network.
3. The method of claim 1 , wherein the input applied across the machine learning model includes a reduced dimensionality embedding of the object of interest.
4. A method implemented by one or more processors, comprising:
determining an object of interest;
receiving vision data, the vision data generated based on vision sensor output from one or more vision sensors of a vision component of a robot, the vision data including at least one visual frame that captures at least a portion of an environment in which a robot operates or will operate;
applying the at least one visual frame as input across a machine learning model to generate output, wherein the output identifies one or more other portions of the environment that are outside of the portion of the environment captured by the at least one visual frame, wherein the one or more other portions of the environment potentially include an instance of the object of interest;
altering a position or pose of the robot based on the output to relocate one or more of the vision sensors to have a direct view of a given other portion of the one or more other portions of the environment;
obtaining, from one or more of the vision sensors, at least one subsequent visual frame that captures the given other portion of the environment;
applying the at least one subsequent visual frame as input across the machine learning model to generate subsequent output, wherein the subsequent output identifies the instance of the object of interest; and
operating the robot to perform one or more tasks that relate to the instance of the object of interest.
5. The method of claim 4 , wherein the machine learning model comprises a convolutional neural network.
6. The method of claim 4 , wherein the input applied across the machine learning model includes a reduced dimensionality embedding of the object of interest.