Generating and/or using training instances that include previously captured robot vision data and drivability labels
Implementations set forth herein relate to generating training data, such that each instance of training data includes a corresponding instance of vision data and drivability label(s) for the instance of vision data. A drivability label can be determined using first vision data from a first vision component that is connected to the robot. The drivability label(s) can be generated by processing the first vision data using geometric and/or heuristic methods. Second vision data can be generated using a second vision component of the robot, such as a camera that is connected to the robot. The drivability labels can be correlated to the second vision data and thereafter used to train one or more machine learning models. The trained models can be shared with a robot(s) in furtherance of enabling the robot(s) to determine drivability of areas captured in vision data, which is being collected in real-time using one or more vision components.
1. A method implemented by one or more processors of a robot, the method comprising:
generating first vision data using one or more first vision components that are connected to the robot,
wherein the first vision data characterizes a portion of an area that the robot is approaching;
determining, based on the first vision data, whether the portion of the area includes a surface that is traversable by the robot;
generating second vision data that characterizes a separate portion of the area,
wherein the second vision data is generated using one or more second vision components that are: separate from the one or more first vision components, and also connected to the robot;
determining, based on the second vision data, whether an additional surface included in the separate portion of the area is traversable by the robot,
wherein determining whether the additional surface that is traversable by the robot is performed using one or more machine learning models, and
wherein the one or more machine learning models are trained using one or more instances of training data that include vision data characterizing one or more particular surfaces and label data characterizing drivability of the one or more particular surfaces; and
causing the robot to operate according to whether the surface and the additional surface are determined to be traversable.
2. The method of claim 1 , wherein determining whether the portion of the area includes the surface that is traversable by the robot includes:
identifying, based on the first vision data, one or more particular objects that are present in the area, and
determining a height of the one or more particular objects relative to a ground surface that is supporting the robot when the one or more first vision components captured the first vision data,
wherein determining whether the portion of the area includes the surface that is traversable by the robot is at least partially based on the height of the one or more particular objects.
3. The method of claim 1 , wherein determining whether the portion of the area includes the surface that is traversable by the robot includes:
processing portions of the first vision data that do not correspond to one or more particular objects identified, via the first vision data, as present in the area,
wherein determining whether the portion of the area includes the surface that is traversable by the robot is at least partially based on processing the portions of the first vision data that do not correspond to one or more particular objects identified.
4. The method of claim 1 , further comprising:
prior to generating the second vision data:
receiving the one or more machine learning models from a separate computing device that is in communication with the robot.
5. The method of claim 1 , wherein the one or more first vision components includes a LIDAR device and the one or more second vision components include a camera.
6. The method of claim 1 , wherein determining whether the portion of the area includes the surface that is traversable by the robot includes determining whether the robot can autonomously drive over the surface.
7. A robot, comprising:
one or more processors, and
memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations that include:
generating first vision data using one or more first vision components that are in communication with the one or more processors,
wherein the first vision data characterizes a portion of an area that is within a viewable region of the one or more first vision components;
determining, based on the first vision data, whether the portion of the area includes a surface that is traversable;
generating second vision data that characterizes a separate portion of the area,
wherein the second vision data is generated using one or more second vision components that are: separate from the one or more first vision components, and also in communication with the one or more processors;
determining, based on the second vision data, whether an additional surface included in the separate portion of the area is traversable,
wherein determining whether the additional surface that is traversable is performed using one or more machine learning models, and
wherein the one or more machine learning models are trained using one or more instances of training data that characterize drivability of one or more particular surfaces; and
operating according to whether the surface and the additional surface are determined to be traversable.
8. The robot of claim 7 , wherein determining whether the portion of the area includes the surface that is traversable includes:
identifying, based on the first vision data, one or more particular objects that are present in the area, and
determining a height of the one or more particular objects relative to a ground surface,
wherein determining whether the portion of the area includes the surface that is traversable is at least partially based on the height of the one or more particular objects.
9. The robot of claim 7 , wherein determining whether the portion of the area includes the surface that is traversable includes:
processing portions of the first vision data that do not correspond to one or more particular objects identified, via the first vision data, as present in the area,
wherein determining whether the portion of the area includes the surface that is traversable is at least partially based on processing the portions of the first vision data that do not correspond to one or more particular objects identified.
10. The robot of claim 7 , wherein the operations further include:
prior to generating the second vision data:
receiving the one or more machine learning models from a separate computing device that is in communication with the one or more processors.
11. The robot of claim 7 , wherein determining whether the portion of the area includes the surface that is traversable includes determining whether the surface can be autonomously driven over by one or more particular robots.