Object climbing by legged robots using training objects

1 . A method comprising:

receiving, at data processing hardware of a robot, sensor data of an environment of the robot from one or more sensors of the robot;

identifying, by the data processing hardware, an object in the environment based on the sensor data;

determining, by the data processing hardware, that a characteristic of the object identified in the environment corresponds to a characteristic of a simulated training object of a plurality of simulated training objects generated by a simulation system;

identifying, by the data processing hardware, a climbing operation from a plurality of climbing operations for the robot based on determining that the characteristic of the object identified in the environment corresponds to the characteristic of the simulated training object, wherein each of the plurality of simulated training objects is associated with a respective climbing operation of the plurality of climbing operations; and

instructing, by the data processing hardware, the robot to climb on the object identified in the environment based on the climbing operation.

2 . The method of claim 1 , wherein each of the plurality of climbing operations is associated with one or more of:

a respective foot placement location for one or more feet of the robot;

a respective foot placement timing for one or more feet of the robot;

a respective force at one or more feet of the robot;

a respective pose for the robot;

a respective orientation for the robot;

a respective leg placement location for one or more legs of the robot;

a respective leg placement timing for one or more legs of the robot;

a placement location for a non-foot portion of one or more legs of the robot; or

a placement timing for a non-foot portion of one or more legs of the robot.

3 . The method of claim 1 , further comprising instructing the robot to balance on two feet of the robot based on the climbing operation.

4 . The method of claim 1 , further comprising determining the plurality of climbing operations using a controller, wherein the controller is configured to determine a respective climbing operation of the plurality of climbing operations based on one or more of:

an input to the controller;

a joint position; or

a joint torque command.

5 . The method of claim 1 , further comprising determining the plurality of climbing operations using a controller, wherein the controller is configured to determine a respective climbing operation of the plurality of climbing operations based on an input to the controller, wherein the input to the controller comprises one or more of:

a respective foot placement location for one or more feet of the robot;

a respective pose for the robot;

a respective orientation for the robot;

a respective leg placement location for one or more legs of the robot;

the characteristic of the object identified in the environment;

a height of the object identified in the environment; or

a state of the robot.

6 . The method of claim 1 , further comprising:

receiving second sensor data of the environment from the one or more sensors of the robot;

identifying a second object in the environment based on the second sensor data;

determining that a characteristic of the second object identified in the environment corresponds to a characteristic of a second simulated training object of the plurality of simulated training objects, wherein the characteristic of the second object identified in the environment and the characteristic of the object identified in the environment are different characteristics;

identifying a second climbing operation from the plurality of climbing operations for the robot based on determining that the characteristic of the second object identified in the environment corresponds to the characteristic of the second simulated training object, wherein the second climbing operation and the climbing operation are different climbing operations; and

instructing the robot to climb on the second object identified in the environment based on the second climbing operation.

7 . The method of claim 1 , further comprising:

receiving second sensor data of the environment from the one or more sensors of the robot;

identifying a second object in the environment based on the second sensor data;

determining that a characteristic of the second object identified in the environment does not correspond to a characteristic of a respective simulated training object of the plurality of simulated training objects;

determining that the robot cannot climb on the second object identified in the environment based on determining that the characteristic of the second object identified in the environment does not correspond to a characteristic of a respective simulated training object of the plurality of simulated training objects; and

causing display of a user interface at a user computing device, wherein the user interface includes information indicating that the robot cannot climb on the second object identified in the environment and reflects a virtual representation of one or more of the robot or the second object identified in the environment.

8 . The method of claim 1 , wherein the characteristic of the object identified in the environment comprises one or more of:

a height of the object identified in the environment; or

information indicating that the object identified in the environment comprises a vehicle.

9 . The method of claim 1 , further comprising:

determining that one or more of the robot or the sensor data is associated with noise;

adjusting the characteristic of the simulated training object based on the noise to generate a noisy simulated training object; and

adding the noisy simulated training object to the plurality of simulated training objects.

10 . The method of claim 1 , wherein the plurality of simulated training objects is associated with a plurality of characteristics, the plurality of characteristics comprising a plurality of heights.

11 . The method of claim 1 , wherein the plurality of simulated training objects comprises one or more of a vehicle, a trailer, a box, a platform, a rock wall, or a stage.

12 . The method of claim 1 , wherein the robot comprises two front legs and two rear legs, wherein each of the two front legs have a rear shank portion, wherein instructing the robot to climb on the object identified in the environment comprises:

instructing the robot to place a rear shank portion of a first front leg of the two front legs and a rear shank portion of a second front leg of the two front legs on the object identified in the environment.

13 . The method of claim 1 , wherein the robot comprises two front legs and two rear legs, wherein instructing the robot to climb on the object identified in the environment comprises:

instructing the robot to shift a position of the two rear legs relative to the object identified in the environment;

instructing the robot to provide a downward force via one or more of the two rear legs or the two front legs;

instructing the robot to place the two front legs on the object identified in the environment; and

shifting a weight distribution of the robot.

14 . The method of claim 1 , wherein the robot comprises two front legs and two rear legs, wherein instructing the robot to climb on the object identified in the environment comprises:

instructing the robot to flex the two rear legs;

instructing the robot to extend the two rear legs;

instructing the robot to flex the two front legs and place the two front legs on the object identified in the environment;

instructing the robot to extend the two front legs; and

instructing the robot to flex the two rear legs and place the two rear legs on the object identified in the environment.

15 . The method of claim 1 , wherein instructing the robot to climb on the object identified in the environment based on the climbing operation comprises instructing the robot, using a first controller of the robot, to climb on the object identified in the environment, wherein the robot is configured to utilize a second controller of the robot for stair navigation.

16 . The method of claim 1 , wherein determining that the characteristic of the object identified in the environment corresponds to the characteristic of the simulated training object comprises one or more of:

determining that the characteristic of the object identified in the environment matches the characteristic of the simulated training object;

determining that the characteristic of the object identified in the environment is within a particular degree of similarity of the characteristic of the simulated training object; or

determining that the characteristic of the object identified in the environment corresponds to the characteristic of the simulated training object further based on comparing the characteristic of the object identified in the environment to a respective characteristic of one or more simulated training objects of the plurality of simulated training objects.

17 . The method of claim 1 , wherein identifying the climbing operation comprises accessing a machine learning model, wherein the machine learning model is trained to output a particular climbing operation based on an input of a particular simulated training object, wherein the machine learning model is trained utilizing the plurality of simulated training objects, the method further comprising:

identifying an updated plurality of simulated training objects; and

retraining the machine learning model utilizing the updated plurality of simulated training objects.

18 . The method of claim 1 , further comprising:

causing display of a user interface at a user computing device, wherein the user interface reflects a virtual representation of the object identified in the environment; and

obtaining one or more instructions to climb the object identified in the environment based on the user interface, wherein instructing the robot to climb on the object identified in the environment is further based on the one or more instructions.

19 . A robot comprising:

at least one sensor;

at least two legs;

data processing hardware in communication with the at least one sensor; and

memory in communication with the data processing hardware, the memory storing instructions that when executed on the data processing hardware cause the data processing hardware to:

obtain sensor data captured from an environment by the at least one sensor;

identify an object in the environment based on the sensor data;

determine that the object identified in the environment corresponds to a simulated training object of a plurality of simulated training objects generated by a simulation system based on a characteristic of the object identified in the environment;

identify an operation from a plurality of operations based on determining that the object identified in the environment corresponds to the simulated training object, wherein each of the plurality of simulated training objects is associated with a respective operation of the plurality of operations; and

instruct traversal of the object identified in the environment using the at least two legs based on the operation.

20 . The robot of claim 19 , wherein the operation indicates a series of iterative movements.

21 . The robot of claim 19 , wherein the operation indicates a respective series of iterative movements for each of the at least two legs.

22 . The robot of claim 19 , wherein execution of the instructions on the data processing hardware further causes the data processing hardware to:

determine that one or more of the robot or the sensor data is associated with noise; and

generate the operation based on the noise.

23 . The robot of claim 19 , wherein to identify the operation, execution of the instructions on the data processing hardware further causes the data processing hardware to access a machine learning model, wherein the machine learning model is trained to output a particular operation based on an input of a particular simulated training object, wherein the machine learning model is trained utilizing the plurality of simulated training objects, wherein the execution of the instructions on the data processing hardware further causes the data processing hardware to:

identify a second object;

determine the plurality of simulated training objects does not comprise a particular simulated training object corresponding to the second object; and

retrain the machine learning model utilizing the particular simulated training object corresponding to the second object based on determining the plurality of simulated training objects does not comprise the particular simulated training object corresponding to the second object.

24 . The robot of claim 19 , wherein execution of the instructions on the data processing hardware further causes the data processing hardware to:

access a set of policies, wherein each policy of the set of policies links a particular operation of the plurality of operations to a particular simulated training object of the plurality of simulated training objects, wherein identifying the operation is further based on the set of policies.