Object placement
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for performing planning for robotic placement tasks. One of the methods includes determining an initial in-hand state for a grasped object. A show pose for the grasped object is determined, and the object is moved to the show pose. A refined in-hand state for the grasped object is determined based on the show pose, and a placement plan is determined based on the refined in-hand state for the grasped object.
1. A method performed by one or more computers in communication with a robot, the method comprising:
providing one or more commands to the robot that cause the robot to grasp an object by an end effector of the robot;
determining an initial in-hand state for the grasped object representing an initial pose of the grasped object relative to the end effector of the robot grasping the grasped object;
determining a show pose for the grasped object relative to a sensor in the operating environment of the robot;
moving, by the robot, the object to have the show pose in the operating environment;
determining, using sensor data captured of the object in the show pose, a refined in-hand state for the grasped object representing a refined pose of the grasped object relative to the end effector of the robot;
determining a placement plan based on the refined in-hand state of the grasped object; and
providing one or more commands to the robot that cause the robot to place the grasped object according to the refined in-hand state of the grasped object.
2. The method of claim 1 , wherein determining the initial in-hand state for the grasped object comprises determining an initial estimate of the in-hand state of a to-be-grasped object based on the object's to-be-grasped state.
3. The method of claim 1 , wherein determining the initial in-hand state for the grasped object comprises using a predetermined in-hand state.
4. The method of claim 1 , wherein determining the show pose for the grasped object comprises determining the show pose based on the initial in-hand pose.
5. The method of claim 1 , wherein determining the show pose for the grasped object comprises determining the show pose for a particular vision algorithm.
6. The method of claim 1 , wherein determining the refined in-hand state comprises determining a refined in-hand pose using validation measurements captured at the show pose.
7. The method of claim 1 , wherein determining the refined in-hand state comprises capturing the sensor data of the object at the show pose and using a machine learning model to generate a refined in-hand pose.
8. The method of claim 1 , further comprising executing the placement plan based on a comparison between the initial in-hand pose and the refined in-hand pose.
9. The method of claim 1 , wherein determining the placement plan comprises updating a pre-planned placement plan based on a difference between the refined in-hand state and the initial in-hand state.
10. The method of claim 9 , wherein determining the placement plan comprises:
determining that the difference between the refined in-hand state and the initial in-hand state is not similar;
and in response, discarding a pre-planned placement plan.
11. The method of claim 1 , wherein determining the placement plan comprises computing a new placement plan based on the refined in-hand pose.
12. A system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers in communication with a robot, to cause the one or more computers to perform operations comprising:
providing one or more commands to the robot that cause the robot to grasp an object by an end effector of the robot;
determining an initial in-hand state for the grasped object representing an initial pose of the grasped object relative to the end effector of the robot grasping the grasped object;
determining a show pose for the grasped object relative to a sensor in the operating environment of the robot;
moving, by the robot, the object to have the show pose in the operating environment;
determining, using sensor data captured of the object in the show pose, a refined in-hand state for the grasped object representing a refined pose of the grasped object relative to the end effector of the robot;
determining a placement plan based on the refined in-hand state of the grasped object; and
providing one or more commands to the robot that cause the robot to place the grasped object according to the refined in-hand state of the grasped object.
13. The system of claim 12 , wherein determining the initial in-hand state for the grasped object comprises determining an initial estimate of the in-hand state of a to-be-grasped object based on the object's to-be-grasped state.
14. The system of claim 12 , wherein determining the initial in-hand state for the grasped object comprises using a predetermined in-hand state.
15. The system of claim 12 , wherein determining the show pose for the grasped object comprises determining the show pose based on the initial in-hand pose.
16. The system of claim 12 , wherein determining the show pose for the grasped object comprises determining the show pose for a particular vision algorithm.
17. The system of claim 12 , wherein determining the refined in-hand state comprises determining a refined in-hand pose using validation measurements captured at the show pose.
18. The system of claim 12 , wherein determining the refined in-hand state comprises capturing the sensor data of the object at the show pose and using a machine learning model to generate a refined in-hand pose.
19. The system of claim 12 , further comprising executing the placement plan based on a comparison between the initial in-hand pose and the refined in-hand pose.
20. A computer storage medium encoded with a computer program, the program comprising instructions that are operable, when executed by data processing apparatus in communication with a robot, to cause the data processing apparatus to perform operations comprising:
providing one or more commands to the robot that cause the robot to grasp an object by an end effector of the robot;
determining an initial in-hand state for the grasped object representing an initial pose of the grasped object relative to the end effector of the robot grasping the grasped object;
determining a show pose for the grasped object relative to a sensor in the operating environment of the robot;
moving, by the robot, the object to have the show pose in the operating environment;
determining, using sensor data captured of the object in the show pose, a refined in-hand state for the grasped object representing a refined pose of the grasped object relative to the end effector of the robot;
determining a placement plan based on the refined in-hand state of the grasped object; and
providing one or more commands to the robot that cause the robot to place the grasped object according to the refined in-hand state of the grasped object.