Sensor-based adaptation for manipulation of deformable workpieces

1 . A computer-implemented method comprising:

receiving a nominal representation of a workpiece for a robotic process that involves one or more robots performing a manipulation task on multiple instances of the workpiece, wherein the nominal representation represents geometry of the workpiece before deformation;

receiving point cloud data captured by a sensor for a deformed workpiece;

generating deformation data for the deformed workpiece that represents how the deformed workpiece differs from the nominal representation of the workpiece, including performing an iterative process that repeatedly updates a current estimate of the deformation data until a stopping condition is reached, including, on each step of the iterative process, performing operations comprising:

computing edge-to-edge constraints that match one or more edges in the point cloud data to one or more edges in the nominal representation, including:

analyzing the current estimate of the deformation data to extract edges,

classifying the extracted edges as being interior edges or silhouette edges, and

matching edges classified as being interior edges while ignoring silhouette edges, and

adjusting a current estimate of the deformation data according to the computed edge-to-edge constraints;

adjusting the manipulation task according to the generated deformation data; and

performing the robotic process on the deformed workpiece using the manipulation task adjusted according to the generated deformation data.

2 . The method of claim 1 , wherein extracting edges comprises extracting only visible edges that can be observed by the sensor.

3 . The method of claim 1 , wherein computing the edge-to-edge constraints comprises:

determining whether a first edge in the nominal representation has a same edge direction as a second edge extracted from the point cloud data.

4 . The method of claim 1 , wherein computing the edge-to-edge constraints comprises:

determining whether normal vectors of surfaces along a first edge in the nominal representation match normal vectors of corresponding surfaces in the point cloud data.

5 . The method of claim 1 , further comprising:

evaluating the deformed workpiece according to the edge-to-edge constraints; and

rejecting the deformed workpiece if a result of the evaluation does not satisfy one or more evaluation criteria.

6 . The method of claim 5 , wherein evaluating the deformed workpiece according to the edge-to-edge constraints comprises:

computing a number of matching edges between the nominal representation and the edges extracted from the point cloud data,

wherein determining whether the result of the evaluation satisfies the one or more evaluation criteria comprises comparing the computed number of matching edges to a threshold number of matching edges.

7 . The method of claim 1 , further comprising:

computing a score for the deformation data according to the edge-to-edge constraints; and

raising an error if the score for the deformation data does not satisfy one or more quality criteria.

8 . A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

receiving a nominal representation of a workpiece for a robotic process that involves one or more robots performing a manipulation task on multiple instances of the workpiece, wherein the nominal representation represents geometry of the workpiece before deformation;

receiving point cloud data captured by a sensor for a deformed workpiece;

generating deformation data for the deformed workpiece that represents how the deformed workpiece differs from the nominal representation of the workpiece, including performing an iterative process that repeatedly updates a current estimate of the deformation data until a stopping condition is reached, including, on each step of the iterative process, performing operations comprising:

computing edge-to-edge constraints that match one or more edges in the point cloud data to one or more edges in the nominal representation, including:

analyzing the current estimate of the deformation data to extract edges,

classifying the extracted edges as being interior edges or silhouette edges, and

matching edges classified as being interior edges while ignoring silhouette edges, and

adjusting a current estimate of the deformation data according to the computed edge-to-edge constraints;

adjusting the manipulation task according to the generated deformation data; and

performing the robotic process on the deformed workpiece using the manipulation task adjusted according to the generated deformation data.

9 . The system of claim 8 , wherein extracting edges comprises extracting only visible edges that can be observed by the sensor.

10 . The system of claim 8 , wherein computing the edge-to-edge constraints comprises:

determining whether a first edge in the nominal representation has a same edge direction as a second edge extracted from the point cloud data.

11 . The system of claim 8 , wherein computing the edge-to-edge constraints comprises:

determining whether normal vectors of surfaces along a first edge in the nominal representation match normal vectors of corresponding surfaces in the point cloud data.

12 . The system of claim 8 , wherein the operations further comprise:

evaluating the deformed workpiece according to the edge-to-edge constraints; and

rejecting the deformed workpiece if a result of the evaluation does not satisfy one or more evaluation criteria.

13 . The system of claim 12 , wherein evaluating the deformed workpiece according to the edge-to-edge constraints comprises:

computing a number of matching edges between the nominal representation and the edges extracted from the point cloud data,

wherein determining whether the result of the evaluation satisfies the one or more evaluation criteria comprises comparing the computed number of matching edges to a threshold number of matching edges.

14 . The system of claim 8 , wherein the operations further comprise:

computing a score for the deformation data according to the edge-to-edge constraints; and

raising an error if the score for the deformation data does not satisfy one or more quality criteria.

15 . One or more non-transitory computer storage media encoded with computer program instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

receiving a nominal representation of a workpiece for a robotic process that involves one or more robots performing a manipulation task on multiple instances of the workpiece, wherein the nominal representation represents geometry of the workpiece before deformation;

receiving point cloud data captured by a sensor for a deformed workpiece;

generating deformation data for the deformed workpiece that represents how the deformed workpiece differs from the nominal representation of the workpiece, including performing an iterative process that repeatedly updates a current estimate of the deformation data until a stopping condition is reached, including, on each step of the iterative process, performing operations comprising:

computing edge-to-edge constraints that match one or more edges in the point cloud data to one or more edges in the nominal representation, including:

analyzing the current estimate of the deformation data to extract edges,

classifying the extracted edges as being interior edges or silhouette edges, and

matching edges classified as being interior edges while ignoring silhouette edges, and

adjusting a current estimate of the deformation data according to the computed edge-to-edge constraints;

adjusting the manipulation task according to the generated deformation data; and

performing the robotic process on the deformed workpiece using the manipulation task adjusted according to the generated deformation data.

16 . A computer-implemented method comprising:

receiving a nominal representation of a workpiece for a robotic process that involves one or more robots performing a manipulation task on multiple instances of the workpiece, wherein the nominal representation represents geometry of the workpiece before deformation;

receiving point cloud data captured by a sensor for a deformed workpiece;

generating deformation data for the deformed workpiece that represents how the deformed workpiece differs from the nominal representation of the workpiece, including performing an iterative process that repeatedly updates a current estimate of the deformation data until a stopping condition is reached, including, on each step of the iterative process, performing operations comprising:

computing edge-to-edge constraints that match one or more edges in the point cloud data to one or more edges in the nominal representation, including determining whether normal vectors of surfaces along a first edge in the nominal representation match normal vectors of corresponding surfaces in the point cloud data, and

adjusting a current estimate of the deformation data according to the computed edge-to-edge constraints;

adjusting the manipulation task according to the generated deformation data; and

performing the robotic process on the deformed workpiece using the manipulation task adjusted according to the generated deformation data.

17 . A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

receiving a nominal representation of a workpiece for a robotic process that involves one or more robots performing a manipulation task on multiple instances of the workpiece, wherein the nominal representation represents geometry of the workpiece before deformation;

receiving point cloud data captured by a sensor for a deformed workpiece;

generating deformation data for the deformed workpiece that represents how the deformed workpiece differs from the nominal representation of the workpiece, including performing an iterative process that repeatedly updates a current estimate of the deformation data until a stopping condition is reached, including, on each step of the iterative process, performing operations comprising:

computing edge-to-edge constraints that match one or more edges in the point cloud data to one or more edges in the nominal representation, including determining whether normal vectors of surfaces along a first edge in the nominal representation match normal vectors of corresponding surfaces in the point cloud data, and

adjusting a current estimate of the deformation data according to the computed edge-to-edge constraints;

adjusting the manipulation task according to the generated deformation data; and

performing the robotic process on the deformed workpiece using the manipulation task adjusted according to the generated deformation data.

18 . A computer-implemented method comprising:

receiving a nominal representation of a workpiece for a robotic process that involves one or more robots performing a manipulation task on multiple instances of the workpiece, wherein the nominal representation represents geometry of the workpiece before deformation;

receiving point cloud data captured by a sensor for a deformed workpiece;

generating deformation data for the deformed workpiece that represents how the deformed workpiece differs from the nominal representation of the workpiece, including performing an iterative process that repeatedly updates a current estimate of the deformation data until a stopping condition is reached, including, on each step of the iterative process, performing operations comprising:

computing edge-to-edge constraints that match one or more edges in the point cloud data to one or more edges in the nominal representation, and

adjusting a current estimate of the deformation data according to the computed edge-to-edge constraints;

evaluating the deformed workpiece according to the edge-to-edge constraints, including computing a number of matching edges between the nominal representation and the edges extracted from the point cloud data;

rejecting the deformed workpiece if the result of the evaluation does not satisfy one or more evaluation criteria, wherein the one or more evaluation criteria comprises comparing the computed number of matching edges to a threshold number of matching edges; and

if the result of the evaluation satisfies the one or more evaluation criteria, adjusting the manipulation task according to the generated deformation data and performing the robotic process on the deformed workpiece using the manipulation task adjusted according to the generated deformation data.

19 . A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

receiving a nominal representation of a workpiece for a robotic process that involves one or more robots performing a manipulation task on multiple instances of the workpiece, wherein the nominal representation represents geometry of the workpiece before deformation;

receiving point cloud data captured by a sensor for a deformed workpiece;

generating deformation data for the deformed workpiece that represents how the deformed workpiece differs from the nominal representation of the workpiece, including performing an iterative process that repeatedly updates a current estimate of the deformation data until a stopping condition is reached, including, on each step of the iterative process, performing operations comprising:

computing edge-to-edge constraints that match one or more edges in the point cloud data to one or more edges in the nominal representation, and

adjusting a current estimate of the deformation data according to the computed edge-to-edge constraints;

evaluating the deformed workpiece according to the edge-to-edge constraints, including computing a number of matching edges between the nominal representation and the edges extracted from the point cloud data;

rejecting the deformed workpiece if the result of the evaluation does not satisfy one or more evaluation criteria, wherein the one or more evaluation criteria comprises comparing the computed number of matching edges to a threshold number of matching edges; and

if the result of the evaluation satisfies the one or more evaluation criteria, adjusting the manipulation task according to the generated deformation data and performing the robotic process on the deformed workpiece using the manipulation task adjusted according to the generated deformation data.