Device and method for training a machine-learning model for determining a grasp of a multi-finger gripper for manipulating an object

1 . A computer-implemented method for training a machine-learning model to determine grasps of a multi-finger gripper for manipulating an object, the method comprising:

generating, for a scene that includes an instance of the object, a three-dimensional point cloud representation of the scene including the object;

obtaining, from the three-dimensional point cloud representation, a subset of distinct three-dimensional points that collectively represent three-dimensional contours of an exterior surface of the object;

identifying, from the subset of distinct three-dimensional points that have been obtained from the three-dimensional point cloud representation, a plurality of distinct surface points located at different respective three-dimensional positions on the exterior surface of the object, wherein at least a subset of the plurality of distinct surface points each is associated with a respective set of one or more of a plurality of ground-truth grasps;

determining, by the machine-learning model, a plurality of predicted grasps for the plurality of distinct surface points, wherein each respective one of the plurality of predicted grasps:

is associated with a respective surface point of the plurality of distinct surface points; and

includes a predicted palm pose and predicted finger-joint configurations of the multi-finger gripper;

determining, for each respective surface point of the subset of the plurality of distinct surface points, a respective grasp-loss value representing a difference between a respective predicted grasp associated with the respective surface point and a selected ground-truth grasp from the respective set associated with the respective surface point, wherein the grasp-loss values are based on a collision loss and/or a grasp stability loss;

combining the grasp-loss values for the plurality of distinct surface points to determine an overall training loss for the scene; and

adjusting parameters of the machine-learning model to reduce the overall training loss.

2 . The method of claim 1 , wherein the grasp stability loss is an upper bound of a contact-stability metric representing a quality of each respective one of the determined predicted grasps.

3 . The method of claim 1 , further comprising:

determining, from the subset of distinct three-dimensional points that were obtained from the three-dimensional point cloud, a surface mesh of the object, wherein the determination of the overall training loss includes determining, for each respective one of the determined predicted grasps, a guidance loss which penalizes distance between contact points of the multi-finger gripper according to a respective predicted grasp and a surface of the object as given by the surface mesh.

4 . The method of claim 1 , further comprising, for each respective one of the determined predicted grasps, determining, by the machine-learning model, a confidence score for a respective predicted grasp and reducing a loss contribution of the respective predicted grasp according to the confidence score, where the higher the confidence score, the greater is the reduction.

5 . The method according to claim 1 , further comprising:

obtaining a further point cloud representation, which is of a further object that is to be manipulated;

determining a further grasp for the further object by feeding the further point cloud representation of the further object to the trained machine-learning model; and

controlling a robot to perform the determined further grasp to manipulate the further object.

6 . The method of claim 1 , further comprising downsampling the three-dimensional point cloud representation of the scene to obtain a downsampled set of three-dimensional points, wherein the plurality of distinct surface points comprises the downsampled set, and wherein the machine-learning model determines the respective predicted grasp for each respective surface point of the downsampled set.

7 . The method of claim 1 , wherein each respective one of the plurality of predicted grasp determined by the machine-learning model is represented in coordinates relative to the respective surface point with which the respective predicted grasp is associated.

8 . The method of claim 1 , further comprising determining, by the machine-learning model, a respective confidence score for each respective one of the plurality of predicted grasps, wherein the confidence score is an output of the machine-learning model that represents an estimated reliability of the respective predicted grasp, and wherein the confidence scores are used in selecting, from the plurality of predicted grasps, at least one grasp to be executed by the multi-finger gripper.

9 . The method of claim 1 , wherein identifying the respective set of one or more ground-truth grasps associated with each surface point of the subset comprises, for each respective surface point, selecting ground-truth grasps whose respective palm positions are within a predetermined spatial distance of the respective surface point, such that the respective set includes ground-truth grasps having palm positions sufficiently near that surface point.

10 . The method of claim 9 , wherein when the respective set of ground-truth grasps associated with a surface point is empty, the surface point is excluded from the determination of the respective grasp-loss values, and wherein, when the respective set is non-empty, determining the respective grasp-loss value comprises:

comparing each ground-truth grasp of the respective set of ground-truth grasps associated with the surface point to the respective predicted grasp for that surface point using a predefined metric of comparison to find which of the compared ground-truth grasps is a best-matching ground-truth grasp that matches most closely to the respective predicted grasp; and

selecting, from the respective set of ground-truth grasps associated with the surface point, the best-matching ground-truth grasp; and

determining the grasp-loss value based on a difference between the predicted grasp and the selected best-matching ground-truth grasp.

11 . The method of claim 1 , further comprising:

processing a plurality of additional scenes that each includes a respective instance of the object by repeating, for each of the additional scenes, the generating, obtaining, identifying, determining of predicted grasps, determining of grasp-loss values, and combining of grasp-loss values to obtain a respective overall training loss for each of the additional scenes; and

adjusting the parameters of the machine-learning model based on an aggregation of the overall training losses for the scene and the additional scenes.

12 . A data processing device configured to train a machine-learning model to determine grasps of a multi-finger gripper for manipulating an object, the data processing device comprising a processing system that includes at least one processor, wherein the processing system is programmed to:

generate, for a scene that includes an instance of the object, a three-dimensional point cloud representation of the scene including the object;

obtain, from the three-dimensional point cloud representation, a subset of distinct three-dimensional points that collectively represent three-dimensional contours of an exterior surface of the object;

identify, from the subset of distinct three-dimensional points that have been obtained from the three-dimensional point cloud representation, a plurality of distinct surface points located at different respective three-dimensional positions on the exterior surface of the object, wherein at least a subset of the plurality of distinct surface points each is associated with a respective set of one or more of a plurality of ground-truth grasps;

determine, by the machine-learning model, a plurality of predicted grasps for the plurality of distinct surface points, wherein each respective one of the plurality of predicted grasps:

is associated with a respective surface point of the plurality of distinct surface points; and

includes a predicted palm pose and predicted finger-joint configurations of the multi-finger gripper;

determine, for each respective surface point of the subset of the plurality of distinct surface points, a respective grasp-loss value representing a difference between a respective predicted grasp associated with the respective surface point and a selected ground-truth grasp from the respective set associated with the respective surface point, wherein the grasp-loss values are based on a collision loss and/or a grasp stability loss;

combine the grasp-loss values for the plurality of distinct surface points to determine an overall training loss for the scene; and

adjust parameters of the machine-learning model to reduce the overall training loss.

13 . A non-transitory computer-readable medium on which are stored instructions for training a machine-learning model to determine grasps of a multi-finger gripper for manipulating an object, the instructions, when executed by a processor, causing the processor to perform the following steps:

generating, for a scene that includes an instance of the object, a three-dimensional point cloud representation of the scene including the object;

obtaining, from the three-dimensional point cloud representation, a subset of distinct three-dimensional points that collectively represent three-dimensional contours of an exterior surface of the object;

identifying, from the subset of distinct three-dimensional points that have been obtained from the three-dimensional point cloud representation, a plurality of distinct surface points located at different respective three-dimensional positions on the exterior surface of the object, wherein at least a subset of the plurality of distinct surface points each is associated with a respective set of one or more of a plurality of ground-truth grasps;

determining, by the machine-learning model, a plurality of predicted grasps for the plurality of distinct surface points, wherein each respective one of the plurality of predicted grasps:

is associated with a respective surface point of the plurality of distinct surface points; and

includes a predicted palm pose and predicted finger-joint configurations of the multi-finger gripper;

determining, for each respective surface point of the subset of the plurality of distinct surface points, a respective grasp-loss value representing a difference between a respective predicted grasp associated with the respective surface point and a selected ground-truth grasp from the respective set associated with the respective surface point, wherein the grasp-loss values are based on a collision loss and/or a grasp stability loss;

combining the grasp-loss values for the plurality of distinct surface points to determine an overall training loss for the scene; and

adjusting parameters of the machine-learning model to reduce the overall training loss.