Automated bin-picking based on deep learning

1 . A method of determining a grasp proposal for object picking by a robot gripper comprising:

capturing an image comprising an object to be grasped by the robot gripper;

providing the image to an input of a deep neural network system that is trained to generate an object segmentation map for identifying pixels in the image that are associated with the object and that is trained to generate a plurality of object property maps associated with the object segmentation map, the plurality of object property maps being distinct from the object segmentation map, each object property map linking each pixel of the object that is identified by the object segmentation map to information about a predetermined object property; and,

determining a grasp proposal for a controller of the robot gripper using an algorithm and one or more generated object property maps generated by the deep neural network system, wherein the algorithm is not based on a trainable model.

2 . The method according to claim 1 wherein an object property map defines a pixel-labelled image wherein each pixel value of the object property map defines a measurable object property at a corresponding pixel location of the image.

3 . The method according to claim 1 wherein the plurality of object property maps comprises an angle property map wherein a pixel value of the angle map-property map that is associated with the object includes information about an angle θ x,y between an axis parallel to a path along which the robot gripper approaches the object in a world-frame of the object and a vector that is normal to a surface of the object at a corresponding pixel location of the image.

4 . The method according to claim 1 wherein the plurality of object property maps comprises a curvature property map wherein a pixel value of the curvature property map that is associated with the object includes information about a curvature of the object at a corresponding pixel location of the image.

5 . The method according to claim 4 wherein the plurality of object property maps comprises a distance property map which is computed based on the curvature property map, wherein for each non-zero pixel value of the curvature property map a distance to a nearest zero pixel value is computed.

6 . The method according to claim 4 wherein the plurality of object property maps comprises a coverage property map wherein a pixel value of the curvature property map that is associated with the object includes information about an edge and/or an overhanging surface of the object at the corresponding pixel location of the image.

7 . The method according to claim 1 wherein determining a grasp proposal further includes:

determining a score map by combining information of different object property maps; and

determining one or more surfaces of the object based on the score map.

8 . The method according to claim 1 wherein the deep neural network system is configured to simultaneously generate the object segmentation map and the plurality of object property maps.

9 . The method according to claim 1 wherein the image may include a 2D image frame and a point cloud.

10 . The method according to claim 1 wherein the deep neural network system comprises a plurality output branches, each output branch being associated with a deep neural network that is trained to generate one of the plurality of object property maps.

11 . The method according to claim 1 wherein the trained deep neural network system is configured to simultaneously generate the object segmentation map and the plurality of object property maps.

12 . A method of training a deep neural network for object picking by a robot gripper comprising:

providing training data, comprising an image comprising an object to be grasped by the robot gripper, to an input of a deep neural network system comprising a plurality output branches, each output branch being associated with a deep neural network, the training data further comprising coordinates of an object segmentation map associated with the object in the image and a plurality of object property maps associated with the object segmentation map, the plurality of object property maps being distinct from the object segmentation map, each object property map linking each pixel of the object that is identified by the object segmentation map in the image to information about a predetermined object property; and,

training the deep neural network system based on the training data, the deep neural network being trained to generate an object segmentation map for identifying pixels in the image that are associated with the object and to generate a plurality of object property maps, each object property map linking pixels of the object to information about a predetermined object property, wherein the training includes evaluating outputs of the plurality output branches based on a loss function using the coordinates of the object segmentation map and the plurality of object property maps as a target and adjusting training parameters of the deep neural network system based on the evaluation of the loss function.

13 . The method according to claim 12 wherein the deep neural network system is trained to simultaneously generate an object segmentation map and a plurality of object property maps based on the image that is provided to the input.

14 . A robot system configured to grasp an object comprising:

a robotic system including a controller for controlling a gripper for grasping one or more objects;

an imaging system for capturing images of the one or more objects;

a computer connected to the robotic system and the imaging system, the computer comprising a computer readable storage medium having computer readable program code embodied therewith; and

a processor, coupled to the computer readable storage medium, wherein responsive to executing the computer readable program code, the processor is configured to perform executable operations comprising:

capturing an image representing an object to be grasped by the gripper;

providing the image to an input of a deep neural network system that is trained to generate an object segmentation map for identifying pixels in the image that are associated with the object and that is trained to generate a plurality of object property maps associated with the object segmentation map, the plurality of object property maps being distinct from the object segmentation map, each object property map linking each pixels of the object that is identified by the object segmentation map to information about a predetermined object property; and,

determining a grasp proposal for the controller of the gripper using an algorithm and one or more generated object property maps generated by the deep neural network system, wherein the algorithm is not based on a trainable model.

15 . A method of automatic generation of training data for training a deep neural network comprising:

capturing image data associated with an object to be picked up by a robot gripper, the image data including a 2D image and an associated point cloud;

determining a plurality of locations on the point cloud and using a patch fitting algorithm to fit points of the point cloud associated with each of the plurality of locations to a curved surface patch, each curved surface patch being associated with one or more patch parameters defining at least one of: an orientation of the curved surface patch in a reference frame of the object, a curvature of the curved surface patch, or, dimensions of the carved surface patch; and

generating training data for training the deep neural network, wherein the generating includes determining one or more object property maps based on the one or more patch parameters, each object property map linking pixels of the object in the 2D image to information about a predetermined object property.

16 . The method according to claim 15 wherein determining one or more object property maps includes:

determining an angle property map based on the one or more patch parameters defining an orientation of the curved surface patches in a reference frame of the object; and/or,

determining a curvature property map based on the one or more patch parameters defining curvature of the curved surface patches.

17 . A non-transitory computer-readable storage medium storing at least one software code portion, the software code portion, when run on a computer system, being configured for executing the method steps according to claim 1 .

18 . The method of claim 12 wherein adjusting training parameters comprises adjusting weights of activation functions of the deep neural network.

19 . The method of claim 7 wherein combining includes pixel-wise multiplication of pixel values in the different object property maps.

20 . The method of claim 9 wherein the point cloud is pixel-aligned with the 2D image frame.