Granted Patent B2

US 12,508,713 · App. 18/179,936 · Granted Dec 30, 2025

Inventor: Floris Gaisser (Tokyo, JP)

Assignee: MUJIN, INC.

B25J9/1697B25J9/1664B65G47/905

Abstract

Claims (60)

1 . A method of object transfer comprising performed by a control system having at least one processing circuit and being configured to communicate with a robot having a robot arm that includes or is attached to an end effector apparatus and to communicate with a camera, the method comprising:

obtaining image information of one or more objects contained within a source of objects;

identifying a pickable region of one or more of selected object from the objects by:

generating a surface cost map according to height gradients of the image information,

segmenting the surface cost map to obtain one or more image segments identifying one or more pickable regions corresponding to the one or more selected object; and

generating a pickable region detection result including at least the one or more pickable regions; and

generating a motion plan for a robotic system to transfer the one or more selected objects, the motion plan being based on the pickable region detection result.

2 . The method of claim 1 , wherein the surface cost map is representative of a smoothness of the one or more selected objects.

3 . The method of claim 1 , wherein the image information includes three-dimensional information, the method further comprising generating the surface cost map according to the height gradients and normal differences between defined cells of the image information.

4 . The method of claim 3 , further comprising generating the surface cost map according to surface cost map parameters.

5 . The method of claim 4 , further comprising:

registering the one or more objects based on the image information to create object registration information; and

determining the surface cost map parameters according to the object registration information.

6 . The method of claim 1 , further comprising generating detection mask information indicating the one or more pickable regions of the image segments, the detection mask information including detected areas and occluded areas within the one or more image segments.

7 . The method of claim 1 , wherein segmenting the surface cost map includes:

applying a cost threshold to the surface cost map to generate a thresholded mask;

eroding the thresholded mask to generate an eroded mask; and

applying a connected components analysis to the eroded mask to identify a first image segment.

8 . The method of claim 7 , wherein segmenting the surface cost map further includes:

removing the first image segment from surface cost map;

applying a second cost threshold to a remaining portion of the surface cost map to generate a second thresholded mask;

eroding the second thresholded mask to generate a second eroded mask; and

applying the connected components analysis to the second eroded mask to identify a second image segment.

9 . The method of claim 1 , wherein generating the pickable region detection result further includes

generating a safety volume around the one or more pickable regions, the safety volume indicating an estimated remaining portion of the one or more selected objects.

10 . A non-transitory computer readable medium, configured with executable instructions for object transfer to be performed by a control system having at least one processing circuit and being configured to communicate with a robot having a robot arm that includes or is attached to an end effector apparatus and to communicate with a camera, the instructions being configured for:

obtaining image information of one or more objects contained within a source of objects;

identifying a pickable region of one or more of selected object from the objects by:

generating a surface cost map according to height gradients of the image information,

segmenting the surface cost map to obtain one or more image segments identifying one or more pickable regions corresponding to the one or more selected object; and

generating a pickable region detection result including at least the one or more pickable regions; and

generating a motion plan for a robotic system to transfer the one or more selected objects, the motion plan being based on the pickable region detection result.

11 . The non-transitory computer readable medium of claim 10 , wherein the image information includes three-dimensional information, the instructions being further configured for generating the surface cost map according to the height gradients and normal differences between defined cells of the image information.

12 . A computing system comprising:

a control system configured to communicate with a robot having a robot arm that includes or is attached to an end effector apparatus and to communicate with a camera;

at least one processing circuit configured, when the robot is in an object handling environment including a source of objects for transfer to a destination within the object handling environment, to:

obtaining image information of the objects;

identifying a pickable region of one or more of selected object from the objects by:

generating a surface cost map according to height gradients of the image information,

segmenting the surface cost map to obtain one or more image segments identifying one or more pickable regions corresponding to the one or more selected object; and

generating a pickable region detection result including at least the one or more pickable regions; and

generating a motion plan for a robotic system to transfer the one or more selected objects, the motion plan being based on the pickable region detection result.

13 . The system of claim 12 , wherein the surface cost map is representative of a smoothness of the one or more selected objects.

14 . The system of claim 12 , wherein the image information includes three-dimensional information and the processing circuit is further configured for generating the surface cost map according to the height gradients and normal differences between defined cells of the image information.

15 . The system of claim 14 , wherein the at least one processing circuit is further configured for generating the surface cost map according to surface cost map parameters.

16 . The system of claim 15 , wherein the at least one processing circuit is further configured for:

registering the one or more objects based on the image information to create object registration information; and

determining the surface cost map parameters according to the object registration information.

17 . The system of claim 12 , wherein the at least one processing circuit is further configured for generating detection mask information indicating the one or more pickable regions of the image segments, the detection mask information including detected areas and occluded areas within the one or more image segments.

18 . The system of claim 12 , wherein segmenting the surface cost map includes:

applying a cost threshold to the surface cost map to generate a thresholded mask;

eroding the thresholded mask to generate an eroded mask; and

applying a connected components analysis to the eroded mask to identify a first image segment.

19 . The system of claim 18 , wherein segmenting the surface cost map further includes:

removing the first image segment from surface cost map;

applying a second cost threshold to a remaining portion of the surface cost map to generate a second thresholded mask;

eroding the second thresholded mask to generate a second eroded mask; and

applying the connected components analysis to the second eroded mask to identify a second image segment.

20 . The system of claim 12 , wherein generating the pickable region detection result further includes

generating a safety volume around the one or more pickable regions, the safety volume indicating an estimated remaining portion of the one or more selected objects.

Assignments (1)

Continuity (2)

References Cited (10)