Granted Patent B2

US 12,406,374 · App. 17/652,533 · Granted Sep 2, 2025

Inventors: Anoop Cherian (Belmont, MA); Tim Marks (Newton, MA); Alan Sullivan (Middleton, MA)

G06T7/11B25J9/1697G06T7/155G06T7/50G06T7/60G06T2207/10028G06T2207/20081G06T2207/20084G06T2207/20152

Abstract

Claims (57)

1. An imaging controller for segmenting instances from depth images including objects to be manipulated by a robot comprising:

an input interface configured to receive a depth image that includes objects;

a memory configured to store instructions and a neural network trained to segment instances from the objects in the depth image; and

a processor, coupled with the memory, configured to perform the instructions to segment a pickable instance using the trained neural network, wherein steps of the instructions comprise:

selecting a tallest point in the depth image;

defining a region using a shape such that the region surrounds the tallest point;

sampling points in the region of the depth image;

computing depth-geodesics between the tallest point and the sampled points;

submitting the depth-geodesics to the neural network to segment the pickable instance among instances of the objects in the depth image; and

an output interface configured to output a geometrical feature of the pickable instance to a manipulator controller of the robot.

2. The imaging controller of claim 1 , wherein the depth images are acquired by a camera or sensor.

3. The imaging controller of claim 1 , wherein the steps further comprises:

selecting a next tallest point from the depth images such that a pick radius around the next tallest point excludes an overlap with the pickable instance;

defining a next region using the shape such that the next region surrounds the next tallest point;

sampling next points in the next region of the depth image;

computing next depth-geodesics between the next tallest point and the sampled next points; and

submitting the depth-geodesics to the neural network to segment a next pickable instance among the instances of the objects in the depth image.

4. The imaging controller of claim 3 , wherein the steps of selecting, defining, the sampling, computing and submitting are continued until the steps are performed to a rest of the objects in the depth image.

5. The imaging controller of claim 1 , wherein the shape is a square, a rectangular, a triangle, a circle or an oval.

6. The imaging controller of claim 1 , wherein the neural network is trained to classify each feature vector as belonging to an identical instance or different instance.

7. The imaging controller of claim 1 , wherein end-points of the geodesics are initialized using peaks produced by a Watershed Algorithm.

8. A computer-implemented method for training a neural network for segmenting instances in depth images, wherein the method uses a processor coupled with stored instructions implementing the method, wherein the instructions, when executed by the processor carry out at steps of the method, comprising steps of:

selecting a depth image from a set of depth images;

determining points of xy-locations on a 2-dimensional image grid and corresponding depth points with respect to the selected depth image, wherein the points on the 2-dimensional image grid are respectively annotated with ground truth instance labels;

computing geodesic straight lines between pairs of the annotated determined points;

generating depth geodesics by projecting the geodesic straight lines on the depth image;

discretizing each of the depth geodesics to create discretized vectors, wherein each discretized vector corresponds to one of the depth geodesics between a pair of the annotated determined points; and

submitting the discretized vectors and corresponding annotated labels of the discretized vectors to the neural network, wherein the steps from the selecting through the providing are repeatedly performed until a rest of all the set of depth images are used.

9. The method of claim 8 , wherein the neural network is trained to classify each feature vector as belonging to an identical instance or different instance.

10. The method of claim 9 , further comprises computing a convex hull of all endpoints of the geodesics that are classified as the geodesics belonging to the identical instance as a pickable point.

11. The method of claim 8 , wherein the generated depth geodesics are debiased such that the geodesics are created from points being lower-depth to points being higher depth.

12. The method of claim 8 , wherein end-points of the geodesics are initialized using peaks produced by a Watershed Algorithm.

13. The method of claim 12 , wherein the endpoints of the geodesic are determined using a systematic selection method.

14. A bin-picking system for picking objects from a bin, comprising:

an end-tool configured to pickup an object from among the objects;

a robot arm including the end-tool, wherein the robot arm is configured to be driven by control signals that include end-tool signals to pickup the object from the bin using the end-tool;

an interface configured to transmit and receive the control signals, sensor signals of sensors arranged on the robot arm, imaging signals of at least one imaging device;

a memory configured to store instructions of a robot control program, and a classifier and a trained neural network that segments instances from the objects in the depth image, the trained neural network having been trained by a computer-implemented method of claim 8 ; and

a processor, coupled with the memory, configured to perform the instructions to segment a pickable instance using the trained neural network and generate the control signals that drive the robot arm and the end-tool, wherein steps of the instructions comprise:

selecting a tallest point in the depth image;

defining a region using a shape such that the region surrounds the tallest point;

sampling points in the region of the depth image;

computing depth-geodesics between the tallest point and the sampled points;

submitting the depth-geodesics to the neural network to segment the pickable instance among instances of the objects in the depth image;

generating a geometrical feature of the pickable instance and the control signals based on the imaging signals; and

transmitting the generated geometrical feature and generated control signals to the robot arm such that the end-tool pickups the object corresponding to the pickable instance from the bin using the end-tool.

15. The bin-picking system of claim 14 , wherein the depth images are acquired by a camera or sensor.

16. The bin-picking system of claim 14 , wherein the steps further comprises:

selecting a next tallest point from the depth images such that a pick radius around the next tallest point excludes an overlap with the pickable instance;

defining a next region using the shape such that the next region surrounds the next tallest point;

sampling next points in the next region of the depth image;

computing next depth-geodesics between the next tallest point and the sampled next points; and

submitting the depth-geodesics to the neural network to segment a next pickable instance among the instances of the objects in the depth image.

17. The bin-picking system of claim 16 , wherein the steps of selecting, defining, the sampling, computing and submitting are continued until the steps are performed to a rest of the objects in the depth image.

18. The bin-picking system of claim 14 , wherein the shape of the region is a square, a rectangular, a triable, a circle or an oval.

19. The bin-picking system of claim 14 , wherein the neural network is trained to classify each feature vector as belonging to an identical instance or different instance.

20. The bin-picking system of claim 14 , wherein end-points of the geodesics are initialized using peaks produced by a Watershed Algorithm.

Continuity (2)

References Cited (19)