Systems and methods for a vision guided end effector

1. A computer-implemented method for picking an object from a plurality of objects by a robot having an end effector, the method comprising:

obtaining an image of a scene containing the plurality of objects;

generating a segmentation map for the plurality of objects in the scene;

determining shapes of the plurality of objects based on the segmentation map including obtaining, for each of one or more objects of the plurality of objects in the segmentation map, a respective 3D CAD model of the object and generating a respective shape of the object from the 3D CAD model of the object;

adjusting the end effector including shaping the end effector according to a shape belonging to an object of the plurality of objects;

approaching the plurality of objects; and

picking the object of the plurality of objects with the end effector adjusted according to the shape of the object.

2. The method of claim 1 , wherein shaping the end effector includes moving a portion of the end effector from a first state to a second state, wherein the first state is an equilibrium state, and the second state is a non-equilibrium state.

3. The method of claim 2 , wherein in the second state, the portion of the end effector retracts by an amount determined by one of the shapes.

4. The method of claim 1 , wherein shaping the end effector includes:

predicting a shape of the end effector that will provide an optimal grasp of the one of the plurality of objects, wherein shaping the end effector is based on the predicted shape.

5. The method of claim 1 , wherein the end effector is at least one of a pin array, a tube, or a suction cup.

6. The method of claim 1 , further comprising:

identifying a grasp point on the one of the plurality of objects, wherein shaping the end effector is based on identifying the grasp point.

7. The method of claim 1 further comprising:

in response to approaching the plurality of objects, re-shaping the end effector based on determining a second shape.

8. The method of claim 1 further comprising: determining poses of the plurality of the objects in the scene, wherein determining the shapes is based on determining the poses.

9. A system for picking an object from a plurality of objects with a robot having an end effector, the system comprising:

one or more cameras for obtaining an image of a scene containing the plurality of objects;

a processing system coupled to the one or more cameras, the processing system comprising one or more electronic circuits and memory storing instructions that, when executed by the processing system, cause the processing system to perform operations comprising:

generating a segmentation map for the plurality of objects in the scene;

determining shapes of the plurality of objects based on the segmentation map including obtaining, for each of one or more objects of the plurality of objects in the segmentation map, a respective 3D CAD model of the object and generating a respective shape of the object from the 3D CAD model of the object;

adjusting the end effector including shaping the end effector according to a shape belonging to an object of the plurality of objects;

approaching the plurality of objects; and

picking the object of the plurality of objects with the end effector adjusted according to the shape of the object.

10. The system of claim 9 , wherein the shaping of end effector includes moving a portion of the end effector from a first state to a second state, wherein the first state is an equilibrium state, and the second state is a non-equilibrium state.

11. The system of claim 10 , wherein in the second state, the portion of the end effector is configured to retract by an amount determined by one of the shapes.

12. The system of claim 9 , wherein shaping the end effector includes:

predicting a shape of the end effector that will provide an optimal grasp of the one of the plurality of objects, wherein shaping the end effector is based on predicted shape.

13. The system of claim 9 , wherein the end effector is at least one of a pin array, a tube, or a suction cup.

14. The system of claim 9 , wherein the operations further comprise:

identifying a grasp point on the one of the plurality of objects, wherein shaping the end effector is based on identifying the grasp point.

15. The system of claim 9 , wherein the operations further comprise:

in response to approaching the plurality of objects, re-shaping the end effector based on determining a second shape.

16. The system of claim 9 , wherein the operations further comprise: determining poses of the plurality of the objects in the scene, wherein determining the shapes is based on determining the poses.

17. Memory storing instructions that when executed by a computer system comprising one or more electronic circuits cause the computer system to perform operations using a robot having an end effector, the operations comprising:

obtaining an image of a scene containing a plurality of objects;

generating a segmentation map for the plurality of objects in the scene;

determining shapes of the plurality of objects based on the segmentation map including obtaining, for each of one or more objects of the plurality of objects in the segmentation map, a respective 3D CAD model of the object and generating a respective shape of the object from the 3D CAD model of the object;

adjusting the end effector including shaping the end effector according to a shape belonging to an object of the plurality of objects;

approaching the plurality of objects; and

picking the object of the plurality of objects with the end effector adjusted according to the shape of the object.