Method and system for handling deformable objects

1. A system for handling one or more deformable objects that are arranged in a stack, the system comprising:

a dual-arm robotic manipulator comprising:

first and second robotic arms; and

first and second end effectors connected to the first and second robotic arms, respectively;

a set of image sensors configured to capture a first set of images of the stack; and

a control server in communication with the dual-arm robotic manipulator and the set of image sensors, the control server configured to:

detect the one or more deformable objects arranged in the stack based on the first set of images captured by the set of image sensors;

determine a contour of the detected one or more deformable objects based on the first set of images;

determine, based on the contour, a sequence of a plurality of actions to be performed by the dual-arm robotic manipulator for handling a first deformable object in the stack, wherein the handling includes a pick operation to be executed on the first deformable object; and

control, in the pick operation based on the determined sequence of the plurality of actions:

the first robotic arm to grip the first deformable object from a gripping end by way of the first end effector, and lift the gripped end to a predetermined height to partially lift the first deformable object;

the second robotic arm to slide the second end effector beneath the partially lifted first deformable object, and lift the first deformable object in entirety; and

the first robotic arm to release the grip of the first end effector on the first deformable object for successfully executing the pick operation.

2. The system of claim 1 , wherein the handling further includes a put-down operation to be executed on the successfully picked first deformable object, and wherein, in the put-down operation, the control server is further configured to control, based on the determined sequence of the plurality of actions, the second robotic arm to put-down the successfully picked first deformable object at a destination location.

3. The system of claim 2 , further comprising a database associated with the control server, wherein the control server is further configured to store, upon successful handling of the first deformable object, plan information associated with the sequence of the plurality of actions in the database.

4. The system of claim 1 , wherein the sequence of the plurality of actions to be performed by the dual-arm robotic manipulator is further determined based on historical data associated with the one or more deformable objects, wherein the historical data includes at least one of a set of physical attributes of each of the one or more deformable objects or information associated with previous handling of the one or more deformable objects, and wherein the set of physical attributes of each deformable object includes at least one of a shape, a size, a weight, a set of dimensions, a number of folds, or actual depth information, of the corresponding deformable object.

5. The system of claim 4 , wherein the control server is further configured to:

identify one or more layers in the contour, wherein a first layer in the contour corresponds to a fold of the first deformable object or the first deformable object in entirety;

identify, based on the contour, perceived depth information of each of the one or more deformable objects; and

determine an orientation of the first deformable object with respect to the stack based on the perceived depth information of each of the one or more deformable objects and the historical data, wherein the sequence of the plurality of actions to be performed by the dual-arm robotic manipulator is further determined based on the one or more layers in the contour and the orientation of the first deformable object.

6. The system of claim 5 , wherein the control server is further configured to identify the gripping end based on the contour, the one or more layers, the orientation of the first deformable object, and the historical data.

7. The system of claim 1 , further comprising:

a storage unit having a plurality of shelves, wherein the one or more deformable objects are arranged in the stack on a first shelf of the plurality of shelves; and

a mobile robot that is configured to transport to the storage unit from a first location to a second location that is within an operational range of the dual-arm robotic manipulator.

8. The system of claim 1 , wherein the set of image sensors is further configured to capture a second set of images of the stack, while the first robotic arm lifts the gripping end of the first deformable object, and wherein the control server is further configured to determine, based on the second set of images, whether the gripping end of the first deformable object is lifted to the predetermined height.

9. The system of claim 1 , wherein the second end effector comprises a set of pressure sensors that is configured to record, when the second end effector lifts the first deformable object, pressure data corresponding to pressure exerted by the lifted first deformable object on the second end effector, wherein the control server is further configured to determine whether the second end effector has accurately lifted the first deformable object based on the recorded pressure data, and wherein the accurate lifting of the first deformable object corresponds to a uniform weight distribution of the first deformable object on the second end effector.

10. The system of claim 1 , wherein the first end effector is a vacuum gripper, and wherein the second end effector includes a spatula-shaped base, an adjustable arm connected to the spatula-shaped base, and a roller connected to the adjustable arm such that the roller is oriented parallel to and at an adjustable height above the spatula-shaped base.

11. A method for handling one or more deformable objects that are arranged in a stack, the method comprising:

detecting, by a control server, the one or more deformable objects arranged in the stack based on a first set of images of the stack captured by a set of image sensors;

determining, by the control server, a contour of the detected one or more deformable objects based on the first set of images;

determining, by the control server, based on the contour, a sequence of a plurality of actions to be performed by a dual-arm robotic manipulator for handling a first deformable object in the stack, wherein the handling includes a pick operation to be executed on the first deformable object; and

controlling, by the control server in the pick operation based on the determined sequence of the plurality of actions:

a first robotic arm of the dual-arm robotic manipulator to grip the first deformable object from a gripping end by way of a first end effector connected to the first robotic arm, and lift the gripped end to a predetermined height to partially lift the first deformable object;

a second robotic arm of the dual-arm robotic manipulator to slide a second end effector connected to the second robotic arm beneath the partially lifted first deformable object, and lift the first deformable object in entirety; and

the first robotic arm to release the grip of the first end effector on the first deformable object for successfully executing the pick operation.

12. The method of claim 11 , wherein the handling further includes a put-down operation to be executed on the successfully picked first deformable object, and wherein, in the put-down operation based on the determined sequence of the plurality of actions, the second robotic arm is further controlled by the control server to place the successfully picked first deformable object at a destination location.

13. The method of claim 12 , further comprising storing, by the control server upon successful handling of the first deformable object, plan information associated with the sequence of the plurality of actions in a database associated with the control server.

14. The method of claim 11 , wherein the sequence of the plurality of actions to be performed by the dual-arm robotic manipulator is further determined based on historical data associated with the one or more deformable objects, wherein the historical data includes at least one of a set of physical attributes of each of the one or more deformable objects or information associated with previous handling of the one or more deformable objects, and wherein the set of physical attributes of each deformable object includes at least one of a shape, a size, a weight, a set of dimensions, a number of folds, or actual depth information, of the corresponding deformable object.

15. The method of claim 14 , further comprising:

identifying, by the control server, one or more layers in the contour, wherein a first layer in the contour corresponds to a fold of the first deformable object or the first deformable object in entirety;

identifying, by the control server based on the contour, perceived depth information of each of the one or more deformable objects; and

determining, by the control server, an orientation of the first deformable object with respect to the stack based on the perceived depth information of each of the one or more deformable objects and the historical data, wherein the sequence of the plurality of actions to be performed by the dual-arm robotic manipulator is further determined based on the one or more layers in the contour and the orientation of the first deformable object.

16. The method of claim 15 , further comprising identifying, by the control server, the gripping end based on the contour, the one or more layers, the orientation of the first deformable object, and the historical data.

17. The method of claim 11 , further comprising determining, by the control server, whether the gripping end of the first deformable object is lifted to the predetermined height based on a second set of images of the stack captured by the set of image sensors, wherein the second set of images of the stack is captured by the set of image sensors while the first robotic arm lifts the gripping end of the first deformable object.

18. The method of claim 11 , further comprising determining, by the control server, whether the second end effector has accurately lifted the first deformable object based on pressure data recorded by a set of pressure sensors installed on the second end effector, wherein the pressure data corresponds to pressure exerted by the first deformable object on the second end effector when the first deformable object is lifted by the second end effector, and wherein the accurate lifting of the first deformable object corresponds to a uniform weight distribution of the first deformable object on the second end effector.

19. A system for handling one or more deformable objects that are arranged in a stack, the system comprising:

a control server configured to:

detect the one or more deformable objects arranged in the stack based on a first set of images of the stack captured by a set of image sensors;

determine a contour of the detected one or more deformable objects based on the first set of images;

determine, based on the contour, a sequence of a plurality of actions to be performed by a dual-arm robotic manipulator for handling a first deformable object in the stack, wherein the handling includes a pick operation to be executed on the first deformable object; and

control, in the pick operation based on the determined sequence of the plurality of actions:

a first robotic arm of the dual-arm robotic manipulator to grip the first deformable object from a gripping end by way of a first end effector connected to the first robotic arm, and lift the gripped end to a predetermined height to partially lift the first deformable object;

a second robotic arm of the dual-arm robotic manipulator to slide a second end effector connected to the second robotic arm beneath the partially lifted first deformable object, and lift the first deformable object in entirety; and

the first robotic arm to release the grip of the first end effector on the first deformable object for successfully executing the pick operation.

20. The system of claim 19 , wherein the handling further includes a put-down operation to be executed on the successfully picked first deformable object, and wherein, in the put-down operation based on the determined sequence of the plurality of actions, the control server is further configured to control the second robotic arm to place the successfully picked first deformable object at a destination location.