Locating, separating, and picking boxes with a sensor-guided robot
Techniques are described that enable robotic picking systems to locate and pick boxes from an unstructured pallet using computer vision and/or one or more “exploratory picks” to determine the sizes and locations of boxes on the pallet.
1. A computer-implemented method, comprising:
capturing a first image of a loaded pallet, the loaded pallet having a plurality of boxes stacked thereon;
generating a first representation of a surface of the loaded pallet using the first image;
positioning a robotic arm relative to a corner of the surface of the loaded pallet based on a minimum box size for the boxes on the loaded pallet;
moving a first box at the corner of the surface of the loaded pallet a programmable amount using the robotic arm;
capturing a second image of the loaded pallet including the first box as moved by the robotic arm;
generating a second representation of the surface of the loaded pallet using the second image;
releasing the first box with the robotic arm;
processing the first and second representations of the surface of the loaded pallet to generate a representation of a surface of the first box;
positioning the robotic arm relative to the first box using the representation of the surface of the first box; and
removing the first box from the loaded pallet using the robotic arm.
2. The method of claim 1 , wherein processing the first and second representations of the surface of the loaded pallet includes determining a difference between the first and second representations of the surface of the loaded pallet, and determining a size and a position of the surface of the first box using the difference.
3. The method of claim 1 , wherein the second image includes a representation of the robotic arm obscuring a portion of the surface of the first box, and wherein processing the first and second representations of the surface of the loaded pallet to generate a representation of the surface of the first box includes using data from both of the first and second representations of the surface of the loaded pallet to generate the representation of the surface of the first box.
4. The method of claim 1 , further comprising receiving input representing the minimum box size for the boxes on the loaded pallet, wherein positioning the robotic arm relative to the first box using the representation of the surface of the first box includes using the minimum box size.
5. The method of claim 1 , further comprising receiving input representing a maximum box size for the boxes on the loaded pallet, wherein removing the first box from the loaded pallet using the robotic arm includes (1) lifting the first box above a top surface of the loaded pallet by at least a height associated with the maximum box size, or (2) moving the first box away from a side surface of the loaded pallet by at least a distance associated with the maximum box size.
6. The method of claim 1 , wherein removing the first box from the loaded pallet using the robotic arm is accomplished without a priori knowledge of the number, sizes, orientations, and locations of the boxes on the loaded pallet.
7. The method of claim 1 , wherein the first and second images are captured using a camera, the method further comprising determining a location of the camera using a plurality of images of a fiducial held by the robotic arm in a corresponding plurality of positions.
8. The method of claim 1 , further comprising determining a location of the loaded pallet using an image of a fiducial on an empty pallet.
9. The method of claim 1 , further comprising unloading all of the boxes from the loaded pallet while performing the capturing, generating, lifting, capturing, generating, releasing, and processing for fewer than all of the boxes.
10. The method of claim 1 , wherein moving the first box comprises one of (1) lifting the first box the programmable amount relative to the loaded pallet using the robotic arm, (2) moving the first box laterally relative to the loaded pallet using the robotic arm, or (3) tilting the first box relative to the surface of the loaded pallet using the robotic arm.
11. The method of claim 1 , further comprising:
storing the representation of the surface of the first box;
capturing a third image of the loaded pallet using the image capture device, the third image not including the first box;
generating a third representation of the surface of the loaded pallet using the third image;
identifying one or more instances of a first box type corresponding to the first box by comparing the third representation of the surface of the loaded pallet and the stored representation of the surface of the first box, including determining a location and orientation of each of the instances of the first box type; and
picking each of the instances of the first box type using the corresponding location and orientation.
12. The method of claim 1 , further comprising:
storing the representation of the surface of the first box;
capturing an image of a different loaded pallet using the image capture device;
generating a representation of a surface of the different loaded pallet using the image;
identifying one or more instances of a first box type corresponding to the first box by comparing the representation of the surface of the different loaded pallet and the stored representation of the surface of the first box, including determining a location and orientation of each of the instances of the first box type; and
picking each of the instances of the first box type using the robotic arm and the corresponding location and orientation of each instance.
13. The method of claim 1 , wherein the first and second representations of the surface of the loaded pallet are three-dimensional representations.
14. The method of claim 1 , wherein moving the first box the programmable amount involves separating the first box from a top layer of the loaded pallet.
15. A system, comprising a computing device, a robotic arm, a robotic control system, and an image capture device, the computing device being configured to:
capture a first image of a loaded pallet using the image capture device, the loaded pallet having a plurality of boxes stacked thereon;
generate a first representation of a surface of the loaded pallet using the first image;
position the robotic arm relative to a corner of the surface of the loaded pallet based on a minimum box size for the boxes on the loaded pallet;
move a first box at the corner of the surface of the loaded pallet a programmable amount using the robotic arm via the robotic control system;
capture a second image of the loaded pallet using the image capture device, the second image including the first box as moved by the robotic arm;
generate a second representation of the surface of the loaded pallet using the second image;
release the first box using the robotic arm via the robotic control system;
process the first and second representations of the surface of the loaded pallet to generate a representation of a surface of the first box;
position the robotic arm relative to the first box via the robotic control system using the representation of the surface of the first box; and
remove the first box from the loaded pallet using the robotic arm via the robotic control system.
16. The system of claim 15 , wherein the computing device is configured to process the first and second representations of the surface of the loaded pallet by determining a difference between the first and second representations of the surface of the loaded pallet, and determining a size and a position of the surface of the first box using the difference.
17. The system of claim 15 , wherein the second image includes a representation of the robotic arm obscuring a portion of the surface of the first box, and wherein the computing device is configured to process the first and second representations of the surface of the loaded pallet to generate a representation of the surface of the first box by using data from both of the first and second representations of the surface of the loaded pallet to generate the representation of the surface of the first box.
18. The system of claim 15 , wherein the computing device is further configured to receive input representing the minimum box size for the boxes on the loaded pallet, and wherein the computing device is configured to position the robotic arm relative to the first box using the representation of the surface of the first box by using the minimum box size.
19. The system of claim 15 , wherein the computing device is further configured to receive input representing a maximum box size for the boxes on the loaded pallet, and wherein the computing device is configured to remove the first box from the loaded pallet using the robotic arm by (1) lifting the first box above a top surface of the loaded pallet by at least a height associated with the maximum box size, or (2) moving the first box away from a side surface of the loaded pallet by at least a distance associated with the maximum box size.
20. The system of claim 15 , wherein the computing device is configured to remove the first box from the loaded pallet using the robotic arm without a priori knowledge of the number, sizes, orientations, and locations of the boxes on the loaded pallet.
21. The system of claim 15 , wherein the computing device is further configured to determine a location of the image capture device using a plurality of images of a fiducial held by the robotic arm in a corresponding plurality of positions.
22. The system of claim 15 , wherein the computing device is further configured to determine a location of the loaded pallet using an image of a fiducial on an empty pallet.
23. The system of claim 15 , wherein the computing device is further configured to unload all of the boxes from the loaded pallet using the robotic arm while performing the capturing, generating, lifting, capturing, generating, releasing, and processing for fewer than all of the boxes.
24. The system of claim 15 , wherein the computing device is further configured to move the first box by one of (1) lifting the first box the programmable amount relative to the loaded pallet using the robotic arm, (2) moving the first box laterally relative to the loaded pallet using the robotic arm, or (3) tilting the first box relative to the surface of the loaded pallet using the robotic arm.
25. The system of claim 15 , wherein the computing device is further configured to:
store the representation of the surface of the first box;
capture a third image of the loaded pallet using the image capture device, the third image not including the first box;
generate a third representation of the surface of the loaded pallet using the third image;
identify one or more instances of a first box type corresponding to the first box by comparing the third representation of the surface of the loaded pallet and the stored representation of the surface of the first box, including determining a location and orientation of each of the instances of the first box type; and
pick each of the instances of the first box type using the robotic arm and the corresponding location and orientation of each instance.
26. The system of claim 15 , wherein the computing device is further configured to:
store the representation of the surface of the first box;
capture an image of a different loaded pallet using the image capture device;
generate a representation of a surface of the different loaded pallet using the image;
identify one or more instances of a first box type corresponding to the first box by comparing the representation of the surface of the different loaded pallet and the stored representation of the surface of the first box, including determining a location and orientation of each of the instances of the first box type; and
pick each of the instances of the first box type using the robotic arm and the corresponding location and orientation of each instance.
27. The system of claim 15 , wherein the first and second representations of the surface of the loaded pallet are three-dimensional representations.
28. The system of claim 15 , wherein the computing device is configured to move the first box the programmable amount by separating the first box from a top layer of the loaded pallet.
29. A computer program product, comprising one or more computer-readable media having computer program instructions stored therein, the computer program instructions being configured such that, when executed by one or more computing devices, the computer program instructions cause the one or more computing devices to:
capture a first image of a loaded pallet, the loaded pallet having a plurality of boxes stacked thereon;
generate a first representation of a surface of the loaded pallet using the first image;
position a robotic arm relative to a corner of the surface of the loaded pallet based on a minimum box size for the boxes on the loaded pallet;
move a first box at the corner of the surface of the loaded pallet a programmable amount using the robotic arm;
capture a second image of the loaded pallet including the first box as moved by the robotic arm;
generate a second representation of the surface of the loaded pallet using the second image;
release the first box using the robotic arm;
process the first and second representations of the surface of the loaded pallet to generate a representation of a surface of the first box;
position the robotic arm relative to the first box using the representation of the surface of the first box; and
remove the first box from the loaded pallet using the robotic arm.