Systems and methods for robotic picking
Various embodiments of the present technology generally relate to robotic devices and artificial intelligence. More specifically, some embodiments relate to a robotic device for picking items from a bin and perturbing items in a bin. In some implementations, the device may include one or more computer-vision systems. A computer-vision system, in accordance with the present technology, may use at least two two-dimensional images to generate three-dimensional (3D) information about the bin and items in the bin. Based on the 3D information, a strategy for picking up items from the bin is determined. When no strategies with high probability of success exist, the robotic device may perturb the contents of the bin to create new available pick-up points and re-attempt to pick up an item.
1. A method of operating an apparatus for picking objects from a bin, the method comprising:
collecting image data from a computer-vision system, wherein the image data comprises information related to one or more objects in the bin;
identifying a plurality of distinct objects in the bin based on the image data;
providing at least a portion of the image data to one or more machine learning algorithms trained to identify picking strategies, wherein the picking strategies identify an order of the plurality of distinct objects to attempt to pick up and a location on each object of the plurality of distinct objects at which to grasp each object;
attempting to pick up, with a picking mechanism, a first object of the plurality of distinct objects from the bin at a first location on the first object according to a picking strategy of the picking strategies identified by the one or more machine learning algorithms;
in response to determining that the first object was not successfully picked up, repositioning the first object in the bin such that a position of the first object changes;
after repositioning the first object, providing new image data to the one or more machine learning algorithms, wherein the new image data comprises new information related to the one or more objects in the bin; and
attempting to pick up, with the picking mechanism, the first object from the bin at a second location on the first object according to a new picking strategy identified by the one or more machine learning algorithms.
2. The method of claim 1 , wherein repositioning the first object comprises repositioning the first object with a perturbation mechanism comprising at least one pneumatic air valve that blows compressed air into the bin to reposition the first object.
3. The method of claim 1 , wherein repositioning the first object comprises repositioning the first object with a perturbation element that physically contacts the first object to change the position of the first object via pushing.
4. The method of claim 1 , wherein the picking mechanism comprises at least one of: a suction mechanism, a gripping mechanism, a robotic hand, and a magnet.
5. The method of claim 1 , further comprising, in response to determining that the first object was successfully picked up, placing the first object in a new location, wherein the new location is external to the bin.
6. The method of claim 1 , wherein the apparatus comprises the computer-vision system and wherein the computer-vision system uses at least one camera and at least one neural network to generate a depth map of a region inside the bin that is used for identifying the one or more distinct objects in the bin.
7. The method of claim 1 , wherein the one or more machine learning algorithms comprise one or more artificial neural networks.
8. An apparatus for the picking and perturbation of objects, the apparatus comprising:
a picking element, wherein the picking element is configured to pick up objects from an area;
a perturbation element, wherein the perturbation element is configured to reposition the objects within the area;
one or more cameras, wherein the one or more cameras collect three-dimensional image data to use for operating the picking element and the perturbation element; and
a computing system comprising one or more processors and one or more memories operably coupled to the one or more processors, the one or more memories having stored thereon software instructions that, upon execution by the one or more processors, cause the one or more processors to:
obtain one or more images from the one or more cameras, wherein the images comprise information related to the objects within the area;
identify a plurality of distinct objects in the area based on the three-dimensional image data;
provide at least a portion of the images to one or more machine learning algorithms trained to identify picking strategies, wherein the picking strategies identify an order of the plurality of distinct objects to attempt to pick up and a location on each object of the plurality of distinct objects at which to grasp each object;
direct the picking element to attempt to pick up an object of the plurality of distinct objects at a first location on the object according to a picking strategy of the picking strategies identified by the one or more machine learning algorithms;
in response to determining that the object was not successfully picked up, direct the perturbation element to reposition the object such that a position of the object changes;
after repositioning the object, providing new image data to the one or more machine learning algorithms, wherein the new image data comprises new information related to the objects in the area; and
direct the picking element to pick up the object at a second location on the object according to a different picking strategy identified by the one or more machine learning algorithms.
9. The apparatus of claim 8 , wherein the picking element is a vacuum device that uses suction to pick up objects.
10. The apparatus of claim 8 , wherein the perturbation element is a compressed air valve configured to reposition the objects within the area by blowing compressed air into the area.
11. The apparatus of claim 8 , wherein a single component of the apparatus comprises the picking element and the perturbation element.
12. The apparatus of claim 8 , wherein the picking element and the perturbation element are coupled to a robotic arm controlled by the computing system.
13. The apparatus of claim 12 , wherein the computing system operates the robotic arm to place the object in a new location external to the area.
14. The apparatus of claim 8 , wherein the picking element comprises at least one of: a suction mechanism, a gripping mechanism, a robotic, hand, and a magnet.
15. A robotic system comprising:
a picking element configured to pick up items in a region;
one or more cameras configured to collect visual information about the region; and
a computing system comprising one or more processors and one or more memories operably coupled to the one or more processors, the one or more memories having stored thereon software instructions that, upon execution by the one or more processors, cause the one or more processors to:
identify a plurality of distinct items in the region based on the visual information collected by the one or more cameras;
provide at least a portion of the visual information to one or more machine learning algorithms trained to identify picking strategies, wherein the picking strategies identify an order of the plurality of distinct items to attempt to pick up and a location on each item of the plurality of distinct items at which to grasp each item;
attempt to pick up an item of the plurality of distinct items by directing the picking element to pick up the item at a first location on the item using a picking strategy of the picking strategies identified by the one or more machine learning algorithms;
in response to determining that the item was not successfully picked up, repositioning the item by directing a perturbation element of the robotic system to change a position of the item;
after repositioning the item, providing new visual information to the one or more machine learning algorithms, wherein the new visual information comprises new information about the region; and
attempt to pick up the item by directing the picking element to pick up the item at a second location on the item using a different picking strategy identified by the one or more machine learning algorithms.
16. The robotic system of claim 15 , wherein the one or more machine learning algorithms identify the picking strategy based at least in part on an associated probability of successful pick up.
17. The robotic system of claim 15 , wherein the perturbation element changes the position of the item using at least one pneumatic are valve that blows compressed air into the region to reposition the item.
18. The robotic system of claim 15 , wherein the software instructions comprise further instructions that, upon execution by the one or more processors, cause the one or more processors to determine that the item should be repositioned based on a probability of successful pick up of the item in a current position of the item.
19. The robotic system of claim 15 , wherein the picking element comprises at least one of: a suction mechanism, a gripping mechanism, a robotic hand, and a magnet.
20. The robotic system of claim 15 , wherein the picking strategy further identifies an orientation of the picking element for picking up the item.