Method and system for manipulating a target item supported on a substantially horizontal support surface
A method and system for manipulating a target item supported on a substantially horizontal support surface supporting a configuration of substantially identical items are provided. The system includes at least one light source configured to illuminate at least one surface including a top surface of the items to obtain backscattered illumination. A volumetric sensor such as a 2-D/3-D sensor is configured to sense the backscattered illumination to obtain at least one depth image and at least one appearance image. At least one processor is configured to determine geometric information and appearance information from the images. The at least one processor is also configured to generate a plurality of hypotheses and surprisals of the information and to rank the hypotheses based on their surprisals. In one embodiment, the best hypothesis is selected and the item is manipulated or picked from a pallet by a vision-guided robot based on the best hypothesis.
1. A method of manipulating a target item supported on a substantially horizontal support surface supporting a configuration of substantially identical items aligned substantially perpendicular to the support surface, the method comprising the steps of:
gathering information about the configuration of items within a scene wherein the step of gathering includes the steps of illuminating at least one surface including a top surface of the items in the scene and sensing the irradiance of the scene to obtain at least one image of the geometry of the scene and at least one image of the appearance of the scene;
generating a plurality of hypotheses, each of the hypotheses describing an observation of an item in the configuration, the observations including an observation of the appearance of the item and an observation of the geometry of the item; and
ranking the hypotheses wherein the step of ranking includes the steps of computing a surprisal of the observation of the appearance of the item and a surprisal of the observation of the geometry of the item and combining the surprisals for each hypothesis.
2. The method as claimed in claim 1 , wherein the step of illuminating the at least one surface illuminates one or more side surfaces of the target item to obtain geometry and appearance images of the one or more side surfaces.
3. The method as claimed in claim 1 , wherein the step of illuminating includes the step of uniformly illuminating the at least one surface including the top surface of the item with light having an intensity within a relatively narrow range of wavelengths such that the light overwhelms the intensity of ambient light within the narrow range.
4. The method as claimed in claim 1 , wherein the at least one image of the appearance of the scene is determined from an albedo of the target item.
5. The method as claimed in claim 1 , wherein the step of sensing is performed with a hybrid 2-D/3-D sensor.
6. The method as claimed in claim 1 , wherein the target item is a box-like object such as a box or a large bag.
7. The method as claimed in claim 1 , wherein a pallet has the support surface.
8. The method as claimed in claim 1 , wherein the step of generating includes the step of associating each element of a 3-dimensional matrix with a hypothesis regarding the presence of an item at the horizontal and vertical position, and an orientation angle determined by the coordinates of the element in the matrix.
9. The method as claimed in claim 1 wherein the plurality of hypotheses describes the observation of a printed pattern atop the item.
10. The method as claimed in claim 1 , further comprising the step of selecting the hypothesis with the largest surprisal.
11. The method as claimed in claim 1 , further comprising determining printed pattern information which is based on a printed pattern in the at least one appearance image and located on the top or side surfaces of the target item.
12. The method as claimed in claim 2 , wherein the plurality of hypotheses describes the observation of the appearance of a vertical boundary of the target item.
13. The method as claimed in claim 2 , wherein the plurality of hypotheses describes the observation of the geometry of a vertical boundary of the target item.
14. The method as claimed in claim 3 , wherein the step of illuminating includes the step of polarizing the light and wherein the step of sensing senses unpolarized light and rejects the polarized light to obtain images of backscattered light.
15. The method as claimed in claim 9 , wherein the step of ranking includes the steps of computing the surprisal of the observation of the appearance of the printed pattern atop the target item and combining the surprisal to the combined surprisals for the hypothesis.
16. The method as claimed in claim 10 , wherein the step of selecting includes the steps of determining the location and orientation of the target item to be manipulated based on the selected hypothesis, and wherein the method further comprises the steps of providing an autonomous manipulator and a manipulating tool attached to the autonomous manipulator and utilizing the autonomous manipulator and the manipulating tool to manipulate the selected item based on the ranked hypotheses.
17. The method as claimed in claim 11 , wherein at least one of the hypotheses is based on the printed pattern and wherein the step of ranking includes calculating printed pattern surprisals from the printed pattern information.
18. The method as claimed in claim 12 , wherein the step of ranking includes the steps of computing the surprisal of the observation of the appearance of the vertical boundary of the target item and combining the surprisal to the combined surprisals for the hypothesis.
19. The method as claimed in claim 13 , wherein the step of ranking includes the steps of computing the surprisal of the observation of the geometry of the vertical boundary of the target item and adding the surprisal to the combined surprisals for the hypothesis.
20. The method as claimed in claim 16 , wherein the autonomous manipulator is vision-guided robot.
21. A system for manipulating a target item supported on a substantially horizontal support structure, the system comprising:
at least one light source configured to illuminate at least one surface including a top surface of the items in a scene to obtain backscattered illumination;
a volumetric sensor having a field of view and configured to sense the backscattered illumination to obtain at least one depth image and at least one appearance image of the item in the scene; and
at least one processor configured to determine depth information which is based on changes in depth in the at least one depth image and configured to determine grayscale information which is based on changes in brightness in the at least one grayscale image wherein the depth information and the grayscale information comprise first and second information sources respectively, and wherein for each possible voxel location in the depth and grayscale images and each possible item orientation, the at least one processor being configured to generate a hypothesis that the item appears at that voxel location with that item orientation to obtain a plurality of hypotheses and to rank the plurality of hypotheses; the at least one processor being configured to calculate surprisals from the depth and grayscale information for each of the hypotheses to obtain a plurality of surprisals from each information source; and the at least one processor being configured to determine location of the item to be manipulated based on the ranked hypotheses.
22. The system as claimed in claim 21 , wherein the at least one light source is configured to illuminate one or more side surfaces of the item to obtain depth and grayscale images of the one or more side surfaces.
23. The system as claimed in claim 21 , wherein the item is a box-like object such as a box or large bag.
24. The system as claimed in claim 21 , wherein the at least one light source is configured to uniformly illuminate at least one surface including the top surface of the item with light having an intensity within a relatively narrow range of wavelengths such that the light overwhelms the intensity of ambient light within the narrow range to obtain the backscattered illumination.
25. The system as claimed in claim 21 , wherein the at least one light source is a polarized light source and wherein the sensor includes a polarization analyzer configured to reject polarized light and to pass unpolarized light.
26. The system as claimed in claim 21 , further comprising an autonomous manipulator registered with the sensor and a manipulating tool attached to the autonomous manipulator to manipulate the item based on the ranked hypotheses.
27. The system as claimed in claim 21 , wherein the sensor is a hybrid 2-D/3-D sensor.
28. The system as claimed in claim 21 , wherein the at least one processor is configured to compute rotation and position invariant albedo of the item based on the images wherein the brightness is the intrinsic brightness of the item.
29. The system as claimed in claim 21 , wherein a pallet has the support surface.
30. The system as claimed in claim 22 , wherein the at least one processor is configured to determine printed pattern information which is based on a printed pattern in the at least one grayscale image and located on the top or side surfaces of the item.
31. The system as claimed in claim 26 , wherein the autonomous manipulator is a vision-guided robot.
32. The system as claimed in claim 30 , wherein at least one of the hypotheses is based on the printed pattern and wherein the at least one processor is configured to calculate a printed pattern surprisal from the at least one hypothesis.