Detecting boxes
A method for detecting boxes includes receiving a plurality of image frame pairs for an area of interest including at least one target box. Each image frame pair includes a monocular image frame and a respective depth image frame. For each image frame pair, the method includes determining corners for a rectangle associated with the at least one target box within the respective monocular image frame. Based on the determined corners, the method includes the following: performing edge detection and determining faces within the respective monocular image frame; and extracting planes corresponding to the at least one target box from the respective depth image frame. The method includes matching the determined faces to the extracted planes and generating a box estimation based on the determined corners, the performed edge detection, and the matched faces of the at least one target box.
1. A robot comprising:
an arm comprising an end effector;
a mobile base, wherein the mobile base and the arm are independently moveable;
a plurality of sensors comprising a first sensor and a second sensor; and
a computing system in communication with the plurality of sensors, wherein the computing system is configured to:
obtain, from the first sensor, first sensor data reflecting a package, wherein the first sensor data comprises two-dimensional sensor data;
obtain, from the second sensor, second sensor data reflecting the package,
wherein the second sensor data comprises three-dimensional sensor data;
detect the package using the first sensor data and the second sensor data by:
identifying, using the first sensor data, one or more two-dimensional coordinate locations of one or more corners of the package; and
determining, using the first sensor data and/or the second sensor data,
a face corresponding to the package based on the one or more two-dimensional coordinate locations of the one or more corners of the package; and
instruct the end effector to interact with the face of the package based on detecting the package.
2. The robot of claim 1 , wherein the two-dimensional sensor data comprises a monocular image and the three-dimensional sensor data comprises a depth image.
3. The robot of claim 1 , wherein the two-dimensional sensor data and the three-dimensional sensor data are associated with a same time stamp.
4. The robot of claim 1 , wherein to detect the package, the computing system is further configured to detect a location of the package using the first sensor data and the second sensor data.
5. The robot of claim 1 , wherein one or more of the first sensor or the second sensor are mounted on the arm.
6. The robot of claim 1 , wherein the computing system is further configured to:
determine the face corresponding to the package from the first sensor data; and
determine a plane corresponding to the package from the second sensor data.
7. The robot of claim 1 , wherein, to detect the package, the computing system is further configured to:
determine the face corresponding to the package from the first sensor data;
match the face to a plane determined from the second sensor data;
generate an estimation for the package using the face and the plane; and
derive a location of the package from the estimation.
8. The robot of claim 1 , wherein, to detect the package, the computing system is further configured to:
match the face corresponding to the package to a plane determined from the second sensor data;
generate an estimation for the package using the face and the plane; and
derive a location of the package from the estimation.
9. The robot of claim 1 , wherein the computing system comprises a machine learning model.
10. The robot of claim 1 , wherein the computing system comprises a machine learning model configured to receive a respective pair of images as input and to predict a location of the package as output.
11. The robot of claim 1 , further comprising a body of the robot.
12. The robot of claim 1 , wherein the package comprises a box.
13. The robot of claim 1 , wherein arm comprises an articulated arm.
14. The robot of claim 1 , wherein the computing system comprises an image processing system.
15. The robot of claim 1 , wherein the end effector comprises a plurality of suction cups to interact with the package.
16. The robot of claim 1 , wherein the robot further comprises at least one wheel.
17. The robot of claim 1 , wherein the robot further comprises at least one wheel, wherein the computing system is further configured to drive the robot across a surface to deliver the package to a destination.
18. A computer-implemented method comprising:
obtaining, by data processing hardware, from a first sensor of a plurality of sensors of a robot, first sensor data reflecting a package, wherein the first sensor data comprises two-dimensional sensor data, the robot comprising an arm, a mobile base, and the plurality of sensors, the arm comprising an end effector, the plurality of sensors comprising the first sensor and a second sensor, wherein the mobile base and the arm are independently moveable;
obtaining, by the data processing hardware, from the second sensor, second sensor data reflecting the package, wherein the second sensor data comprises three-dimensional sensor data;
detecting, by the data processing hardware, the package using the first sensor data and the second sensor data by:
identifying, using the first sensor data, one or more two-dimensional coordinate locations of one or more corners of the package; and
determining, using the first sensor data and/or the second sensor data, a face corresponding to the package based on the one or more two-dimensional coordinate locations of the one or more corners of the package; and
instructing, by the data processing hardware, the end effector to interact with the face corresponding to the package based on detecting the package.
19. The computer-implemented method of claim 18 , wherein detecting the package comprises:
determining the face corresponding to the package from the first sensor data;
matching the face to a plane determined from the second sensor data;
generating an estimation for the package using the face and the plane; and
deriving a location of the package from the estimation.