Detecting boxes
A method for detecting boxes includes receiving a plurality of image frame pairs for an area of interest including at least one target box. Each image frame pair includes a monocular image frame and a respective depth image frame. For each image frame pair, the method includes determining corners for a rectangle associated with the at least one target box within the respective monocular image frame. Based on the determined corners, the method includes the following: performing edge detection and determining faces within the respective monocular image frame; and extracting planes corresponding to the at least one target box from the respective depth image frame. The method includes matching the determined faces to the extracted planes and generating a box estimation based on the determined corners, the performed edge detection, and the matched faces of the at least one target box.
1. A robot comprising:
a body having a first end and a second end;
an articulated arm disposed on the body at the first end and comprising an end effector;
a sensor system configured to capture images for a target box within a working environment of the robot;
an image processing system in communication with the sensor system and configured to detect a location of the target box using a pair of the images captured by the sensor system, the pair of the images associated with an identical time stamp and comprising a monocular image and a depth image;
a control system in communication with the image processing system, the control system configured to manipulate the end effector to engage with the target box at the detected location of the target box; and
at least one drive wheel coupled to the second end of the body and configured to drive the robot across a traction surface within the working environment of the robot while the end effector is engaged with a target box.
2. The robot of claim 1 , wherein the end effector comprises a plurality of suction cups to engage with the target box.
3. The robot of claim 1 , wherein the articulated arm has at least five degrees of freedom.
4. The robot of claim 1 , wherein the sensor system comprises one or more sensors mounted on the articulated arm of the robot, the one or more sensors capturing the monocular image and the depth image of the target box.
5. The robot of claim 1 , wherein the at least one drive wheel comprises a first drive wheel and a second drive wheel.
6. The robot of claim 5 , wherein the first drive wheel and the second drive wheel are independently rotatable.
7. The robot of claim 5 , wherein rotation of the first drive wheel in a first direction and rotation of the second drive wheel in a second direction opposite the first direction enables the robot to change orientation by swiveling on the traction surface.
8. The robot of claim 1 , wherein, to detect the location of the target box, the image processing system performs operations comprising:
receiving the monocular image and the depth image from the sensor system;
determining a face corresponding to the target box from the monocular image;
matching the face from the monocular image to a plane from the depth image;
generating a box estimation for the target box using the determined face from the monocular image and the plane that matches the determined face from the depth image; and
deriving the detected location of the target box from the box estimation generated for the target box.
9. The robot of claim 8 , wherein determining the face corresponding to the target box from the monocular image comprises identifying two-dimensional coordinate locations of corners for the target box from the monocular image.
10. The robot of claim 1 , wherein the image processing system comprises a machine learning model configured to receive a respective pair of images as input and to predict the location of the target box as output, the respective pair of images having a respective identical time stamp and comprising a respective monocular image and a respective depth image.
11. A computer-implemented method when executed by data processing hardware causes the data processing hardware to perform operations comprising:
receiving, at an image processing system of a wheel-based mobile robot, a pair of images of a target box captured by a sensor system of the wheel-based mobile robot, the pair of the images associated with an identical time stamp and comprising a monocular image and a depth image;
determining, by the image processing system, a face corresponding to the target box from the monocular image;
matching, by the image processing system, the face from the monocular image to a plane from the depth image;
generating, by the image processing system, a box estimation for the target box using the determined face from the monocular image and the plane that matches the determined face from the depth image; and
controlling an end effector of the wheel-based mobile robot to engage the target box at a location corresponding to the box estimation.
12. The method of claim 11 , wherein the operations further comprise actuating at least one drive wheel coupled to a body of the wheel-based mobile robot to drive the wheel-based mobile robot across a traction surface within a working environment of the wheel-based mobile robot while the end effector is engaged with the target box.
13. The method of claim 11 , wherein the end effector comprises a plurality of suction cups to engage with the target box.
14. The method of claim 11 , wherein an articulated arm of the wheel-based mobile robot comprises the end effector.
15. The method of claim 14 , wherein the articulated arm further comprises at least five degrees of freedom.
16. The method of claim 14 , wherein the sensor system comprises one or more sensors mounted on the articulated arm of the wheel-based mobile robot, the one or more sensors capturing the monocular image and the depth image of the target box.
17. The method of claim 11 , wherein the wheel-based mobile robot comprises a first drive wheel and a second drive wheel.
18. The method of claim 17 , wherein the first drive wheel and the second drive wheel are independently rotatable.
19. The method of claim 17 , wherein rotation of the first drive wheel in a first direction and rotation of the second drive wheel in a second direction opposite a first direction enables the wheel-based mobile robot to change orientation by swiveling on a traction surface.
20. The method of claim 11 , wherein the image processing system comprises a machine learning model configured to receive a respective pair of images as input and to predict the location of the target box as output, the respective pair of images having a respective identical time stamp and comprising a respective monocular image and a respective depth image.