Method, system and apparatus for shelf edge detection

1. A method of detecting an edge of a support surface by an imaging controller, the method comprising:

obtaining image data captured by an image sensor and a plurality of depth measurements captured by a depth sensor, the image data and the plurality of depth measurements corresponding to an area containing the support surface;

detecting preliminary edges in the image data;

applying a Hough transform to the preliminary edges to determine Hough lines representing candidate edges of the support surface;

segmenting the plurality of depth measurements to assign classes to each pixel, each class defined by one of a plurality of seed pixels, wherein the plurality of seed pixels are identified from the depth measurements based on the Hough lines; and

detecting the edge of the support surface by selecting a class of pixels and applying a line-fitting model to the selected class to obtain an estimated edge of the support surface,

wherein segmenting the plurality of depth measurements comprises:

overlaying the Hough lines with the depth measurements and identifying the seed pixels as the depth measurements which correspond to Hough lines;

selecting a seed pixel and defining an object class based on the selected seed pixel;

iteratively growing the object class by:

selecting an unclassified pixel adjacent to a pixel in the object class;

determining if the unclassified adjacent pixel is part of the object class; and

when the determination is positive, adding the unclassified adjacent pixel to the object class.

2. The method of claim 1 , wherein determining if the unclassified adjacent pixel is part of the object class comprises:

determining an angle between an origin in the image frame of reference, the unclassified adjacent pixel, and the pixel in the object class; and

when the angle is above a predefined threshold, determining that the unclassified adjacent pixel is part of the object class.

3. The method of claim 1 , further comprising, prior to segmenting the depth measurements, discarding Hough lines within a threshold angle of vertical.

4. The method of claim 1 , further comprising, prior to segmenting the depth measurements:

selecting a ground seed pixel and defining a ground class for the ground seed pixel;

iteratively growing the ground class by:

selecting an unclassified pixel adjacent to a pixel in the ground class;

determining if the unclassified adjacent pixel is part of the ground class; and

when the determination is positive, adding the unclassified adjacent pixel to the ground class; and

discarding pixels in the ground class.

5. The method of claim 1 , further comprising estimating a support surface plane associated with the support surface as a vertical plane of the estimated edge of the support surface.

6. The method of claim 5 , further comprising, based on the estimated support surface plane:

computing a current distance of an apparatus of the imaging controller to the estimated support surface plane; and

computing a current yaw of the apparatus to the estimated support surface plane.

7. The method of claim 6 , further comprising:

computing an average distance of the apparatus to the estimated support surface plane based on the current distance and a plurality of previous distances; and

computing an average yaw of the apparatus to the estimated support surface plane based on the current yaw and a plurality of previous yaws.

8. The method of claim 6 , further comprising:

obtaining a second estimated support surface plane, the second estimated support surface plane computed by a different detection method;

computing an agreement score of the estimated support surface plane and the second estimated support surface plane; and

when the agreement score is above a threshold score, publishing the estimated support surface plane as a detected support surface plane.

9. The method of claim 1 , wherein selecting the class of pixels comprises selecting a largest object class wherein a difference in height between pixels in the class and its seed pixel are within a first threshold distance; and wherein a distance between consecutive pixels in the class are within a second threshold distance.

10. A mobile automation apparatus, comprising:

a locomotive assembly;

an image sensor and a depth sensor; and

an imaging controller configured to:

obtain image data captured by the image sensor and a plurality of depth measurements captured by the depth sensor, the image data and the plurality of depth measurements corresponding to an area containing the support surface;

detect preliminary edges in the image data;

apply a Hough transform to the preliminary edges to determine Hough lines representing candidate edges of the support surface;

segment the plurality of depth measurements to assign classes to each pixel, each class defined by one of a plurality of seed pixels, wherein the plurality of seed pixels are identified from the depth measurements based on the Hough lines; and

detect an edge of the support surface by selecting a class of pixels and applying a line-fitting model to the selected class to obtain an estimated edge of the support surface,

wherein the imaging controller is further configured to:

estimate a support surface plane associated with the support surface as a vertical plane of the estimated edge of the support surface;

obtain a second estimated support surface plane, the second estimated support surface plan computed by a different detection method;

compute an agreement score of the estimated support surface plane and the second estimated support surface plane; and

when the agreement score is above a threshold score, publish the estimated support surface plane as a detected support surface plane.

11. The mobile automation apparatus of claim 10 , wherein the imaging controller is configured to segment the depth measurements by:

overlaying the Hough lines with the depth measurements and identifying the seed pixels as the depth measurements which correspond to Hough lines;

selecting a seed pixel and defining an object class based on the selected seed pixel;

iteratively growing the object class by:

selecting an unclassified pixel adjacent to a pixel in the object class;

determining if the unclassified adjacent pixel is part of the object class; and

when the determination is positive, adding the unclassified adjacent pixel to the object class.

12. The mobile automation apparatus of claim 11 , wherein the imaging controller is configured to determine if the unclassified adjacent pixel is part of the object class by:

determining an angle between an origin in the image frame of reference, the unclassified adjacent pixel, and the pixel in the object class; and

when the angle is above a predefined threshold, determining that the unclassified adjacent pixel is part of the object class.

13. The mobile automation apparatus of claim 10 , wherein the imaging controller is further configured to, prior to segmenting the depth measurements, discard Hough lines within a threshold angle of vertical.

14. The mobile automation apparatus of claim 10 , wherein the imaging controller is further configured to, prior to segmenting the depth measurements:

select a ground seed pixel and defining a ground class for the ground seed pixel;

iteratively grow the ground class by:

selecting an unclassified pixel adjacent to a pixel in the ground class;

determining if the unclassified adjacent pixel is part of the ground class; and

when the determination is positive, adding the unclassified adjacent pixel to the ground class; and

discard pixels in the ground class.

15. The mobile automation apparatus of claim 10 , wherein the imaging controller is further configured to, based on the estimated support surface plane:

compute a current distance of an apparatus of the imaging controller to the estimated support surface plane; and

compute a current yaw of the apparatus to the estimated support surface plane.

16. The mobile automation apparatus of claim 15 , wherein the imaging controller is further configured to:

compute an average distance of the apparatus to the estimated support surface plane based on the current distance and a plurality of previous distances; and

compute an average yaw of the apparatus to the estimated support surface plane based on the current yaw and a plurality of previous yaws.

17. The mobile automation apparatus of claim 10 , wherein the imaging controller is further configured to select the class of pixels by selecting a largest object class wherein a difference in height between pixels in the class and its seed pixel are within a first threshold distance; and wherein a distance between consecutive pixels in the class are within a second threshold distance.

18. A method of detecting an edge of a support surface by an imaging controller, the method comprising:

obtaining image data captured by an image sensor and a plurality of depth measurements captured by a depth sensor, the image data and the plurality of depth measurements corresponding to an area containing the support surface;

detecting preliminary edges in the image data;

applying a Hough transform to the preliminary edges to determine Hough lines representing candidate edges of the support surface;

segmenting the plurality of depth measurements to assign classes to each pixel, each class defined by one of a plurality of seed pixels, wherein the plurality of seed pixels are identified from the depth measurements based on the Hough lines;

detecting the edge of the support surface by selecting a class of pixels and applying a line-fitting model to the selected class to obtain an estimated edge of the support surface; and

prior to segmenting the depth measurements:

selecting a ground seed pixel and defining a ground class for the ground seed pixel,

iteratively growing the ground class by:

selecting an unclassified pixel adjacent to a pixel in the ground class,

determining if the unclassified adjacent pixel is part of the ground class,

when the determination is positive, adding the unclassified adjacent pixel to the ground class, and

discarding pixels in the ground class.