System and method for using non-axis aligned bounding boxes for retail detection
Disclosed herein is a system and method for generating quadrilateral bonding boxes which tightly cover the most representative faces of retail products having arbitrary poses. The quadrilateral boxes do not include unnecessary background information or miss parts of the objects, as would the axis-aligned bounding boxes produced by prior art detectors. A simple projection transformation can correct the pose of products for downstream tasks.
1 . A system implementing a trained object detector comprising:
a localization sub-network taking an image as input and outputting one or more non-axis-aligned bounding boxes enclosing one or more objects detected in the image, the localization sub-network comprising:
a backbone network for extracting multiple feature maps from the image, the feature maps forming a feature pyramid network that up-samples high feature pyramid levels and sums them to adjacent lower levels to enhance semantic information in the lower levels;
a two-branched anchor-free detection head having a binary classification branch to predict a heatmap for object/background from the feature pyramid and a regression branch to predict offsets from pixel locations to the four corner points of a quadrilateral from the feature pyramid; and
a corner refinement module for determining locations of four corners and a center of the quadrilateral from output of the regression branch, wherein the center of the quadrilateral is a gravity center;
one or more downstream modules using the output of the localization sub-network.
2 . The system of claim 1 wherein the localization sub-network outputs non-axis aligned quadrilateral-shaped bounding boxes.
3 . The system of claim 2 wherein the quadrilateral-shaped bounding boxes are defined by a center point and four pairs of coordinates defining vertices of the quadrilateral as offsets from the center point.
4 . The system of claim 1 wherein the backbone network is Resnet.
5 . The system of claim 1 wherein the binary classification branch predicts a heatmap for differentiating objects from background in the image.
6 . The system of claim 5 wherein the binary classification branch comprises three stacks of convolutional layers followed by a single-channel convolutional layer.
7 . The system of claim 1 wherein the regression branch predicts the offsets from the central point defining the vertices of the quadrilateral bounding box.
8 . The system of claim 7 wherein the regression branch comprises three stacks of convolutional layers followed by an eight-channel convolutional layer.
9 . The system of claim 8 wherein the localization sub-network is trained on a dataset comprising images annotated with ground-truth quadrilateral-shaped bounding boxes.
10 . The system of claim 9 wherein a soft scale strategy is used to assign object to levels of the feature pyramid, wherein each object is assigned to two adjacent levels of the feature pyramid.
11 . The system of claim 1 , wherein the corner refinement module:
extracts features representing the central point and four vertices of the quadrilateral bounding box regression branch;
concatenates the features; and
inputs the concatenated features to a 1×1 convolutional layer to predict differences between ground-truth and a previous prediction of the features of the quadrilateral bounding box.
12 . The system of claim 10 wherein a loss applied to the detector is a sum of the losses from the regression branch and the binary classification branch.
13 . The system of claim 1 further comprising:
a processor; and
memory, containing instructions that, when executed by the processor, causes the system to implement the object detector.
14 . The system of claim 1 further comprising:
a processor; and
memory, containing instructions that, when executed by the processor, causes the system to train the object detector.
15 . The object detector of claim 1 wherein a projection transformation is applied to the non-axis aligned quadrilateral bounding boxes to correct the pose of the objects for the downstream modules.
16 . The system of claim 1 wherein the downstream modules perform tasks including one or more of pose estimation, classification and similarity matching.
17 . The system of claim 1 wherein the localization sub-network outputs triangle-shaped bounding boxes.
18 . The system of claim 17 wherein the triangle-shaped bounding boxes are defined by a center point and three pairs of coordinates defining vertices of the triangle as offsets from the center point.
19 . The system of claim 1 wherein the localization sub-network outputs elliptical-shaped bounding boxes.
20 . The system of claim 19 wherein the elliptical-shaped bounding boxes are defined by a center point and two channels defining lengths of minor and major axes of the ellipse.
21 . The system of claim 1 wherein the localization sub-network outputs three-dimensional bounding boxes.
22 . The system of claim 21 wherein the three-dimensional bounding boxes are defined by a center point and N channels defining parameters of the three-dimensional shape.