Deep learning system for cuboid detection
Systems and methods for cuboid detection and keypoint localization in images are disclosed. In one aspect, a deep cuboid detector can be used for simultaneous cuboid detection and keypoint localization in monocular images. The deep cuboid detector can include a plurality of convolutional layers and non-convolutional layers of a trained convolution neural network for determining a convolutional feature map from an input image. A region proposal network of the deep cuboid detector can determine a bounding box surrounding a cuboid in the image using the convolutional feature map. The pooling layer and regressor layers of the deep cuboid detector can implement iterative feature pooling for determining a refined bounding box and a parameterized representation of the cuboid.
1. A system for training a cuboid detector, the system comprising:
non-transitory memory configured to store executable instructions; and
one or more hardware processors in communication with the non-transitory memory, the one or more hardware processors programmed by the executable instructions to:
access a plurality of training images, wherein the plurality of training images includes a first training image;
generate a cuboid detector, wherein the cuboid detector comprises:
a plurality of convolutional layers and non-convolutional layers of a first convolutional neural network (CNN);
a region proposal network (RPN) connected to a first layer of the plurality of convolutional layers and non-convolutional layers;
a pooling layer; and
at least one regressor layer;
wherein the pooling layer and the at least one regressor layer are both connected to a second layer of the plurality of convolutional layers and non-convolutional layers; and
train the cuboid detector, wherein training the cuboid detector comprises:
determining, by applying the cuboid detector to the first training image, a region of interest (RoI) at a cuboid image location;
determining, by applying the cuboid detector to the first training image, a representation of a cuboid in the training image;
determining a first difference between a reference cuboid image location and the cuboid image location;
determining a second difference between a reference representation of the cuboid and the determined representation of the cuboid; and
updating weights of the cuboid detector based on the first difference and the second difference.
2. The system of claim 1 , wherein the cuboid comprises a cuboid, a cylinder, a sphere, or any combination thereof.
3. The system of claim 1 , wherein the first layer and the second layer are identical.
4. The system of claim 1 , wherein the one or more hardware processors is further programmed to:
generate, using the plurality of convolutional layers and the non-convolutional layers, a convolutional feature map for the first training image;
determine, using the RPN, at least one RoI comprising the cuboid at an initial cuboid image location in the training image;
determine, using the initial cuboid image location, a submap of the convolutional feature map corresponding to the at least one RoI comprising the cuboid; and
determine, using the pooling layer, the at least one regressor layer, and the submap of the convolutional feature map corresponding to the at least one RoI comprising the cuboid, the RoI at the cuboid image location and the representation of the cuboid.
5. The system of claim 4 , wherein the initial cuboid image location is represented as a two-dimensional (2D) bounding box.
6. The system of claim 1 , wherein the one or more hardware processors is further programmed to:
iteratively determine, using the pooling layer, the at least one regressor layer, and the submap of the convolutional feature map corresponding to the RoI comprising the cuboid, the RoI at the cuboid image location and the representation of the cuboid.
7. The system of claim 6 , wherein the initial cuboid image location is represented as a two-dimensional (2D) bounding box.
8. The system of claim 1 , wherein the one or more hardware processors is further programmed to:
update the weights of the RPN; and
update the weights of the at least one regressor layer.
9. The system of claim 1 , wherein the one or more hardware processors is further programmed to:
update the weights of the first CNN;
update the weights of the RPN; and
update the weights of the at least one regressor layer.
10. The system of claim 1 , wherein the one or more hardware processors is further programmed to:
receive the first CNN.
11. The system of claim 1 , wherein the at least one regressor layer comprises two or more layers.
12. The system of claim 11 , wherein the two or more layers comprise a fully connected layer, a non-fully connected layer, or any combination thereof.
13. The system of claim 1 , wherein the at least one regressor layer is associated with at least three loss functions during training of the cuboid detector.
14. The system of claim 1 , wherein RPN comprises a deep neural network (DNN).
15. The system of claim 1 , wherein the RPN is associated with at least two loss functions during the training of the cuboid detector.
16. The system of claim 1 , wherein the representation of the cuboid comprises a parameterized representation of the cuboid.
17. A method for training a cuboid detector, the method comprising:
accessing a plurality of training images, wherein the plurality of training images includes a first training image;
generating a cuboid detector, wherein the cuboid detector comprises:
a plurality of convolutional layers and non-convolutional layers of a first convolutional neural network (CNN),
a region proposal network (RPN) connected to a first layer of the plurality of convolutional layers and non-convolutional layers,
a pooling layer; and
at least one regressor layer, wherein the pooling layer and the at least one regressor layer are both connected to a second layer of the plurality of convolutional layers and non-convolutional layers; and
training the cuboid detector, wherein training the cuboid detector comprises:
determining, by applying the cuboid detector to the first training image, a region of interest (RoI) at a cuboid image location;
determining, by applying the cuboid detector to the first training image, a representation of a cuboid in the training image;
determining a first difference between a reference cuboid image location and the cuboid image location;
determining a second difference between a reference representation of the cuboid and the determined representation of the cuboid; and
updating weights of the cuboid detector based on the first difference and the second difference.
18. The method of claim 17 , wherein the first layer and the second layer are identical.
19. The method of claim 17 , further comprising:
generating, using the plurality of convolutional layers and the non-convolutional layers, a convolutional feature map for the first training image;
determining, using the RPN, at least one RoI comprising the cuboid at an initial cuboid image location in the training image;
determining, using the initial cuboid image location, a submap of the convolutional feature map corresponding to the at least one RoI comprising the cuboid; and
determining, using the pooling layer, the at least one regressor layer, and the submap of the convolutional feature map corresponding to the at least one RoI comprising the cuboid, the RoI at the cuboid image location and the representation of the cuboid.
20. The method of claim 17 , further comprising:
iteratively determining, using the pooling layer, the at least one regressor layer, and the submap of the convolutional feature map corresponding to the RoI comprising the cuboid, the RoI at the cuboid image location and the representation of the cuboid.