Method and apparatus for training object detection model

1. An object detection training method comprising:

estimating a pose and a dimension of an object based on a feature extracted from an input image, using an object detection model;

calculating a three-dimensional (3D) bounding box from the pose and the dimension;

determining a first output bounding box corresponding to the object by projecting the 3D bounding box to a first projection image;

determining a second output bounding box corresponding to the object by projecting the 3D bounding box to a second projection image; and

training the object detection model based on the pose, the dimension, the first output bounding box and the second output bounding box, the training comprising a fusion operation over the first output bounding box and the second output bounding box,

wherein the training of the object detection model comprises:

calculating a loss based on the pose, the dimension, the first output bounding box and the second output bounding box, and

wherein the calculating of the loss comprises:

calculating a first loss based on a degree of overlap between the first output bounding box and a first reference bounding box; and

calculating a second loss based on a degree of overlap between the second output bounding box and a second reference bounding box.

2. The object detection training method of claim 1 , wherein the determining of the first output bounding box comprises determining a bird's eye view bounding box corresponding to the object by projecting the 3D bounding box to a bird's eye view projection image.

3. The object detection training method of claim 1 , wherein the determining of the second output bounding box comprises determining a perspective bounding box corresponding to the object by projecting the 3D bounding box to a perspective projection image.

4. The object detection training method of claim 1 , wherein the estimating of the pose and the dimension of the object comprises extracting features from i) a two-dimensional (2D) bounding box corresponding to the object detected from the input image and ii) a crop image corresponding to the 2D bounding box.

5. The object detection training method of claim 1 , wherein the training of the object detection model comprises:

training the object detection model based on the loss, the training comprising updating weights in a neural network.

6. The object detection training method of claim 5 , wherein the calculating of the loss comprises:

determining a target box based on a front face box indicating a front face of the object;

and

calculating a loss of a perspective bounding box based on a degree of overlap between the target box and a reference box.

7. The object detection training method of claim 5 , wherein

the estimating of the pose and the dimension of the object comprises:

estimating a location offset based on the feature extracted from the input image;

estimating an initial location from the pose and the dimension;

determining an output location by applying the location offset to the estimated initial location; and

the calculating the loss is based on the determined output location.

8. The object detection training method of claim 1 , wherein the estimating of the pose and the dimension comprises estimating a confidence level of each of the estimated pose and the estimated dimension based on the feature extracted from the input image.

9. The object detection training method of claim 1 , wherein the estimating of the dimension of the object comprises estimating a width, a length and a height of the object.

10. The method of claim 1 , wherein the fusion operation over the first output bounding box and the second output bounding box comprises extracting a feature by identifying a feature obtained both by cropping the first output bounding box and obtained by cropping the second output bounding box.

11. An object detection training method comprising:

estimating, using an object detection model, a pose of an object based on features extracted from both an input image and a depth image;

estimating a three-dimensional (3D) bounding box corresponding to the object based on the features extracted from both the input image and the depth image, using the object detection model;

determining an output bounding box corresponding to the object by projecting the 3D bounding box to a projection image; and

training the object detection model based on the pose, the 3D bounding box and the output bounding box, the training comprising a fusion operation over the 3D bounding box and the output bounding box,

wherein the training of the object detection model comprises:

calculating a loss based on the pose, the 3D bounding box and the output bounding box, and

wherein the calculating of the loss comprises:

calculating a loss of the output bounding box based on a degree of overlap between the output bounding box and a reference bounding box.

12. The object detection training method of claim 11 , wherein the determining of the output bounding box comprises determining a perspective bounding box corresponding to the object by projecting the 3D bounding box to a perspective projection image.

13. The object detection training method of claim 11 , wherein the extracting the features from both the input image and the depth image is based on a region calculated by a region proposal network (RPN).

14. The object detection training method of claim 11 , wherein the training of the object detection model comprises:

training the object detection model based on the loss, the training comprising updating weights in a neural network.

15. The object detection training method of claim 14 , wherein the calculating of the loss comprises:

determining, from a perspective bounding box corresponding to the object, a target box based on a front face box indicating a front face of the object;

and

calculating a loss of the perspective bounding box based on a degree of overlap between the target box and a reference box.

16. The object detection training method of claim 11 , wherein the estimating of the pose and the 3D bounding box comprises estimating a location of a point that at least partially defines the 3D bounding box.

17. The object detection training method of claim 11 , wherein the estimating of the pose and the 3D bounding box is based on a depth image corresponding to a bird's eye view.

18. An object detection training apparatus comprises:

a memory configured to store an object detection model; and

a processor configured to:

estimate a pose and a dimension of an object based on a feature extracted from an input image, using the object detection model,

calculate a three-dimensional (3D) bounding box from the pose and the dimension,

determine a first output bounding box corresponding to the object by projecting the 3D bounding box to a first projection image,

determine a second output bounding box corresponding to the object by projecting the 3D bounding box to a second projection image, and

train the object detection model based on the pose, the dimension, the first output bounding box and the second output bounding box, and further based on a fusion operation over the first output bounding box and the second output bounding box,

wherein the processor is further configured to train the object detection model by:

calculating a loss based on the pose, the dimension, the first output bounding box and the second output bounding box, and

wherein the processor is further configured to calculate the loss by:

calculating a first loss based on a degree of overlap between the first output bounding box and a first reference bounding box; and

calculating a second loss based on a degree of overlap between the second output bounding box and a second reference bounding box.