Method and apparatus for training 6D pose estimation network based on deep learning iterative matching
The present disclosure relates to a method and an apparatus for training a 6D pose estimation network based on deep learning iterative matching. The method includes: obtaining a rendered image and a first segmentation mask of a target object by using a 3D model and an initial 6D pose estimation of the target object; inputting the rendered image, the first segmentation mask, an observed image of the target object, and a second segmentation mask of the target object in the observed image into a deep convolutional neural network to obtain a 6D pose estimation, a third segmentation mask and an optical flow; and performing said obtaining and said inputting again by updating the initial 6D pose estimation using the obtained relative 6D pose estimation and replacing the second segmentation mask with the third segmentation mask, to iteratively train the deep convolutional neural network.
1. A method for training a 6D pose estimation network based on deep learning iterative matching, comprising:
obtaining a rendered image and a first segmentation mask of a target object by using a 3D model and an initial 6D pose estimation of the target object;
inputting the rendered image, the first segmentation mask, an observed image of the target object, and a second segmentation mask of the target object in the observed image into a deep convolutional neural network to obtain a 6D pose estimation, a third segmentation mask and an optical flow; and
performing said obtaining and said inputting again by updating the initial 6D pose estimation using the obtained relative 6D pose estimation and replacing the second segmentation mask with the third segmentation mask, to iteratively train the deep convolutional neural network.
2. The method according to claim 1 , wherein said inputting the rendered image, the first segmentation mask, the observed image of the target object, and the second segmentation mask of the target object in the observed image into the deep convolutional neural network comprises:
enlarging the rendered image of the target object together with a rectangular region surrounding the second segmentation mask of the target object in the observed image, in such a manner that a 2D projection center of the 3D model of the target object is located at a center of the enlarged rendered image and that the target object in the observed image is completely located in the enlarged observed image; and
inputting the enlarged rendered image, the first segmentation mask, the enlarged observed image, and the second segmentation mask of the target object in the observed image into the deep convolutional neural network.
3. The method according to claim 1 , further comprising performing a 6D pose estimation, using the trained deep convolutional neural network, on a target object to be estimated by:
obtaining a rendered image and a fourth segmentation mask of the target object to be estimated by using a 3D model and an initial 6D pose estimation of the target object to be estimated;
inputting the rendered image of the target object to be estimated, the fourth segmentation mask, an observed image of the target object to be estimated, and a fifth segmentation mask of the target object to be estimated in the observed image into the trained deep convolutional neural network to obtain a 6D pose estimation; and
performing said obtaining and said inputting again by updating the initial 6D pose estimation of the target object to be estimated using the obtained relative 6D pose estimation, to iteratively improve the initial 6D pose estimation of the target object to be estimated.
4. The method according to claim 3 , wherein said inputting the rendered image of the target object to be estimated, the fourth segmentation mask, the observed image of the target object to be estimated, and the fifth segmentation mask of the target object to be estimated in the observed image into the trained deep convolutional neural network comprises:
enlarging the rendered image of the target object to be estimated together with a rectangular region surrounding the fifth segmentation mask of the target object to be estimated in the observed image of an initial prediction, in such a manner that a 2D projection center of the 3D model of the target object to be estimated is located at a center of the enlarged rendered image and that the target object to be estimated in the observed image is located completely in the enlarged observed image; and
inputting the enlarged rendered image, the fourth segmentation mask, the enlarged observed image, and the fifth segmentation mask of the target object to be estimated in the observed image into the trained deep convolutional neural network.
5. The method according to claim 1 , wherein the 6D pose estimation output by the deep convolutional neural network is represented by a relative pose transformation relative to a target pose, where the target pose is a pose of the target object in the observed image, the relative pose transformation includes a relative rotation transformation and a relative translation transformation, the relative rotation transformation takes a center point of the target object in a camera coordinate system as an origin of the camera coordinate system, and the relative translation transformation is represented by an offset and a scale change in the 2D pixel space.
6. The method according to claim 1 , wherein the deep convolutional neural network is constructed based on a FlowNet model for predicting the optical flow.
7. The method according to claim 1 , further comprising: in said iteratively training, constituting a loss function based on a weighted sum of loss functions of three branches for the 6D pose estimation, the optical flow and the third segmentation mask, wherein the branches for the optical flow and for the third segmentation mask are only used for said iteratively training.
8. The method according to claim 7 , wherein the loss function of the branch for the 6D pose estimation is:
L
pose
(
p
,
p
^
)
=
1
n
∑
j
=
1
n
L
1
(
(
Rx
j
+
t
)
-
(
R
^
x
j
+
t
^
)
)
,
where p=[R|t] denotes a target pose, {circumflex over (p)}=[{circumflex over (R)}|{circumflex over (t)}] denotes an estimated pose, R denotes the target rotation, {circumflex over (R)} denotes the estimated rotation, t denotes the target translation, {circumflex over (t)} denotes the estimated translation, x j denotes coordinates of a j th point in the 3D model of the target object, L 1 denotes 1-norm, and n denotes the total number of points in the 3D model.
9. An apparatus for training a 6D pose estimation network based on deep learning iterative matching, comprising:
a processor; and
a memory configured to store processor executable instructions,
wherein the processor is configured to, when executing the processor executable instructions:
obtain a rendered image and a first segmentation mask of a target object by using a 3D model and an initial 6D pose estimation of the target object;
input the rendered image, the first segmentation mask, an observed image of the target object, and a second segmentation mask of the target object in the observed image into a deep convolutional neural network to obtain a 6D pose estimation, a third segmentation mask and an optical flow; and
perform said obtaining and said inputting again by updating the initial 6D pose estimation using the obtained relative 6D pose estimation and replacing the second segmentation mask with the third segmentation mask, to iteratively train the deep convolutional neural network.
10. The apparatus according to claim 9 , wherein said inputting the rendered image, the first segmentation mask, the observed image of the target object, and the second segmentation mask of the target object in the observed image into the deep convolutional neural network comprises:
enlarging the rendered image of the target object together with a rectangular region surrounding the second segmentation mask of the target object in the observed image, in such a manner that a 2D projection center of the 3D model of the target object is located at a center of the enlarged rendered image and that the target object in the observed image is completely located in the enlarged observed image; and
inputting the enlarged rendered image, the first segmentation mask, the enlarged observed image, and the second segmentation mask of the target object in the observed image into the deep convolutional neural network.
11. The apparatus according to claim 9 , wherein the processor is further configured to, when executing the processor executable instructions, perform a 6D pose estimation, using the trained deep convolutional neural network, on a target object to be estimated by:
obtaining a rendered image and a fourth segmentation mask of the target object to be estimated by using a 3D model and an initial 6D pose estimation of the target object to be estimated;
inputting the rendered image of the target object to be estimated, the fourth segmentation mask, an observed image of the target object to be estimated, and a fifth segmentation mask of the target object to be estimated in the observed image into the trained deep convolutional neural network to obtain a 6D pose estimation; and
performing said obtaining and said inputting again by updating the initial 6D pose estimation of the target object to be estimated using the obtained relative 6D pose estimation, to iteratively improve the initial 6D pose estimation of the target object to be estimated.
12. The apparatus according to claim 11 , wherein said inputting the rendered image of the target object to be estimated, the fourth segmentation mask, the observed image of the target object to be estimated, and the fifth segmentation mask of the target object to be estimated in the observed image into the trained deep convolutional neural network comprises:
enlarging the rendered image of the target object to be estimated together with a rectangular region surrounding the fifth segmentation mask of the target object to be estimated in the observed image of an initial prediction, in such a manner that a 2D projection center of the 3D model of the target object to be estimated is located at a center of the enlarged rendered image and that the target object to be estimated in the observed image is located completely in the enlarged observed image; and
inputting the enlarged rendered image, the fourth segmentation mask, the enlarged observed image, and the fifth segmentation mask of the target object to be estimated in the observed image into the trained deep convolutional neural network.
13. The apparatus according to claim 9 , wherein the 6D pose estimation output by the deep convolutional neural network is represented by a relative pose transformation relative to a target pose, where the target pose is a pose of the target object in the observed image, the relative pose transformation includes a relative rotation transformation and a relative translation transformation, the relative rotation transformation takes a center point of the target object in a camera coordinate system as an origin of the camera coordinate system, and the relative translation transformation is represented by an offset and a scale change in the 2D pixel space.
14. The apparatus according to claim 9 , wherein the deep convolutional neural network is constructed based on a FlowNet model for predicting the optical flow.
15. The apparatus according to claim 10 , wherein the processor is further configured to, when executing the processor executable instructions: in said iteratively training, constitute a loss function based on a weighted sum of loss functions of three branches for the 6D pose estimation, the optical flow and the third segmentation mask, wherein the branches for the optical flow and for the third segmentation mask are only used for said iteratively training.
16. The apparatus according to claim 15 , wherein the loss function of the branch for the 6D pose estimation is:
L
pose
(
p
,
p
^
)
=
1
n
∑
j
=
1
n
L
1
(
(
Rx
j
+
t
)
-
(
R
^
x
j
+
t
^
)
)
,
where p=[R|t] denotes a target pose, {circumflex over (p)}=[{circumflex over (R)}|{circumflex over (t)}] denotes an estimated pose, R denotes the target rotation, {circumflex over (R)} denotes the estimated rotation, t denotes the target translation, {circumflex over (t)} denotes the estimated translation, x j denotes coordinates of a j th point in the 3D model of the target object, L 1 denotes 1-norm, and n denotes the total number of points in the 3D model.
17. A non-volatile computer readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when being executed by a processor, perform the method according to claim 1 .