Method and apparatus with object pose estimation

1. A processor-implemented method with object pose estimation, the method comprising:

determining an image feature corresponding to a point cloud of an input image;

determining semantic segmentation information, instance mask information, and keypoint information of an object, based on the image feature; and

estimating a pose of the object based on the semantic segmentation information, the instance mask information, and the keypoint information.

2. The method of claim 1 , wherein the input image comprises a depth image, and wherein the determining of the image feature comprises: extracting a point cloud feature based on the depth image; and determining the point cloud feature to be the image feature.

3. The method of claim 2 , wherein the extracting of the point cloud feature comprises:

determining point cloud information corresponding to the depth image; and

extracting the point cloud feature based on any one or any combination of any two or more of the point cloud information, a color feature, and a normal feature.

4. The method of claim 2 ,

wherein the input image further includes either one or both of a color image and a grayscale image, and

wherein the determining of the image feature comprises:

extracting a first image feature based on either one or both of the color image and the grayscale image; and

determining the image feature by fusing the point cloud feature and the first image feature.

5. The method of claim 4 , wherein the determining of the image feature by fusing the point cloud feature and the first image feature comprises determining the image feature by pixel-wise fusing the point cloud feature and the first image feature.

6. The method of claim 1 , wherein the determining of the semantic segmentation information, the instance mask information, and the keypoint information based on the image feature comprises:

determining the semantic segmentation information corresponding to the point cloud based on the image feature;

generating a three-dimensional (3D) grid based on point cloud information corresponding to the input image and determining the instance mask information based on the 3D grid; and

determining the keypoint information using the instance mask information or using the semantic segmentation information and the instance mask information.

7. The method of claim 6 ,

wherein the instance mask information indicates grid information corresponding to the point cloud in the 3D grid, and

wherein network information corresponding to each point cloud of the object is determined based on grid information corresponding to a point cloud of a center of the object.

8. The method of claim 6 , wherein the generating of the 3D grid based on the point cloud information comprises any one or any combination of any two or more of:

determining a 3D grid by dividing a 3D space corresponding to the point cloud information at equal intervals;

determining multiple 3D grids by dividing the 3D space corresponding to the point cloud information into different intervals; and

determining multiple 3D grids by dividing the 3D space corresponding to the point cloud information based on same division starting points of different intervals.

9. The method of claim 6 , wherein the determining of the keypoint information using the instance mask information comprises either one or both of:

estimating a first offset of a keypoint corresponding to each point cloud based on the image feature and determining the keypoint information through regression based on the first offset and the instance mask information; and

estimating a second offset of a keypoint corresponding to each point cloud in each cell of the 3D grid based on the image feature and the instance mask information, and determining the keypoint information through regression based on the second offset.

10. The method of claim 6 , wherein the determining of the keypoint information using the instance mask information comprises:

estimating a second offset of a keypoint corresponding to each point cloud in each cell of the 3D grid based on the image feature and the instance mask information; and

determining the keypoint information through regression based on the second offset by determining a target predicted value of a key point predicted based on a point cloud, based on the second offset and the point cloud information, and determining the keypoint information through regression based on the target predicted value.

11. The method of claim 10 , wherein the determining of the target predicted value and the determining of the keypoint information through the regression based on the target predicted value comprises any one or any combination of any two or more of:

for each keypoint of the object, determining an average value of a target predicted value corresponding to a keypoint and each point cloud, as keypoint information;

for each keypoint of the object, determining a weighted average value of a probability value corresponding to each point cloud in the instance mask information and a target predicted value corresponding to a keypoint and each point cloud, as keypoint information;

for each keypoint of the object, determining a weighted average value of a target predicted value corresponding to a keypoint and a point cloud of a preset value closest to a central point of the object and a probability value corresponding to the point cloud of the preset value in the instance mask information, as keypoint information of the object; and

for each keypoint of the object, determining a weighted average value of a distance approximate value corresponding to a keypoint and each point cloud, a target predicted value corresponding to a keypoint and each point cloud, and a probability value corresponding to each point cloud in the instance mask information, as keypoint information.

12. The method of claim 6 , wherein the determining of the keypoint information using the semantic segmentation information and the instance mask information comprises:

determining instance segmentation information based on the semantic segmentation information and the instance mask information;

estimating a first offset of a keypoint corresponding to each point cloud based on the image feature; and

determining the keypoint information through regression based on the first offset and the instance segmentation information.

13. The method of claim 12 , wherein the determining of the keypoint information through the regression based on the first offset and the instance segmentation information comprises:

determining an initial predicted value of a keypoint predicted based on a point cloud, based on the first offset and the point cloud information;

determining a target predicted value of a keypoint in the 3D grid based on the initial predicted value and the instance mask information; and

determining the keypoint information through regression based on the target predicted value.

14. The method of claim 12 , wherein the determining of the keypoint information through the regression based on the first offset and the instance segmentation information comprises:

determining an initial predicted value of a keypoint predicted based on a point cloud, based on the first offset and the point cloud information;

determining a target predicted value of a keypoint in the 3D grid based on the initial predicted value and the instance segmentation information; and

determining keypoint information of an object through a regression scheme based on the target predicted value.

15. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the method of claim 1 .

16. The method of claim 1 , wherein the estimating of the pose of the object includes estimating an orientation of the object.

17. The method of claim 16 , wherein the estimated pose is a 6 Degree-of-Freedom (DoF) pose.

18. The method of claim 1 , wherein the keypoint information is a determined location of the object.

19. The method of claim 1 , wherein the determining of the keypoint information includes determining the keypoint information based on determined instance segmentation information of the object.

20. The method of claim 19 , further comprising determining the instance segmentation information based on the semantic segmentation information and the instance mask information.

21. The method of claim 1 , wherein the instance mask information indicates in which regions in a multi-dimensional space include point cloud information of the object is present and/or in which regions in the multi-dimensional space the point cloud information of the object is not present.

22. An apparatus with object pose estimation, the apparatus comprising:

a processor configured to:

determine an image feature corresponding to a point cloud of an input image;

determine semantic segmentation information, instance mask information, and keypoint information of an object, based on the image feature; and

estimate a pose of the object based on the semantic segmentation information, the instance mask information, and the keypoint information.

23. The apparatus of claim 22 ,

wherein, for the determining of the semantic segmentation information, the instance mask information, and the keypoint information, the processor is configured to:

determine the semantic segmentation information corresponding to the point cloud based on the image feature;

generate a three-dimensional (3D) grid based on point cloud information corresponding to the input image and determine the instance mask information based on the 3D grid; and

determine the keypoint information using the instance mask information or using the semantic segmentation information and the instance mask information,

wherein the instance mask information indicates grid information corresponding to the point cloud in the 3D grid, and

wherein network information corresponding to each point cloud of the object is determined based on grid information corresponding to a point cloud of a center of the object.

24. The apparatus of claim 23 , wherein, for the determining of the keypoint information using the instance mask information, the processor is configured to:

estimate a first offset of a keypoint corresponding to each point cloud based on the image feature;

estimate a second offset of a keypoint corresponding to each point cloud in each cell of the 3D grid based on the image feature and the instance mask information; and

determine the keypoint information through either one or both of regression based on the first offset and the instance mask information and regression based on the second offset.

25. The apparatus of claim 22 , further comprising a memory storing instructions that, when executed by the processor, configure the processor to perform the determining of the image feature, the determining of the semantic segmentation information, the instance mask information, and the keypoint information, and the estimating of the pose.

26. An electronic device comprising:

a camera configured to generate an input image; and

a processor configured to:

determine an image feature corresponding to a point cloud of the input image;

determine semantic segmentation information, instance mask information, and keypoint information of an object, based on the image feature; and

estimate a pose of the object based on the semantic segmentation information, the instance mask information, and the keypoint information.

27. The device of claim 26 ,

wherein, for the determining of the semantic segmentation information, the instance mask information, and the keypoint information, the processor is configured to:

determine the semantic segmentation information corresponding to the point cloud based on the image feature;

generate a three-dimensional (3D) grid based on point cloud information corresponding to the input image and determine the instance mask information based on the 3D grid; and

determine the keypoint information using the instance mask information or using the semantic segmentation information and the instance mask information,

wherein the instance mask information indicates grid information corresponding to the point cloud in the 3D grid, and

wherein network information corresponding to each point cloud of the object is determined based on grid information corresponding to a point cloud of a center of the object.

28. A processor-implemented method with object pose estimation, the method comprising:

determining an image feature based on three-dimensional (3D) space information corresponding to an input image;

determining semantic segmentation information based on the image feature;

determining instance mask information based on the 3D space information;

determining keypoint information based on the instance mask information; and

determining a pose of an object based on the semantic segmentation information, the instance mask information, and the keypoint information.

29. The method of claim 28 , wherein the 3D space information comprises point cloud information.

30. The method of claim 29 , wherein the determining of the keypoint information comprises:

estimating, for each point cloud of the point cloud information, an offset of a keypoint based on the image feature; and

determining the keypoint information through regression based on the offsets.