Point cloud data processing method, and robot and robot control method using the same

1 . A point cloud data processing method, comprising:

collecting, through an RGBD camera, image data including an RGB image and a depth image;

obtaining an original mask image by segmenting out one or more targets from the RGB image using a target segmentation model;

obtaining an optimized mask image by performing a pixel-level processing on the original mask image;

obtaining, based on the optimized mask image and the depth image, a plane equation of each of the targets in the optimized mask image;

performing, using the plane equation of each of the targets, a depth value assignment on a plane position of the target in the optimized mask image in response to the depth value not being assigned at the plane position; and

obtaining target point cloud data by performing a point cloud conversion on the depth image after the depth value assignment.

2 . The method of claim 1 , wherein obtaining the optimized mask image by performing the pixel-level processing on the original mask image comprises:

obtaining a pixel number of each of the targets in the original mask image by counting pixels of the target in the original mask image using a hash table;

obtaining a first mask image by performing a primary denoising processing on the original mask image according to the pixel number of each of the targets; and

obtaining the optimized mask image by performing a secondary denoising processing on the first mask image.

3 . The method of claim 2 , wherein performing the primary denoising processing on the original mask image according to the pixel number of each of the targets comprises:

setting, according to the pixel number of each of the targets, a target area of the target in the original mask image that has pixels of the pixel number less than a first preset threshold as a background.

4 . The method of claim 2 , wherein performing the secondary denoising processing on the first mask image comprises:

determining a connected area of each of mask areas in the first mask image, and setting the mask area with the connected area having pixels less than a second preset threshold as a background.

5 . The method of claim 1 , wherein obtaining, based on the optimized mask image and the depth image, a plane equation of each of the targets in the optimized mask image comprises:

traversing each of target areas in the optimized mask image to determine a plane coordinate of the target area with a non-zero pixel value, and obtaining initial point cloud information of each of the targets by extracting a point cloud of a corresponding position of the plane coordinate in the depth image, wherein the corresponding position has a non-zero original depth value; and

obtaining the plane equation of each of the targets by performing a plane fitting according to the initial point cloud information of the target.

6 . The method of claim 5 , wherein obtaining the plane equation of each of the targets by performing the plane fitting according to the initial point cloud information of the target comprises:

obtaining the plane equation of each of the targets by performing the plane fitting on the initial point cloud information of the target using a least squares method.

7 . The method of claim 1 , wherein performing, using the plane equation of each of the targets, a depth value assignment on a plane position of the target in the optimized mask image comprises:

traversing each of target areas in the optimized mask image to determine a plane coordinate of the target area with a non-zero pixel value, and obtaining to-be-supplemented point cloud information of each of the targets by extracting a point cloud of a corresponding position of the plane coordinate in the depth image, wherein the corresponding position has a non-zero original depth value; and

calculating a corresponding operation depth value according to the plane equation of each of the targets and the plane coordinate corresponding to the to-be-supplemented point cloud information, and setting the depth value of the corresponding plane coordinate position in the depth image as the operation depth value.

8 . The method of claim 1 , wherein the target segmentation model is a neural network.

9 . A control method for a robot having an RGBD camera, comprising:

collecting, through the RGBD camera, image data including an RGB image and a depth image;

obtaining an original mask image by segmenting out one or more targets from the RGB image using a target segmentation model;

obtaining an optimized mask image by performing a pixel-level processing on the original mask image;

obtaining, based on the optimized mask image and the depth image, a plane equation of each of the targets in the optimized mask image;

performing, using the plane equation of each of the targets, a depth value assignment on a plane position of the target in the optimized mask image in response to the depth value not being assigned at the plane position;

obtaining target point cloud data by performing a point cloud conversion on the depth image after the depth value assignment; and

performing, based on the target point cloud data, obstacle avoidance on the robot.

10 . The method of claim 9 , wherein performing, based on the target point cloud data, obstacle avoidance on the robot comprises:

determining, based on the target point cloud data, a position of an obstacle; and

planning, based on the position of the obstacle, a path for the robot to avoid the obstacle.

11 . A robot, comprising:

an RGBD camera;

a processor;

a memory coupled to the processor; and

one or more computer programs stored in the memory and executable on the processor;

wherein, the one or more computer programs comprise:

instructions for collecting, through the RGBD camera, image data including an RGB image and a depth image;

instructions for obtaining an original mask image by segmenting out one or more targets from the RGB image using a target segmentation model;

instructions for obtaining an optimized mask image by performing a pixel-level processing on the original mask image;

instructions for obtaining, based on the optimized mask image and the depth image, a plane equation of each of the targets in the optimized mask image;

instructions for performing, using the plane equation of each of the targets, a depth value assignment on a plane position of the target in the optimized mask image in response to the depth value not being assigned at the plane position; and

instructions for obtaining target point cloud data by performing a point cloud conversion on the depth image after the depth value assignment.

12 . The robot of claim 11 , wherein the instructions for obtaining the optimized mask image by performing the pixel-level processing on the original mask image comprise:

instructions for obtaining a pixel number of each of the targets in the original mask image by counting pixels of the target in the original mask image using a hash table;

instructions for obtaining a first mask image by performing a primary denoising processing on the original mask image according to the pixel number of each of the targets; and

instructions for obtaining the optimized mask image by performing a secondary denoising processing on the first mask image.

13 . The robot of claim 12 , wherein the instructions for performing the primary denoising processing on the original mask image according to the pixel number of each of the targets comprise:

instructions for setting, according to the pixel number of each of the targets, a target area of the target in the original mask image that has pixels of the pixel number less than a first preset threshold as a background.

14 . The robot of claim 12 , wherein the instructions for performing the secondary denoising processing on the first mask image comprise:

instructions for determining a connected area of each of mask areas in the first mask image, and setting the mask area with the connected area having pixels less than a second preset threshold as a background.

15 . The robot of claim 11 , wherein the instructions for obtaining, based on the optimized mask image and the depth image, a plane equation of each of the targets in the optimized mask image comprise:

instructions for traversing each of target areas in the optimized mask image to determine a plane coordinate of the target area with a non-zero pixel value, and obtaining initial point cloud information of each of the targets by extracting a point cloud of a corresponding position of the plane coordinate in the depth image, wherein the corresponding position has a non-zero original depth value; and

instructions for obtaining the plane equation of each of the targets by performing a plane fitting according to the initial point cloud information of the target.

16 . The robot of claim 15 , wherein the instructions for obtaining the plane equation of each of the targets by performing the plane fitting according to the initial point cloud information of the target comprise:

instructions for obtaining the plane equation of each of the targets by performing the plane fitting on the initial point cloud information of the target using a least squares method.

17 . The robot of claim 11 , wherein the instructions for performing, using the plane equation of each of the targets, a depth value assignment on a plane position of the target in the optimized mask image comprise:

instructions for traversing each of target areas in the optimized mask image to determine a plane coordinate of the target area with a non-zero pixel value, and obtaining to-be-supplemented point cloud information of each of the targets by extracting a point cloud of a corresponding position of the plane coordinate in the depth image, wherein the corresponding position has a non-zero original depth value; and

instructions for calculating a corresponding operation depth value according to the plane equation of each of the targets and the plane coordinate corresponding to the to-be-supplemented point cloud information, and setting the depth value of the corresponding plane coordinate position in the depth image as the operation depth value.

18 . The robot of claim 11 , wherein the target segmentation model is a neural network.

19 . The robot of claim 11 , the one or more computer programs further comprise:

instructions for performing, based on the target point cloud data, obstacle avoidance on the robot.

20 . The robot of claim 19 , wherein the instructions for performing, based on the target point cloud data, obstacle avoidance on the robot comprise:

instructions for determining, based on the target point cloud data, a position of an obstacle; and

instructions for planning, based on the position of the obstacle, a path for the robot to avoid the obstacle.