Method, apparatus, and storage medium of generating data set containing RGBD image of robot hand and 3D pose of 21 key points on hand by game engine
The present disclosure discloses a data generation method and apparatus, and a computer-readable storage medium, the method including: importing a robot model by using a game engine; simulating a Red-Green-Blue Depth (RGBD) camera by a scene capture component in the game engine; controlling a human hand of the imported robot model to move within a field of view of the RGBD camera by using a joint control module in the game engine; acquiring RGBD image data by using the RGBD camera; and generating an annotated data set with coordinates of 21 key points according to the RGBD image data and coordinate information of a 3D pose of the 21 key points.
1 . A data generation method, comprising:
importing a robot model by using a game engine;
simulating an RGBD (Red, Green, Blue, Depth) camera by a scene capture component in the game engine;
controlling a human hand of the imported robot model to move within a field of view of the RGBD camera by using a joint control module in the game engine;
acquiring RGBD image data by using the RGBD camera; and
generating an annotated data set with coordinate of a key point according to the RGBD image data and coordinate information of a 3D pose of the key point;
wherein the simulating the RGBD camera by a scene capture component in the game engine comprises:
capturing a scene by using the scene capture component to obtain image data;
rendering the image data to a texture render target;
selecting a capture data source to recombine color image data and depth image data in the image data to obtain recombined image data; and
performing channel isolation of the color image data and unit unified processing of depth image data on the recombined image data so as to simulate the RGBD camera.
2 . The method according to claim 1 , wherein the importing a robot model by using a game engine comprises:
importing every joint of a robot into the game engine in accordance with a joint stacking manner according to a robot 3D model.
3 . The method according to claim 1 , wherein the RGBD image comprises a 2D color image and a depth image; and the generating an annotated data set with coordinates of 21 key points according to the RGBD image data and coordinate information of a 3D pose of the 21 key points, comprises:
transforming the coordinates of the 3D pose of the 21 key points into the 2D color image so as to annotate the 2D color image with a position of every key point; and
obtaining depth information of every key point by using the depth image.
4 . The method according to claim 3 , wherein prior to the transforming coordinates of the 3D pose of the 21 key points into the 2D color image, the method further comprises:
transforming the coordinates of the 3D pose of the 21 key points into coordinates in a coordinate system of the RGBD camera to obtain relative coordinates of the 21 key points; and
associating the RGBD image data with the relative coordinates of the 21 key points.
5 . A data generation apparatus, comprising: one or more processors; a memory configured to store one or more programs which, when executed by the one or more processors, cause the one or more processors to perform the data generation method of claim 1 .
6 . A computer-readable non-transitory storage medium, comprising a set of computer-executable instructions for performing the data generation method of claim 1 when the instructions are executed.
7 . The method according to claim 1 , wherein the key point comprises 21 key points, and the annotated data set with coordinate of a key point comprises annotated data set with coordinates of 21 key points.
8 . A data generation apparatus, comprising:
a model import module configured to import a robot model by using a game engine;
a camera simulation module configured to simulate an RGBD (Red, Green, Blue, Depth) camera by a scene capture component in the game engine;
a joint control module configured to control a human hand of the imported robot model to move within a field of view of the RGBD camera;
an image acquisition control module configured to acquire RGBD image data using the RGBD camera; and
a data generation module configured to generate an annotated data set with coordinate of a key point according to the RGBD image data and coordinate information of the 3D pose of the key point;
the camera simulation module is configured to capture a scene by using the scene capture component to obtain image data; render the image data to a texture render target; select a capture data source to recombine color image data and depth image data in the image data to obtain recombined image data; and perform channel isolation of color image data and unit unified processing of depth image data of the recombined image data so as to simulate an RGBD camera.
9 . The apparatus according to claim 8 , wherein
the model import module is configured to import every joint of a robot into the game engine in accordance with a joint stacking manner according to a robot 3D model.
10 . The apparatus according to claim 8 , wherein the key point comprises 21 key points, and the annotated data set with coordinate of a key point comprises annotated data set with coordinates of 21 key points.