Text-tagged motion generation apparatus and operation method thereof

1 . A motion generation device for generating a text-tagged motion, comprising:

a memory configured to store at least one instruction; and

at least one processor configured to execute the at least one instruction stored in the memory,

wherein the processor is configured to:

obtain animation data including a character,

convert the obtained animation data into intermediate data for generating a motion of the character over a plurality of frames,

generate the motion of the character based on the converted intermediate data, using a motion rendering module,

generate a caption for each of the plurality of frames included in the generated motion of the character,

generate a text corresponding to the motion of the character by providing the plurality of generated captions to a language model, and

generate the text-tagged motion by labeling the generated text to the motion of the character,

wherein the obtained animation data includes data about the motion of the character and data obtained through motion capture,

wherein the motion of the character includes motion data including at least one of position information of a joint of the character, rotation information of the joint, or contact information between a floor and a foot in each of the plurality of frames, and a motion image rendered by retargeting the motion data to the character,

wherein the motion rendering module includes a data conversion module and a motion generation module,

wherein the processor is configured to:

convert the obtained animation data into SMPL (Skinned Multi-Person Linear Model)-based intermediate data using the data conversion module, and

sequentially play the converted intermediate data using the motion generation module, wherein during a first playback, motion data is generated by extracting motion data from each of the plurality of frames included in the converted intermediate data, and during a second playback, a motion image is generated,

wherein the generated motion data is generated in a Json (Javascript object notation) type, and position information of skeleton joints, rotation information of the joints, and contact information with the floor are stored as key-value pairs,

wherein the processor is configured to, during the second playback, play the motion of the character based on the position information of skeleton joints, the rotation information of the joints, and information on whether the feet are in contact with the floor included in the generated motion data, and

wherein the processor is configured to:

extract a plurality of frame images included in the generated motion image, and

generate the plurality of captions corresponding to each of the plurality of frames based on the extracted plurality of frame images.

2 . The device according to claim 1 , wherein the processor is configured to:

obtain a correction signal requesting to re-generate text corresponding to the motion of the generated character, and

re-generate a text corresponding to the motion of the generated character by providing information included in the correction signal and the plurality of generated captions to the language model.

3 . The device according to claim 1 , wherein the generated text-tagged motion is learning data used to train an artificial intelligence model that receives the motion data and infers the text.

4 . A method of operating a motion generation device for generating a text-tagged motion, performed by a processor, comprising:

obtaining animation data including a character,

converting the obtained animation data into intermediate data for generating a motion of the character over a plurality of frames,

generating the motion of the character based on the converted intermediate data, using a motion rendering module,

generating a caption for each of the plurality of frames included in the generated motion of the character,

generating a text corresponding to the motion of the character by providing the plurality of generated captions to a language model, and

generating the text-tagged motion by labeling the generated text to the motion of the character,

wherein the obtained animation data includes data about the motion of the character and data obtained through motion capture,

wherein the motion of the character includes motion data including at least one of position information of a joint of the character, rotation information of the joint, or contact information between a floor and a foot in each of the plurality of frames, and a motion image rendered by retargeting the motion data to the character,

wherein the motion rendering module includes a data conversion module and a motion generation module,

wherein the processor is configured to:

convert the obtained animation data into SMPL (Skinned Multi-Person Linear Model)-based intermediate data using the data conversion module, and

sequentially play the converted intermediate data using the motion generation module, wherein during a first playback, motion data is generated by extracting motion data from each of the plurality of frames included in the converted intermediate data, and during a second playback, a motion image is generated,

wherein the generated motion data is generated in a Json (Javascript object notation) type, and position information of skeleton joints, rotation information of the joints, and contact information with the floor are stored as key-value pairs,

wherein the processor is configured to, during the second playback, play the motion of the character based on the position information of skeleton joints, the rotation information of the joints, and information on whether the feet are in contact with the floor included in the generated motion data, and

wherein the processor is configured to:

extract a plurality of frame images included in the generated motion image, and

generate the plurality of captions corresponding to each of the plurality of frames based on the extracted plurality of frame images.

5 . The method according to claim 4 , further comprising:

obtaining a correction signal requesting to re-generate text corresponding to the motion of the generated character, and

re-generating a text corresponding to the motion of the generated character by providing information included in the correction signal and the plurality of generated captions to the language model.