Granted Patent B2

US 12,437,457 · App. 18/312,149 · Granted Oct 7, 2025

Inventor: Sang Hoon Lee (Seoul, KR)

Assignee: UIF (University Industry Foundation, Yonsei University

G06T13/40

Abstract

Claims (63)

1. An apparatus for generating a dancing avatar comprising: one or more processors; and a memory that stores one or more programs executed by the one or more processors,

wherein the processors

receive a latent code and map it using a neural network operation to obtain a plurality of genre-specific style codes for each of a plurality of dance genres, and

decode seed motion data and music data, which are motion data that must be referred to when generating an avatar's dance motion, using a genre-specific style code for a dance genre selected among the plurality of genre-specific style codes as a guide, thereby obtaining a dance vector representing a dance motion feature of the avatar in the selected dance genre,

wherein the processors

receive the dance vector, convert it into a format of the motion data to obtain dance data, and

apply an avatar skin to the obtained dance data, thereby generating a dancing avatar,

wherein during training, the processors

receive the dance data and the music data and project them into a virtual common feature vector space to obtain a motion vector and a music vector,

obtain a feature map by transformer encoding the motion vector and the music vector using a neural network operation,

determine the dance genre of the dance data from a genre score obtained by pooling the feature map, and

calculate a loss according to a difference between the determined dance genre and the selected dance genre, and back-propagate it.

2. The apparatus for generating a dancing avatar according to claim 1 ,

wherein the processors obtain the plurality of genre-specific style codes by mapping the latent code to each area divided according to each of a plurality of dance genres in a virtual style space.

3. The apparatus for generating a dancing avatar according to claim 1 ,

wherein the processors randomly and repeatedly generate the latent code.

4. The apparatus for generating a dancing avatar according to claim 1 ,

wherein the processors select one dance genre from the plurality of dance genres in response to a user command.

5. The apparatus for generating a dancing avatar according to claim 1 ,

wherein the processors

project each of the seed motion data and the music data into a virtual common feature vector space to obtain a motion vector and a music vector, and

decode the obtained motion vector and the music vector using a transformer decoder so that features designated by the selected genre-specific style code stand out, thereby obtaining the dance vector.

6. The apparatus for generating a dancing avatar according to claim 1 ,

wherein the processors obtain previously obtained dance data as the seed motion data.

7. The apparatus for generating a dancing avatar according to claim 1 ,

wherein the processors obtain a random value, capture data obtained by capturing user's motions, and 3D dance motion data extracted from a 2D or 3D dance video as an initial value of the seed motion data.

8. The apparatus for generating a dancing avatar according to claim 1 ,

wherein the processors calculate

a style focus loss according to the difference between the dance genre of the seed motion data and the determined dance genre, and

when the dance genre of the seed motion data and the selected dance genre are the same, a dance genre-specific loss calculated as the difference between the seed motion data and the dance data, and

a style diversity loss that maximizes the difference between dance data by obtaining dance data previously obtained from the same selected dance genre as the seed motion data, so that repeatedly generated dance data represents various motions,

and further apply them to the loss.

9. A method for generating a dancing avatar, performed by a computing device having one or more processors and a memory storing one or more programs executed by the one or more processors, comprising the steps of:

receiving a latent code and mapping it using a neural network operation to obtain a plurality of genre-specific style codes for each of a plurality of dance genres; and

decoding seed motion data and music data that must be referred to when generating an avatar's dance motion, using a genre-specific style code for a dance genre selected among the plurality of genre-specific style codes as a guide, thereby obtaining a dance vector representing a dance motion feature of the avatar in the selected dance genre,

wherein the method further includes the steps of

receiving the dance vector, converting it into a format of the motion data to obtain dance data, and applying an avatar skin to the obtained dance data, thereby generating a dancing avatar, and

during training, receiving the dance data and the music data and projecting them into a virtual common feature vector space to obtain a motion vector and a music vector, obtaining a feature map by transformer encoding the motion vector and the music vector using a neural network operation, determining the dance genre of the dance data from a genre score obtained by pooling the feature map, and calculating a loss according to a difference between the determined dance genre and the selected dance genre, and back-propagating it.

10. The method for generating a dancing avatar according to claim 9 ,

wherein the step of obtaining the style codes includes

obtaining the plurality of genre-specific style codes by mapping the latent code to each area divided according to each of a plurality of dance genres in a virtual style space.

11. The method for generating a dancing avatar according to claim 9 ,

wherein the step of obtaining the style codes includes

randomly and repeatedly generating the latent code.

12. The method for generating a dancing avatar according to claim 9 ,

wherein the step of obtaining the style codes includes

selecting one dance genre from the plurality of dance genres in response to a user command.

13. The method for generating a dancing avatar according to claim 9 ,

wherein the step of obtaining a dance vector includes

projecting each of the seed motion data and the music data into a virtual common feature vector space to obtain a motion vector and a music vector, and

decoding the obtained motion vector and the music vector using a transformer decoder so that features designated by the selected genre-specific style code stand out, thereby obtaining the dance vector.

14. The method for generating a dancing avatar according to claim 9 ,

wherein the step of obtaining a dance vector includes

obtaining previously obtained dance data as the seed motion data.

15. The method for generating a dancing avatar according to claim 9 ,

wherein the step of obtaining the feature vector includes

obtaining a random value, capture data obtained by capturing user's motions, and 3D dance motion data extracted from a 2D or 3D dance video as an initial value of the seed motion data.

16. The method for generating a dancing avatar according to claim 9 ,

wherein the step of propagating includes calculating

a style focus loss according to the difference between the dance genre of the seed motion data and the determined dance genre, and

when the dance genre of the seed motion data and the selected dance genre are the same, a dance genre-specific loss calculated as the difference between the seed motion data and the dance data, and

a style diversity loss that maximizes the difference between dance data by obtaining dance data previously obtained from the same selected dance genre as the seed motion data, so that repeatedly generated dance data represents various motions,

and back-propagating by adding them to the loss.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (15)