Granted Patent B2

US 12,646,240 · App. 18/525,291 · Granted Jun 2, 2026

Inventors: Gurunandan Krishnan Gorumkonda (Kirkland, WA); Shree K. Nayar (New York, NY)

Assignee: Snap Inc.

G06T13/40G06T13/205

Abstract

Claims (87)

1 . A method comprising:

receiving, by a processor, a real-time acoustic signal comprising a plurality of acoustic segments;

generating using a music analyzer neural network a tempo value and a dance energy category for each of the acoustic segments;

selecting a plurality of dance tracks based on the tempo value and the dance energy category, wherein each of the plurality of dance tracks comprises a plurality of dance segments including a plurality of movements that are sequential;

generating using the dance tracks a first dance sequence for a first avatar and a second dance sequence for a second avatar, wherein generating the first dance sequence and the second dance sequence comprises:

generating a first dance segment by

selecting a first starting movement of the plurality of movements in a first track of the plurality of tracks, and

selecting a first transition movement in the plurality of movements in the first track, wherein the first dance segment starts at the first starting movement and ends at the first transition movement, and

generating a second dance segment by

identifying a second starting movement in a second dance track of the plurality of tracks that is similar to the first transition movement within a predetermined threshold, wherein the second dance segment starts at the second starting movement,

wherein the first dance sequence and the second dance sequence include the first dance segment and the second dance segment;

generating a first real-time animation of the first avatar and a second real-time animation of a second avatar based on the first dance sequence and the second dance sequence and a plurality of avatar characteristics associated with the first avatar and the second avatar; and

causing to be displayed on a first client device the first real-time animation of the first avatar and the second real-time animation of the second avatar.

2 . The method of claim 1 , wherein generating the first dance sequence and the second dance sequence further comprises:

selecting a second transition movement in the plurality of movements in the second track, wherein the second dance segment ends at the second transition movement; and

generating a third dance segment by

identifying a third starting movement in a third dance track of the plurality of tracks that is similar to the second transition movement within the predetermined threshold, wherein the third dance segment starts at the third starting movement,

wherein the first dance sequence and the second dance sequence include the third dance segment.

3 . The method of claim 1 , wherein the tempo value comprises a value indicating beats per minute, and wherein the dance energy category is one of: idle, slow, lively, or vigorous.

4 . The method of claim 3 , further comprising:

training the music analyzer neural network, wherein training the music analyzer neural network comprises:

receiving a plurality of test acoustic signals including a plurality of test acoustic segments,

determining a plurality of test tempo values associated with the test acoustic segments, and

associating each of the test acoustic segments with one of a plurality of test dance energy categories, wherein the test dance energy categories comprise idle, slow, lively, and vigorous.

5 . The method of claim 4 , wherein associating each of the test acoustic segments with one of the test dance energy categories is based on music features of the test acoustic segments, wherein the music features comprise frequency response, chromagram, tempogram, or any combination thereof.

6 . The method of claim 5 , wherein generating using the music analyzer neural network a tempo value and a dance energy category further comprises:

generating the tempo value and the dance energy category for each of the acoustic segments based on the test tempo values and the test dance energy categories associated with the test acoustic segments.

7 . The method of claim 4 , further comprising:

receiving a plurality of test videos including a dancer performing dance movements and the test acoustic signals, the test videos comprising a plurality of test video segments, wherein each of the test video segments comprises a plurality of test video frames; and

determining body poses for each of the test video frames using skeletal approximation of the dancer, wherein the body poses comprise joint positions and angles.

8 . The method of claim 7 , further comprising:

mapping the body poses for each of the test video frames to a plurality of avatar body poses associated with an avatar skeleton; and

generating a plurality of avatar test videos using the plurality of avatar body poses.

9 . The method of claim 8 , wherein the plurality of avatar test videos comprise the plurality of dance tracks.

10 . The method of claim 1 , wherein generating the first dance sequence and the second dance sequence further comprises:

generating the first dance sequence and the second dance sequence based on a position of the first avatar displayed on the first client device and a position of the second avatar displayed on the first client device to prevent an overlapping display of the first avatar and the second avatar.

11 . The method of claim 10 , wherein the first transition movement in the first dance segment is selected to prevent the overlapping display of the first avatar and the second avatar.

12 . The method of claim 10 , wherein the second starting movement in the second dance track is identified to prevent the overlapping display of the first avatar and the second avatar.

13 . The method of claim 1 , further comprising:

causing to be displayed on a second client device the real-time animation of the first avatar and the second avatar.

14 . The method of claim 1 , wherein the first client device is associated with a first user and the second client device is associated with a second user,

wherein the first user is associated with the first avatar, and the second user is associated with the second avatar.

15 . A system comprising:

a processor; and

a memory storing instructions that, when executed by the processor, cause the system to perform operations comprising:

receiving a real-time acoustic signal comprising a plurality of acoustic segments;

generating using a music analyzer neural network a tempo value and a dance energy category for each of the acoustic segments;

selecting a plurality of dance tracks based on the tempo value and the dance energy category, wherein each of the plurality of dance tracks comprises a plurality of dance segments including a plurality of movements that are sequential;

generating using the dance tracks a first dance sequence for a first avatar and a second dance sequence for a second avatar, wherein generating the first dance sequence and the second dance sequence comprises:

generating a first dance segment by

selecting a first starting movement of the plurality of movements in a first track of the plurality of tracks, and

selecting a first transition movement in the plurality of movements in the first track, wherein the first dance segment starts at the first starting movement and ends at the first transition movement, and

generating a second dance segment by

identifying a second starting movement in a second dance track of the plurality of tracks that is similar to the first transition movement within a predetermined threshold, wherein the second dance segment starts at the second starting movement,

wherein the first dance sequence and the second dance sequence include the first dance segment and the second dance segment;

generating a first real-time animation of the first avatar and a second real-time animation of a second avatar based on the first dance sequence and the second dance sequence and a plurality of avatar characteristics associated with the first avatar and the second avatar; and

causing to be displayed on a first client device the first real-time animation of the first avatar and the second real-time animation of the second avatar.

16 . The system of claim 15 , wherein the tempo value comprises a value indicating beats per minute, and wherein the dance energy category is one of: idle, slow, lively, or vigorous.

17 . The system of claim 16 , wherein the system to perform operations further comprising:

training the music analyzer neural network, wherein training the music analyzer neural network comprises:

receiving a plurality of test acoustic signals including a plurality of test acoustic segments,

determining a plurality of test tempo values associated with the test acoustic segments, and

associating each of the test acoustic segments with one of a plurality of test dance energy categories, wherein the test dance energy categories comprise idle, slow, lively, and vigorous.

18 . The system of claim 17 , wherein generating using the music analyzer neural network a tempo value and a dance energy category further comprises:

generating the tempo value and the dance energy category for each of the acoustic segments based on the test tempo values and the test dance energy categories associated with the test acoustic segments.

19 . The system of claim 17 , wherein the system to perform operations further comprising:

receiving a plurality of test videos including a dancer performing dance movements and the test acoustic signals, the test videos comprising a plurality of test video segments, wherein each of the test video segments comprises a plurality of test video frames;

determining body poses for each of the test video frames using skeletal approximation of the dancer, wherein the body poses comprise joint positions and angles;

mapping the body poses for each of the test video frames to a plurality of avatar body poses associated with an avatar skeleton; and

generating a plurality of avatar test videos using the plurality of avatar body poses, wherein the plurality of avatar test videos comprise the plurality of dance tracks.

20 . The system of claim 15 , wherein generating the first dance sequence and the second dance sequence further comprises:

generating the first dance sequence and the second dance sequence based on a position of the first avatar displayed on the first client device and a position of the second avatar displayed on the first client device to prevent an overlapping display of the first avatar and the second avatar,

wherein the first transition movement in the first dance segment is selected to prevent the overlapping display of the first avatar and the second avatar, or

wherein the second starting movement in the second dance track is identified to prevent the overlapping display of the first avatar and the second avatar.

21 . A non-transitory computer-readable storage medium having stored thereon instructions that when executed by a processor, cause the processor to perform operations comprising:

receiving a real-time acoustic signal comprising a plurality of acoustic segments;

generating using a music analyzer neural network a tempo value and a dance energy category for each of the acoustic segments;

selecting a plurality of dance tracks based on the tempo value and the dance energy category, wherein each of the plurality of dance tracks comprises a plurality of dance segments including a plurality of movements that are sequential;

generating using the dance tracks a first dance sequence for a first avatar and a second dance sequence for a second avatar, wherein generating the first dance sequence and the second dance sequence comprises:

generating a first dance segment by

selecting a first starting movement of the plurality of movements in a first track of the plurality of tracks, and

selecting a first transition movement in the plurality of movements in the first track, wherein the first dance segment starts at the first starting movement and ends at the first transition movement, and

generating a second dance segment by

identifying a second starting movement in a second dance track of the plurality of tracks that is similar to the first transition movement within a predetermined threshold, wherein the second dance segment starts at the second starting movement,

wherein the first dance sequence and the second dance sequence include the first dance segment and the second dance segment;

generating a first real-time animation of the first avatar and a second real-time animation of a second avatar based on the first dance sequence and the second dance sequence and a plurality of avatar characteristics associated with the first avatar and the second avatar; and

causing to be displayed on a first client device the first real-time animation of the first avatar and the second real-time animation of the second avatar.

Assignments (1)

Continuity (2)

References Cited (229)