IP Library Granted Patent US 10,776,981
Granted Patent B1
US 10,776,981 · App. 16/551,756 · Granted Sep 15, 2020

Entertaining mobile application for animating a single image of a human body and applying effects

Inventors: Sergei Gorbatyuk (Saint-Petersburg, RU); Nikolai Smirnov (Sochi, RU); Aleksandr Mashrabov (Sochi, RU); Egor Nemchinov (Saint-Petersburg, RU)
Assignee: Snap Inc.
G06T13/40G06F3/0482G06F3/04845G06T7/194G06T7/20G06T7/70G06T19/00G06T2200/24G06T2207/10016G06T2207/30196
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,776,981
App. No.
16/551,756
Granted
Sep 15, 2020
Kind
B1
Abstract

Provided are systems and methods for animating a single image of a human body and applying effects. An example method includes providing, by a computer device, a database of motions; receiving, by a computing device, an input image, the input image including a body of a person; receiving, by the computing device, a user input including a motion selected from the database of motions; segmenting, by the computing device, the input image into a body portion and a background portion; generating, by the computing device and based on the body portion and the selected motion, a video featuring the body of the person repeating the selected motion; and displaying, by the computing device, the generated video; receiving, by a computer device, a further user input including clothes, scene, illumination effect, and additional objects; and, while generating the video, modifying frames of the video based on the further user input.

Claims (117)

1. A method for animating a single image of a human body and applying effects, the method comprising:

providing, by a computing device, a database of motions;

receiving, by the computing device, an input image, the input image including a body of a person;

receiving, by the computing device, a user input, the user input including a motion selected from the database of motions;

segmenting, by the computing device, the input image into a body portion and a background portion, wherein the body portion includes pixels of the input image, the pixels corresponding to the body of the person;

generating, by the computing device and based on the body portion and the selected motion, a video featuring the body of the person repeating the selected motion, wherein the generating the video includes:

generating an image of the body of a person adopting at least one pose from the selected motion;

generating, based on the background portion and the image, a frame; and

modifying the frame by applying one or more visual effects, wherein the one or more visual effects include an illumination effect, a background substitution, a clothes substitution, and an insertion of an ambient object; and

displaying, by the computing device, the generated video.

2. The method of claim 1 , further comprising:

receiving, by the computing device, a further user input, the further input including a name of at least one social network; and

posting, by the computing device, the generated video on a timeline of the user in the at least one social network.

3. The method of claim 1 , further comprising:

prior to generating the video, receiving, by the computing device, a further user input, the further user input including clothes selected from a list; and

while generating the video, substituting clothes of the person in frames of the video with the selected clothes.

4. The method of claim 1 , further comprising:

prior to generating the video, receiving, by the computing device, a further user input, the further user input including a scene selected from a list of scenes; and

while generating the video, substituting the background portion of frames of the video with the selected scene.

5. The method of claim 1 , further comprising:

prior to generating the video, receiving, by the computing device, a further user input, the further user input including the illumination effect selected from a list of illumination effects; and

while generating the video, applying the illumination effect to frames of the video.

6. The method of claim 1 , further comprising:

prior to generating the video, receiving, by the computing device, a further user input, the further user input including an object selected from a list of objects; and

while generating the video, adding the selected object to frames of the video.

7. The method of claim 6 , wherein the addition of the selected object includes attaching the selected object to a hand of the person.

8. The method of claim 1 , further comprising:

prior to generating the video:

fitting the body portion to a hair model;

while generating the video:

detecting positions of key points associated with a head of the person in a frame of the video;

generating, based on the positions of the key points and the hair model, an image of hair of the person; and

inserting the image of the hair in the frame.

9. The method of claim 1 , further comprising:

capturing, by the computing device, a custom video of the person performing a custom motion;

determining, based on the custom video, a set of parameters representing the custom motion; and

adding the set of parameters to the database of motions as a new motion.

10. The method of claim 1 , wherein the displaying the generating video includes:

determining that a pre-determined number of frames of the video has been generated, the pre-determined number being less than a total number of frames in the generated video; and

displaying the pre-determined number of frames of the video while the rest of the frames of the video are being generated.

11. A method for animating a single image of a human body and applying effects, the method comprising:

providing, by a computing device, a database of motions;

receiving, by the computing device, an input image, the input image including a body of a person;

receiving, by the computing device, a user input, the user input including a motion selected from the database of motions;

segmenting, by the computing device, the input image into a body portion and a background portion, wherein the body portion includes pixels of the input image, the pixels corresponding to the body of the person;

generating, by the computing device and based on the body portion and the selected motion, a video featuring the body of the person repeating the selected motion, wherein the generating the video includes:

generating a set of images of the body of a person adopting poses from the selected motion; and

overlaying each of the set of images with a background scene from a set of pre-generated background scenes to generate a series of frames, wherein each of the set of pre-generated background scenes is modified by applying one or more visual effects, wherein the one or more visual effects include an illumination effect, a background substitution, a clothes substitution, and an insertion of an ambient object; and

displaying, by the computing device, the generated video.

12. A system for animating a single image of a human body and applying effects, the system comprising at least one processor and a memory storing processor-executable codes, wherein the at least one processor is configured to implement the following operations upon executing the processor-executable codes:

providing a database of motions;

receiving an input image, the input image including a body of a person;

receiving a user input, the user input including a motion selected from the database of motions;

segmenting the input image into a body portion and a background portion, wherein the body portion includes pixels of the input image, the pixels corresponding to the body of the person;

generating, based on the body portion and the selected motion, a video featuring the body of the person repeating the selected motion, wherein the generating the video includes:

generating an image of the body of a person adopting at least one pose from the selected motion;

generating, based on the background portion and the image, a frame; and

modifying the frame by applying one or more visual effects, wherein the one or more visual effects include an illumination effect, a background substitution, a clothes substitution, and an insertion of an ambient object; and

displaying the generated video.

13. The system of claim 12 , further comprising:

receiving a further user input, the further input including a name of at least one social network; and

posting the generated video on a timeline of the user in the at least one social network.

14. The system of claim 12 , further comprising:

prior to generating the video, receiving a further user input, the further user input including clothes selected from a list; and

while generating the video, substituting clothes of the person in frames of the video with the selected clothes.

15. The system of claim 12 , further comprising:

prior to generating the video, receiving a further user input, the further user input including a scene selected from a list of scenes; and

while generating the video, substituting the background portion of frames of the video with the selected scene.

16. The system of claim 12 , further comprising:

prior to generating the video, receiving a further user input, the further user input including the illumination effect selected from a list of illumination effects; and

while generating the video, applying the illumination effect to frames of the video.

17. The system of claim 12 , further comprising:

prior to generating the video, receiving a further user input, the further user input including an object selected from a list of objects; and

while generating the video, adding the selected object to frames of the video by attaching the selected object to a hand of the person.

18. The system of claim 12 , further comprising:

prior to generating the video:

fitting the body portion to a hair model; and

while generating the video:

detecting positions of key points associated with a head of the person in a frame of the video;

generating, based on the positions of the key points and the hair model, an image of hair of the person; and

inserting the image of the hair into the frame.

19. The system of claim 12 , further comprising:

capturing, by the computing device, a custom video of the person performing a custom motion;

determining, based on the custom video, a set of parameters representing the custom motion; and

adding the set of parameters to the database of motions as a new motion.

20. The method of claim 12 , wherein the displaying the generating video includes:

determining that a pre-determined number of frames of the video has been generated, the pre-determined number being less than a total number of frames in the generated video; and

displaying the pre-determined number of frames of the video while the rest of the frames of the video being generated.

21. A system for animating a single image of a human body and applying effects, the system comprising at least one processor and a memory storing processor-executable codes, wherein the at least one processor is configured to implement the following operations upon executing the processor-executable codes:

providing a database of motions;

receiving an input image, the input image including a body of a person;

receiving a user input, the user input including a motion selected from the database of motions;

segmenting the input image into a body portion and a background portion, wherein the body portion includes pixels of the input image, the pixels corresponding to the body of the person;

generating, based on the body portion and the selected motion, a video featuring the body of the person repeating the selected motion, wherein the generating the video includes:

generating a set of images of the body of a person adopting poses from the selected motion; and

overlaying each of the set of images with a background scene from a set of pre-generated background scenes to generate a series of frames, wherein each of the set of pre-generated background scenes is modified by applying one or more visual effects, wherein the one or more visual effects include an illumination effect, a background substitution, a clothes substitution, and an insertion of an ambient object; and

displaying the generated video.

22. A non-transitory processor-readable medium having instructions stored thereon, which when executed by one or more processors, cause the one or more processors to implement a method for animating a single image of a human body and applying effects, the method comprising:

providing a database of motions;

receiving an input image, the input image including a body of a person;

receiving a user input, the user input including a motion selected from the database of motions;

segmenting the input image into a body portion and a background portion, wherein the body portion includes pixels of the input image, the pixels corresponding to the body of the person;

generating, based on the body portion and the selected motion, a video featuring the body of the person repeating the selected motion, wherein the generating the video includes:

generating an image of the body of a person adopting at least one pose from the selected motion;

generating, based on the background portion and the image, a frame; and

modifying the frame by applying one or more visual effects, wherein the one or more visual effects include an illumination effect, a background substitution, a clothes substitution, and an insertion of an ambient object; and

displaying the generated video.

23. The non-transitory processor-readable medium of claim 22 , wherein the method further comprises:

receiving a further user input, the further input including a name of at least one social network; and

posting the generated video on a timeline of the user in the at least one social network.

24. The non-transitory processor-readable medium of claim 22 , wherein the method further comprises:

prior to generating the video:

fitting the body portion to a hair model;

while generating the video:

detecting positions of key points associated with a head of the person in a frame of the video;

generating, based on the positions of the key points and the hair model, an image of hair of the person; and

inserting the image of the hair in the frame.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 11, 2020
From: AI FACTORY, INC.
To: SNAP INC.
Reel/Frame 051789/0760 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 27, 2019
From: GORBATYUK, SERGEI; SMIRNOV, NIKOLAI; MASHRABOV, ALEKSANDR; NEMCHINOV, EGOR
To: AI FACTORY, INC.
Reel/Frame 050174/0269 →
Continuity (1)
Continuation In Part 16434185 · Jun 7, 2019
Cited By (5)
US 1,096,792 US 12,293,433 US 12,482,131 US 12,597,206 US 12,705,840