IP Library Granted Patent US 11,211,096
Granted Patent B2
US 11,211,096 · App. 16/597,637 · Granted Dec 28, 2021

Video manipulation with face replacement

Inventors: Jokubas Zukerman (San Francisco, CA); Marco Paglia (San Francisco, CA); Chad Sager (San Francisco, CA); Andrew Poes (Millbrae, CA); Maegan Clawges (San Francisco, CA); Ivan Evfimiou (Burlingame, CA); Gregory Foster (San Francisco, CA); Samuel Keene (Sydney, AU); John Gregg (Seattle, WA); Reed Morse (San Francisco, CA)
Assignee: Google LLC
G11B27/036G06K9/00228G06K9/00765G06K9/00979G06T11/40G06T11/60H04N21/8146
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,211,096
App. No.
16/597,637
Granted
Dec 28, 2021
Kind
B2
Abstract

A user device provides a user interface for video manipulation with face replacement. A method of implementations includes accessing a video comprising a plurality of frames that comprise one or more faces, providing a plurality of stickers comprising alternate face graphics for the one or more faces, receiving, via a user interface of a user device, user selection of one of the stickers and a selected face of the one or more faces, accessing a plurality of face frame sequences of the video, wherein each face frame sequence is a sequence of frames of the video comprising the selected face of the one or more faces, replacing the selected face with the selected sticker in each face frame of the plurality of face frame sequences, and animating the selected sticker in the each face frame of the plurality of face frame sequences.

Claims (58)

1. A method comprising:

accessing a video comprising a plurality of frames that comprise one or more faces;

providing a plurality of stickers comprising alternate face graphics for the one or more faces;

receiving, via a user interface of a user device, user selection of one of the stickers and a selected face of the one or more faces;

accessing a plurality of face frame sequences of the video, wherein each face frame sequence is a sequence of frames of the video comprising the selected face of the one or more faces;

replacing the selected face with the selected sticker in a first face frame sequence of the plurality of face frame sequences and in a second face frame sequence of the plurality of face frame sequences, wherein the first face frame sequence and the second face frame sequence are separated by a third face frame sequence in which the selected face is not replaced; and

animating the selected sticker in the first face frame sequence and the second face frame sequence by using an animation file as the selected sticker.

2. The method of claim 1 , wherein accessing the plurality of face frame sequences comprises receiving the plurality of face frame sequences from a server device.

3. The method of claim 1 , wherein each face frame sequence of the plurality of face frame sequences is a continuous sequence of frames.

4. The method of claim 1 , further comprising receiving user selection of a frame in at least one of the plurality of face frame sequences, wherein the selected frame is the first frame in a combined face frame sequence comprising a merger of selected face frame sequences of the plurality of face frame sequences.

5. The method of claim 4 , further comprising:

receiving, from a server device, a manipulated face frame sequence, wherein the selected face is replaced with the selected sticker in each frame of the manipulated face frame sequence relative to the combined face frame sequence; and

presenting the manipulated frame sequence.

6. The method of claim 4 , further comprising:

receiving selection of a second sticker; and

sending, to a server device, a user request to replace the selected face with the second sticker in each frame of the combined face frame sequence.

7. The method of claim 4 , further comprising:

receiving selection of a second sticker and a second frame, wherein the second frame is after the selected frame in the combined face frame sequence; and

sending, to a server device, a user request to replace the selected sticker with the second sticker in the second frame and each frame after the second frame in the combined face frame sequence.

8. The method of claim 4 , further comprising:

receiving selection of a second sticker and a second frame, wherein the second frame is before the selected frame in the combined face frame sequence; and

sending, to the server device, a user request to replace the selected sticker with the second sticker in the second frame and each frame between the second frame and the selected frame in the combined face frame sequence.

9. The method of claim 4 , further comprising:

providing a plurality of audio tracks;

receiving selection of one of the audio tracks; and

sending, to a server device, a request to overlay the selected audio track on the combined face frame sequence.

10. The method of claim 1 , wherein animating the selected sticker in the each face frame of the plurality of face frame sequences further comprises manipulating a size of the selected sticker in consecutive face frames.

11. A system comprising:

a memory; and

a processor communicably coupled to the memory, the processor to:

access a video comprising a plurality of frames that comprise one or more faces;

provide a plurality of stickers comprising alternate face graphics for the one or more faces;

receive, via a user interface of a user device, user selection of one of the stickers and a selected face of the one or more faces;

access a plurality of face frame sequences of the video, wherein each face frame sequence is a sequence of frames of the video comprising the selected face of the one or more faces;

replacing the selected face with the selected sticker a first face frame sequence of the plurality of face frame sequences and in a second face frame sequence of the plurality of face frame sequences, wherein the first face frame sequence and the second face frame sequence are separated by a third face frame sequence in which the selected face is not replaced; and

animate the selected sticker in the first face frame sequence and the second face frame sequence by using an animation file as the selected sticker.

12. The system of claim 11 , wherein the processor to access the plurality of face frame sequences further comprises the processor to receive the plurality of face frame sequences from a server device.

13. The system of claim 11 , wherein each face frame sequence of the plurality of face frame sequences is a continuous sequence of frames.

14. The system of claim 11 , wherein the processor is further to receive user selection of a frame in at least one of the plurality of face frame sequences, wherein the selected frame is the first frame in a combined face frame sequence comprising a merger of selected face frame sequences of the plurality of face frame sequences.

15. The system of claim 14 , wherein the processor is further to:

receive, from a server device, a manipulated face frame sequence, wherein the selected face is replaced with the selected sticker in each frame of the manipulated face frame sequence relative to the combined face frame sequence; and

present the manipulated frame sequence.

16. The system of claim 14 , wherein the processor is further to:

receive selection of a second sticker; and

send, to a server device, a user request to replace the selected face with the second sticker in each frame of the combined face frame sequence.

17. A non-transitory computer readable storage medium storing executable computer program instructions that when executed cause a processor to:

access a video comprising a plurality of frames that comprise one or more faces;

provide a plurality of stickers comprising alternate face graphics for the one or more faces;

receive, via a user interface of a user device, user selection of one of the stickers and a selected face of the one or more faces;

access a plurality of face frame sequences of the video, wherein each face frame sequence is a sequence of frames of the video comprising the selected face of the one or more faces;

replace the selected face with the selected sticker in a first face frame sequence of the plurality of face frame sequences and in a second face frame sequence of the plurality of face frame sequences, wherein the first face frame sequence and the second face frame sequence are separated by a third face frame sequence in which the selected face is not replaced; and

animate the selected sticker in the first face frame sequence and the second face frame sequence by using an animation file as the selected sticker.

18. The non-transitory computer readable storage medium of claim 17 , wherein the processor to access the plurality of face frame sequences further comprises the processor to receive the plurality of face frame sequences from a server device.

19. The non-transitory computer readable storage medium of claim 17 , wherein each face frame sequence of the plurality of face frame sequences is a continuous sequence of frames.

20. The non-transitory computer readable storage medium of claim 17 , wherein the instructions further cause the processor to receive user selection of a frame in at least one of the plurality of face frame sequences, wherein the selected frame is the first frame in a combined face frame sequence comprising a merger of selected face frame sequences of the plurality of face frame sequences.

21. The non-transitory computer readable storage medium of claim 20 , wherein the instructions further cause the processor to:

receive, from a server device, a manipulated face frame sequence, wherein the selected face is replaced with the selected sticker in each frame of the manipulated face frame sequence relative to the combined face frame sequence; and

present the manipulated frame sequence.

Assignments (2)
CHANGE OF NAME Recorded Oct 11, 2019
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 050712/0987 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 10, 2019
From: ZUKERMAN, JOKUBAS; PAGLIA, MARCO; SAGER, CHAD; POES, ANDREAS; CLAWGES, MAEGAN; EVFIMIOU, IVAN; FOSTER, GREGORY; KEENE, SAMUEL; GREGG, JOHN; MORSE, REED
To: GOOGLE INC.
Reel/Frame 050674/0506 →
Continuity (2)
Continuation 15394437 · Dec 29, 2016
Related Publication 20200043527A1 · Feb 6, 2020