System and method for real-time virtual application of makeup effects during live video streaming
An applicator device receives a request from a client device to initiate a communication session and causes a camera of the client device to capture a video of a user of the client device. The applicator device obtains the video of the user. The applicator device tracks a facial region of the user depicted in the video and displays the tracked facial region in a first window. The applicator device determines a target facial feature in the tracked facial region and displays a target sub-region surrounding the target facial feature within the tracked facial region in a second window. The applicator device performs virtual application of a selected cosmetic effect and displays the virtual application of the selected cosmetic effect in the first window and the second window on the applicator device.
1 . A method implemented in an applicator device, comprising:
receiving a request from a client device to initiate a communication session;
causing a camera of the client device to capture a video of a user of the client device;
obtaining the video of the user;
tracking a facial region of the user depicted in the video and displaying the tracked facial region in a first window;
determining a target facial feature in the tracked facial region;
displaying a target sub-region surrounding the target facial feature within the tracked facial region in a second window, wherein the target sub-region around the target facial feature is enlarged in the second window while the tracked facial region is displayed in the first window, wherein displaying the target sub-region comprises fixing the target facial feature within the second window based on predefined control points;
performing virtual application of a selected cosmetic effect; and
displaying the virtual application of the selected cosmetic effect in the first window and the second window on the applicator device.
2 . The method of claim 1 , wherein the virtual application of the selected cosmetic effect is displayed in a corresponding first window displayed in a user interface on the client device, wherein the corresponding first window displays a same content as the first window shown on the applicator device.
3 . The method of claim 1 , wherein one or both the first window and the second window are displayed based on input by a beauty advisor utilizing the applicator device.
4 . The method of claim 1 , wherein determining the target facial feature in the tracked facial region comprises:
displaying the tracked facial region and a cosmetic effects toolbar in the first window;
obtaining a selection of a cosmetic effect from the cosmetic effects toolbar from one of: a beauty advisor utilizing the applicator device or the user of the client device; and
determining the target facial feature based on the selected cosmetic effect.
5 . The method of claim 1 , wherein determining the target facial feature in the tracked facial region comprises:
displaying the tracked facial region in the first window;
obtaining a selection of a facial feature from one of: a beauty advisor utilizing the applicator device or the user of the client device; and
determining the target facial feature based on the selected facial feature.
6 . The method of claim 1 , wherein fixing the location of the target facial feature within the second window based on the predefined control points comprises:
determining a perspective transformation matrix according to the predefined control points; and
applying the perspective transformation matrix to the target facial feature.
7 . The method of claim 1 , further comprising causing the client device to display a boundary around the tracked facial region in the first window.
8 . The method of claim 1 , further comprising causing the client device to display a boundary around the target sub-region surrounding the target facial feature.
9 . A system comprising:
a memory storing instructions;
a processor coupled to the memory and configured by the instructions to at least:
receive a request from a client device to initiate a communication session;
cause a camera of the client device to capture a video of a user of the client device;
obtain the video of the user;
track a facial region of the user depicted in the video and display the tracked facial region in a first window;
determine a target facial feature in the tracked facial region;
display a target sub-region surrounding the target facial feature within the tracked facial region in a second window, wherein the target sub-region around the target facial feature is enlarged in the second window while the tracked facial region is displayed in the first window, wherein displaying the target sub-region comprises fixing the target facial feature within the second window based on predefined control points;
perform virtual application of a selected cosmetic effect; and
display the virtual application of the selected cosmetic effect in the first window and the second window.
10 . The system of claim 9 , wherein the virtual application of the selected cosmetic effect is displayed in a corresponding first window displayed in a user interface on the client device, wherein the corresponding first window displays a same content as the first window shown on the system.
11 . The system of claim 9 , wherein one or both the first window and the second window are displayed based on input by a beauty advisor utilizing the system.
12 . The system of claim 9 , wherein the processor is configured to determine the target facial feature in the tracked facial region by:
displaying the tracked facial region and a cosmetic effects toolbar in the first window;
obtaining a selection of a cosmetic effect from the cosmetic effects toolbar from one of: a beauty advisor utilizing the system or the user of the client device; and
determining the target facial feature based on the selected cosmetic effect.
13 . A non-transitory computer-readable storage medium storing instructions to be implemented by an applicator device having a processor, wherein the instructions, when executed by the processor, cause the applicator device to at least:
receive a request from a client device to initiate a communication session;
cause a camera of the client device to capture a video of a user of the client device;
obtain the video of the user;
track a facial region of the user depicted in the video and display the tracked facial region in a first window;
determine a target facial feature in the tracked facial region;
display a target sub-region surrounding the target facial feature within the tracked facial region in a second window, wherein the target sub-region around the target facial feature is enlarged in the second window while the tracked facial region is displayed in the first window, wherein displaying the target sub-region comprises fixing the target facial feature within the second window based on predefined control points;
perform virtual application of a selected cosmetic effect; and
display the virtual application of the selected cosmetic effect in the first window and the second window.
14 . The non-transitory computer-readable storage medium of claim 13 , wherein the virtual application of the selected cosmetic effect is displayed in a corresponding first window displayed in a user interface on the client device, wherein the corresponding first window displays a same content as the first window shown on the applicator device.
15 . The non-transitory computer-readable storage medium of claim 13 , wherein one or both the first window and the second window are displayed based on input by a beauty advisor utilizing the applicator device.