Computing devices, computer program products, and methods for efficient rendering pipeline for makeup including VTO UI option configuration/selection and looks
There is provided device, system and method embodiments for streamlining the applying of an effect (e.g. a virtual try on (“VTO”) effect) to an object appearing in a sequence of video frames such as for video chat, conferencing or teleconsultation applications. Embodiments and/or features of a user interface such as for video chatting/conferencing or teleconsultation are provided. UI features can include live and preview modes, recommendations using predefine looks, look refinement, look ordering, and access to e-commerce purchase interface. In an embodiment, operations of i) effect rendering, and ii) object landmark determining are performed in parallel where effect rendering applies an effect in association with landmarks determined for the object to define a sequence of output video frames with the effect applied.
1 . A computing device comprising a processor and a non-transient storage device storing computer executable instructions for execution by the processor to cause the computing device to:
provide a video chat, video conference, or teleconsultation application that streams video frames to at least one other computing device, the application integrated with a virtual try on (VTO) pipeline that applies one or more effects to the video frames-to be streamed, wherein the VTO pipeline comprises an object detection function and an effect rendering function configured to execute in a frame-wise parallel manner to optimize applying of the one or more effects to the video frames;
wherein the application is configured to present a graphical user interface (GUI) via a display screen, the GUI having:
a preview mode that receives input that selects the one or more effects from a plurality of effect options, the preview mode configured to present the one or more effects applied to a sequence of preview frames via the display screen and without streaming the preview frames to the at least one other computing device; and
a live mode configured to apply the one or more effects using the object detection function and the effect rendering function executing in the frame-wise parallel manner and further configured to stream the video frames to which the one or more effects are applied in the live mode; and
wherein subsets of the plurality of effect options are grouped as respective looks, and the GUI provides a look selection interface displaying an option to select one of the looks; and
wherein each of one or more effects grouped for a look is associated with a respective product and, responsive to a selection of one of the looks, the GUI is configured to present a selected look interface having respective sub interfaces for each respective effect of the one or more effects grouped for the look, each respective sub interface providing i) product information for the respective product associated with a respective effect and ii) a buy now control to initiate an ecommerce interface to purchase the respective product.
2 . The computing device of claim 1 , wherein further provides iii) a refinement control to refine a selection of the respective effect.
3 . The computing device of claim 2 , wherein the refinement control is configured for selecting a different product for the respective effect so that the respective effect provides a virtual try on of the different product.
4 . The computing device of claim 2 , wherein the refinement control is configured to turn off the respective effect so that the respective effect is not applied to the video frames.
5 . The computing device of claim 1 , wherein the GUI is configured to provide access to the ecommerce interface to initiate purchase of the respective product.
6 . The computing device of claim 1 , wherein the plurality of effect options are any of makeup effects, hair effects or nail effects.
7 . The computing device of claim 1 , wherein the look selection interface orders a presentation of the respective looks for selection to make user recommendations.
8 . The computing device of claim 7 , wherein the order of the respective looks is determined in accordance with user information processing, the user information comprising one or more of: a users' existing makeup, a user's face shape, a user's skin tone, or any respective looks recently selected.
9 . The computing device of claim 8 , wherein one or more of the users' existing makeup, the user's face shape, or the user's skin tone is determined through processing the sequence of preview frames.
10 . The computing device of claim 7 , wherein responsive to a user favorite selection of a respective look, the look selection interface prioritizes an order of the presentation of the respective looks such that favorite looks are ordered ahead of other respective looks and the other respective looks are ordered in based on user information processing.
11 . The computing device of claim 1 , wherein the preview mode is invokable prior to a user joining a particular instance of the video chat or the conference, and the preview mode is integrated with a video chat or conference user interface to configure video and audio controls for the instance.
12 . The computing device of claim 1 , wherein the preview mode is invokable during participation in an instance of the video chat or the conference, and in response to a user invoking the preview mode, stops streaming of the video frames.
13 . A computer program product comprising a non-transient storage device storing computer readable instructions that when executed by a processor cause a computing device to:
execute a video chat, video conference, or teleconsultation application that streams video frames to at least one other computing device, the application integrated with a virtual try on (VTO) pipeline that applies one or more effects to the video frames, wherein the VTO pipeline comprises an object detection function and an effect rendering function configured to execute in a frame-wise parallel manner to optimize applying of the one or more effects to the video frames;
wherein the application is configured to present a graphical user interface (GUI) via a display screen, the GUI having:
a preview mode that receives input that selects the one or more effects from a plurality of effect options, the preview mode configured to present the one or more effects applied to a sequence of preview frames via the display screen and without streaming the preview frames to the at least one other computing device; and
a live mode configured to apply the one or more effects using the object detection function and the effect rendering function executing in the frame-wise parallel manner and further configured to stream the video frames to which the one or more effects are applied in the live mode; and
wherein subsets of the plurality of effect options are grouped as respective looks, and the GUI provides a look selection interface displaying an option to select one of the looks; and
wherein each of one or more effects grouped for a look is associated with a respective product and, responsive to a selection of one of the looks, the GUI is configured to present a selected look interface having respective sub interfaces for each respective effect of the one or more effects grouped for the look, each respective sub interface providing i) product information for the respective product associated with a respective effect and ii) a buy now control to initiate an ecommerce interface to purchase the respective product.
14 . A computer implemented method comprising:
executing, by a computing device, a video chat, video conference, or teleconsultation application that streams video frames to at least one other computing device, the application integrated with a virtual try on (VTO) pipeline that applies one or more effects to the video frames, wherein the VTO pipeline comprises an object detection function and an effect rendering function that execute in a frame-wise parallel manner to optimize applying of the one or more effects to the video frames;
wherein the application is configured to present a graphical user interface (GUI) via a display screen, the GUI having a preview mode and a live mode;
receiving, by the computing device and during the preview mode, input to select the one or more effects from a plurality of effect options, the preview mode presenting the one or more effects applied to a sequence of preview frames via the display screen and without streaming the preview frames to the at least one other computing device; and
applying, by the computing device and during the live mode, the one or more effects using the object detection function and the effect rendering function executing in the frame-wise parallel manner and streaming, by the computing device and during the live mode, the video frames to which the one or more effects are applied in the live mode; and
grouping subsets of the plurality of effect options as respective looks, and the GUI providing a look selection interface displaying an option to select one of the looks, wherein each of one or more effects grouped for a look is associated with a respective product; and
responsive to a selection of one of the looks, the GUI is configured to present a selected look interface having respective sub interfaces for each respective effect of the one or more effects grouped for the look, each respective sub interface providing i) product information for a respective product associated with the respective effect and ii) a buy now control to initiate an ecommerce interface to purchase the respective product.