Systems and methods for using machine learning models to effect virtual try-on and styling on actual users
Disclosed are example embodiments of systems and methods for virtual try-on of articles of clothing. An example method of virtual try-on of articles of clothing includes selecting a garment from a pre-existing database. The method also includes loading a photo of a source model wearing the selected garment. Additionally, the method includes generating a semantic segmentation of the model image. The method also includes extracting the selected garment from the photo of the model. Additionally, the method includes determining a correspondence between a target model and the source model by performing a feature point detection and description of the target model and the source model, and performing feature matching and correspondence validation. The method also includes performing garment warping and alignment of the extracted garment. Additionally, the method includes overlaying and rendering the garment.
1 . A method of styling virtual try-on of articles of clothing, the method comprising:
identifying a subset of a large dataset of images of garments, wherein the subset of the large dataset of images are labelled with a specific style based on a particular type of clothing and for each image depicting a garment of the particular type of clothing, identifying a set of landmarks on the garment, each landmark having positional information about the garment; generating a set of semantic tags associated with the set of landmarks based on the particular type of clothing; and generating a set of styling options associated with each semantic tag;
generating a digital taxonomy having one or more labels associated with each image in the subset of the large dataset of images, wherein at least one of the one or more labels corresponds to the specific style;
training a style classifier using the digital taxonomy applied to the subset of the large dataset of images of garments; and
generating a first dataset based on executing the trained style classifier on the large dataset of images of garments, the first dataset comprising a plurality of images of garments meeting a threshold probability of depicting the specific style;
generating a second dataset comprising one or more images of garments having a probability of depicting the specific style below the threshold probability; and
providing the second dataset in a user interface configured to enable labelling of the one or more images of garments having the probability of depicting the specific style below the threshold probability.
2 . The method of claim 1 , wherein a first semantic tag of the set of semantic tags is associated with a first landmark of the set of landmarks on the garment, the first semantic tag comprising a collar, and wherein the set of styling options associated with the first semantic tag comprises folded, popped, bow tied, not tied, and untied.
3 . The method of claim 1 , wherein a second semantic tag of the set of semantic tags is associated with a second landmark of the set of landmarks on the garment, the second semantic tag comprising a sleeve, and wherein the set of styling options associated with the second semantic tag comprises down, pushed up, rolled, and cuffed.
4 . The method of claim 1 , wherein a third semantic tag of the set of semantic tags is associated with a third landmark of the set of landmarks on the garment, the third semantic tag comprising a tuck, and wherein the set of styling options associated with the third semantic tag comprises untucked, tucked, front tucked, and half tucked.
5 . The method of claim 1 , wherein a fourth semantic tag of the set of semantic tags is associated with a fourth landmark of the set of landmarks on the garment, the fourth semantic tag comprising a button, and wherein the set of styling options associated with the fourth semantic tag comprises top unbuttoned, fully unbuttoned, top and bottom unbuttoned, deep unbuttoned, fully buttoned, and bottom unbuttoned.
6 . The method of claim 1 , wherein a fifth semantic tag of the set of semantic tags is associated with a fifth landmark of the set of landmarks on the garment, the fifth semantic tag comprising a pant leg, and wherein the set of styling options associated with the fifth semantic tag comprises down, cuffed, and rolled.
7 . A method of e-mail marketing regarding styled digital articles of clothing, the method comprising:
generating an e-mail marketing campaign associated with a user based on one or more user interactions with a virtual try-on system;
determining a plurality of content components in a customized e-mail associated with the e-mail marketing campaign associated with the user, wherein at least one content component in the customized e-mail is selected from a large dataset of images of garments based on the one or more user interactions with the virtual try-on system;
assembling the plurality of content components to generate the customized e-mail associated with the e-mail marketing campaign associated with the user; and
sending the customized e-mail to the user.
8 . The method of claim 7 , the method further comprising:
retrieving an image of a garment overlaid and rendered on a target model based on a user uploaded photo from a database; and
providing the image of the garment overlaid and rendered on the target model based on the user uploaded photo as a selected content component in the plurality of content components determined for the customized e-mail associated with the e-mail marketing campaign associated with the user.
9 . The method of claim 7 , wherein the one or more user interactions with the virtual try-on system comprise at least one of:
selecting a garment from a catalog of an online retailer;
causing a rendering of the garment onto a target model based on a physique of the user;
configuring the rendering of the garment based on a set of styling options; and
storing the configured rendering of the garment onto the target model in the virtual try-on system associated with the user.
10 . The method of claim 7 , wherein determining a plurality of content components in a customized e-mail associated with the e-mail marketing campaign associated with the user comprises:
retrieving an image of a recommended garment overlaid and rendered on a target model based on a user uploaded photo from a database; and
providing the image of the recommended garment overlaid and rendered on the target model based on the user uploaded photo as a recommended content component in the plurality of content components determined for the customized e-mail associated with the e-mail marketing campaign associated with the user.
11 . The method of claim 7 , wherein determining a plurality of content components in a customized e-mail associated with the e-mail marketing campaign associated with the user comprises:
retrieving an image of a user-saved garment overlaid and rendered on a target model based on a user uploaded photo from a database; and
providing the image of the user-saved garment overlaid and rendered on the target model based on the user uploaded photo as a recommended content component in the plurality of content components determined for the customized e-mail associated with the e-mail marketing campaign associated with the user.
12 . The method of claim 11 , the method further comprising:
modifying the image of the user-saved garment overlaid and rendered on a target model based on one or more styling options associated with the user-saved garment; and
providing the modified image of the user-saved garment overlaid and rendered on the target model based on the user uploaded photo as the recommended content component in the plurality of content components determined for the customized e-mail associated with the e-mail marketing campaign associated with the user.
13 . The method of claim 7 , wherein determining a plurality of content components in a customized e-mail associated with the e-mail marketing campaign associated with the user comprises:
retrieving a link to a garment from a catalog of an online retailer, the link enabling the user to purchase the garment from the online retailer;
causing a rendering of the garment onto a target model based on a physique of the user;
configuring the rendering of the garment based on a set of styling options; and
providing the configured rendering of the garment onto the target model and the link to the garment in the catalog of the online retailer as a determined content component in the customized e-mail.
14 . A method of styling multiple digital fashion items, the method comprising:
selecting first data representing a first garment from a pre-existing database of a virtual try-on system;
loading a first photo of a source model wearing the first garment;
selecting second data representing a second garment from the pre-existing database;
loading a second photo of the source model wearing the second garment;
generating a semantic segmentation of the first photo and the second photo of the source model;
extracting third data representing the first garment and the second garment from the first photo and the second photo of the source model based on the semantic segmentation;
determining a correspondence between a target model from a user uploaded photo and the source model by:
performing a feature point detection of the target model and the source model, and
performing feature matching and correspondence validation;
performing garment warping and alignment of the extracted third data representing the first garment and the second garment onto the target model; and
overlaying and rendering the first garment and the second garment on the user uploaded photo for display in the virtual try-on system based on the determined correspondence between the target model and the source model.
15 . The method of claim 14 , wherein selecting first data representing the first garment from the pre-existing database comprises selecting the first garment from a plurality of garments based on category, style, color, or brand.
16 . The method of claim 14 , wherein selecting second data representing the second garment from the pre-existing database comprises:
generating a plurality of recommended garments based on the first garment;
retrieving a set of images of the plurality of recommended garments;
providing the set of images of the plurality of recommended garments for display to a user;
receiving a selection of an image of a recommended garment from the set of images; and
determining the second data representing the second garment based on the selection of the image of the recommended garment.
17 . The method of claim 14 , further comprising:
generating a plurality of recommended accessory options based on the first garment and the second garment;
retrieving a set of images of the plurality of recommended accessory options;
providing the set of images of the plurality of recommended accessory options for display to a user;
receiving a selection of an image of a recommended accessory option from the set of images;
causing a rendering of the recommended accessory option on the user uploaded photo based on the determined correspondence between the target model and the source model; and
storing the rendering of the recommended accessory option on the user uploaded photo in the pre-existing database of the virtual try-on system.
18 . The method of claim 14 , the method further comprising:
retrieving a link to a third garment from a catalog of an online retailer, the link enabling a user to purchase the third garment from the online retailer;
causing a rendering of the third garment onto the target model, the third garment layered onto the first garment and the second garment based on a physique of the user;
configuring the rendering of the third garment based on a set of styling options;
providing the configured rendering of the third garment onto the user uploaded photo based on the determined correspondence between the target model and the source model; and
storing the configured rendering of the third garment onto the user uploaded photo and the link to the third garment in the catalog of the online retailer in the pre-existing database of the virtual try-on system.
19 . The method of claim 18 , wherein configuring the rendering of the third garment based on a set of styling options comprises selecting one or more layering stylings such as draping and using layered looks.