Method and system for personalized outfit compatibility prediction
Unlike visual similarity, visual compatibility is a complex concept. Existing approaches for outfit compatibility prediction does not focus on methods with personalization. The present disclosure proposes a novel approach to model the user's preference for different styles. The outfit compatibility prediction module is a critical component of an outfit recommendation system. An outfit is said to be compatible if all the items are visually compatible and match the user's preferences. The present disclosure represents the outfit as a graph and uses Graph Neural Network (GNN) with attention mechanism to capture the inter-relationship between the items. A graph read-out layer generates the final outfit embedding. The proposed approach efficiently models the preferences of the users for different styles. Finally, the outfit compatibility score is generated by computing the similarity between the outfit embedding and the user embedding.
1 . A processor implemented method, the method comprising:
receiving, via one or more hardware processors, a plurality of images pertaining to a plurality of outfits, wherein the outfit defines a set of fashion items representing a particular style;
extracting, via the one or more hardware processors, a visual embedding corresponding to each of the plurality of outfit using a visual feature extractor, wherein each of the visual embedding pertains to each of the plurality of images associated with each of the plurality of outfits;
generating, via the one or more hardware processors, a fully connected graph for each of the plurality of outfits based on the visual embedding using a Graph Neural Network (GNN) with an attention mechanism;
generating, via the one or more hardware processors, an outfit embedding based on fully connected graph associated with each of the plurality of outfits using a graph read out layer associated with the GNN, wherein the visual feature extractor, the GNN with the attention mechanism and feature transformation network are trained using a plurality of positive outfits and a plurality of negative outfits corresponding to each of the plurality of users until a minimum ranking loss is obtained, and a dot attention mechanism updates the embeddings of the nodes and DOT-GNN uses the dot attention mechanism to capture visual interaction between items;
simultaneously generating, via the one or more hardware processors, a style-specific user embedding corresponding to each of a plurality of users based on user information and style information corresponding to each of the plurality of users using the feature transformation network, wherein the user embedding using multiple learnable style embeddings, producing style-conditioned embeddings for each user;
computing, via the one or more hardware processors, a visual compatibility score between each of a plurality of outfit embedding and each of the plurality of style-specific user embedding using a similarity based matching technique; and
predicting, via the one or more hardware processors, a visually compatible outfit for each of the plurality of users based on the computed visual compatibility score, wherein an outfit with the visual compatibility score above a predefined threshold is selected.
2 . The processor implemented method of claim 1 , wherein the fully connected graph comprises a plurality of nodes and a plurality of edges connecting the plurality of nodes, wherein each of the plurality of nodes represents a visual embedding of fashion items in the outfit.
3 . A system comprising:
at least one memory storing programmed instructions; one or more Input/Output (I/O) interfaces; and one or more hardware processors operatively coupled to the at least one memory, wherein the one or more hardware processors are configured by the programmed instructions to:
receive a plurality of images pertaining to a plurality of outfits, wherein the outfit defines a set of fashion items representing a particular style;
extract a visual embedding corresponding to each of the plurality of outfit using a visual feature extractor, wherein each of the visual embedding pertains to each of the plurality of images associated with each of the plurality of outfits;
generate a fully connected graph for each of the plurality of outfits based on the visual embedding using a Graph Neural Network (GNN) with an attention mechanism;
generate an outfit embedding based on fully connected graph associated with each of the plurality of outfits using a graph read out layer associated with the GNN, wherein the visual feature extractor, the GNN with the attention mechanism and feature transformation network are trained using a plurality of positive outfits and a plurality of negative outfits corresponding to each of the plurality of users until a minimum ranking loss is obtained, and a dot attention mechanism updates the embeddings of the nodes and DOT-GNN uses the dot attention mechanism to capture visual interaction between items;
simultaneously generate a style-specific user embedding corresponding to each of a plurality of users based on user information and style information corresponding to each of the plurality of users using a feature transformation network, wherein the user embedding using multiple learnable style embeddings, producing style-conditioned embeddings for each user;
compute a visual compatibility score between each of a plurality of outfit embedding and each of the plurality of style-specific user embedding using a similarity based matching technique; and
predict a visually compatible outfit for each of the plurality of users based on the computed visual compatibility score, wherein an outfit with the visual compatibility score above a predefined threshold is selected.
4 . The system of claim 3 , wherein the fully connected graph comprises a plurality of nodes and a plurality of edges connecting the plurality of nodes, wherein each of the plurality of nodes represents a visual embedding of fashion items in the outfit.
5 . One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause:
receiving a plurality of images pertaining to a plurality of outfits, wherein the outfit defines a set of fashion items representing a particular style;
extracting a visual embedding corresponding to each of the plurality of outfit using a visual feature extractor, wherein each of the visual embedding pertains to each of the plurality of images associated with each of the plurality of outfits;
generating a fully connected graph for each of the plurality of outfits based on the visual embedding using a Graph Neural Network (GNN) with an attention mechanism;
generating an outfit embedding based on fully connected graph associated with each of the plurality of outfits using a graph read out layer associated with the GNN, wherein the visual feature extractor, the GNN with the attention mechanism and feature transformation network are trained using a plurality of positive outfits and a plurality of negative outfits corresponding to each of the plurality of users until a minimum ranking loss is obtained, and a dot attention mechanism updates the embeddings of the nodes and DOT-GNN uses the dot attention mechanism to capture visual interaction between items;
simultaneously generating a style-specific user embedding corresponding to each of a plurality of users based on user information and style information corresponding to each of the plurality of users using a feature transformation network, wherein the user embedding using multiple learnable style embeddings, producing style-conditioned embeddings for each user;
computing a visual compatibility score between each of a plurality of outfit embedding and each of the plurality of style-specific user embedding using a similarity based matching technique; and
predicting a visually compatible outfit for each of the plurality of users based on the computed visual compatibility score, wherein an outfit with the visual compatibility score above a predefined threshold is selected.
6 . The one or more non-transitory machine-readable information storage mediums of claim 5 , wherein the fully connected graph comprises a plurality of nodes and a plurality of edges connecting the plurality of nodes, wherein each of the plurality of nodes represents a visual embedding of fashion items in the outfit.