Scalable pipeline for machine learning-based base-variant grouping
A system comprises one or more processors and non-transitory computer-readable media storing computing instructions that, when executed, perform operations comprising: generating an adjacency list for candidate items using a distance threshold with the maximum allowable neighbor distance equal to a Siamese model cut-off. The operations can also comprise loading data for the candidate items in the adjacency list and generating graphs of the candidate items in the adjacency list. The operations further can comprise determining, using breakdown logic, first graphs of the graphs that exceed a predetermined size, and building hierarchy dendrograms of nested subclusters of the first graphs. The operations additional can comprise determining cut-off values based on p-th percentiles of density for the first graphs, and identifying recommended variant groups of the candidate item in the nested subclusters of the hierarchy dendrograms below the cut-off values. Other embodiments are disclosed.
1 . A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing computing instructions that, when executed on the one or more processors, perform operations comprising:
generating text embeddings of the candidate items using a first pre-trained Siamese neural network and a first contrastive loss function; and
converting the text embeddings to first N-dimensional feature embeddings, wherein the first N-dimensional feature embeddings comprise encoded vector representations of the candidate items used to compute similarity between the candidate items for variant grouping;
generating an adjacency list for candidate items using a distance threshold with a maximum allowable neighbor distance equal to a Siamese model cut-off, wherein the distance threshold is determined using a union logic for pairs of the candidate items;
loading data for the candidate items in the adjacency list;
generating graphs of the candidate items in the adjacency list;
determining, using breakdown logic, first graphs of the graphs that exceed a predetermined size;
building hierarchy dendrograms of nested subclusters of the first graphs;
determining cut-off values based on p-th percentiles of density for the first graphs;
identifying recommended variant groups of the candidate items in the nested subclusters of the hierarchy dendrograms below the cut-off values; and
presenting the recommended variant groups on electronic media used for online ordering.
2 . The system of claim 1 , wherein the operations further comprise, before generating the adjacency list:
generating image embeddings of the candidate items using a second pre-trained Siamese neural network; and
flattening an output from a second pre-trained Siamese neural network to second N-dimensional feature embeddings comprising numeric vector representations of the candidate items used to predict whether the candidate items are variants of one another.
3 . The system of claim 2 , wherein the operations further comprise:
indexing, using a similarity search algorithm, item embeddings in a feature space for retrieval of N nearest neighbors of the candidate items.
4 . The system of claim 3 , wherein the similarity search algorithm is Facebook Artificial Intelligence Similarity search (FAISS).
5 . The system of claim 1 , wherein generating the adjacency list further comprises:
for each of the candidate items, determining respective nearest neighbors of each of the candidate items based on a respective pre-trained Siamese neural network model threshold generated for each of the candidate items.
6 . The system of claim 1 , wherein the predetermined size is 5000.
7 . The system of claim 1 , wherein the cut-off values are inversely proportional to sizes of the first graphs.
8 . The system of claim 1 , wherein building the hierarchy dendrograms of the nested subclusters further comprises sorting respective linkage distances between item embeddings of the candidate items of pairs of the candidate items for each of the first graphs.
9 . The system of claim 1 , wherein nodes of the graph correspond to candidate item identifiers, edges between the nodes of the graph correspond to predictions that candidate items associated with the candidate item identifiers are candidate item variants, and lengths of the edges correspond to pair distances between text and image embeddings corresponding to the candidate items.
10 . The method of claim 1 , wherein nodes of the graph correspond to candidate item identifiers, edges between the nodes of the graph correspond to predictions that candidate items associated with the candidate item identifiers are candidate item variants, and lengths of the edges correspond to pair distances between text and image embeddings corresponding to the candidate items.
11 . A method being implemented via execution of computing instructions configured to run on one or more processors and stored at one or more non-transitory computer-readable media, the method comprising:
generating text embeddings of the candidate items using a first pre-trained Siamese neural network and a first contrastive loss function; and
converting the text embeddings to first N-dimensional feature embeddings, wherein the first N-dimensional feature embeddings comprise encoded vector representations used to compute similarity between the candidate items for variant grouping;
generating an adjacency list for candidate items using a distance threshold with a maximum allowable neighbor distance equal to a Siamese model cut-off, wherein the distance threshold is determined using a union logic for pairs of the candidate items;
loading data for the candidate items in the adjacency list;
generating graphs of the candidate items in the adjacency list, wherein nodes of the graphs represent the candidate items and wherein edges of the graphs represent predicted variant neighbor links between pairs of the candidate items;
determining, using breakdown logic, first graphs of the graphs that exceed a predetermined size;
building hierarchy dendrograms of nested subclusters of the first graphs;
determining cut-off values based on p-th percentiles of density for the first graphs;
identifying recommended variant groups of the candidate item in the nested subclusters of the hierarchy dendrograms below the cut-off values; and
presenting the recommended variant groups on electronic media used for online ordering.
12 . The method of claim 11 further comprising, before generating the adjacency list:
generating image embeddings of the candidate items using a second pre-trained Siamese neural network; and
flattening an output from a second pre-trained Siamese neural network to second N-dimensional feature embeddings comprising numeric vector representations used to predict whether the candidate items are variants of one another.
13 . The method of claim 12 further comprising:
indexing, using a similarity search algorithm, item embeddings in a feature space for retrieval of N nearest neighbors of the candidate items.
14 . The method of claim 13 , wherein the similarity search algorithm is Facebook Artificial Intelligence Similarity search (FAISS).
15 . The method of claim 11 , wherein generating the adjacency list further comprises:
for each of the candidate items, determining respective nearest neighbors of each of the candidate items based on a respective pre-trained Siamese neural network model threshold generated for each of the candidate items.
16 . The method of claim 11 , wherein the predetermined size is 5000.
17 . The method of claim 11 , wherein the cut-off values are inversely proportional to sizes of the first graphs.
18 . A non-transitory computer-readable medium storing instructions, wherein the instructions, upon execution by a processor, cause the processor to perform operations comprising:
generating text embeddings of the candidate items using a first pre-trained Siamese neural network and a first contrastive loss function; and
converting the text embeddings to first N-dimensional feature embeddings, wherein the first N-dimensional feature embeddings comprise encoded vector representations used to compute similarity between the candidate items for variant grouping;
generating an adjacency list for candidate items using a distance threshold with a maximum allowable neighbor distance equal to a Siamese model cut-off, wherein the distance threshold is determined using a union logic for pairs of the candidate items;
loading data for the candidate items in the adjacency list;
generating graphs of the candidate items in the adjacency list;
determining, using breakdown logic, first graphs of the graphs that exceed a predetermined size;
building hierarchy dendrograms of nested subclusters of the first graphs using respective linkage distances between item embeddings of the candidate items of pairs of the candidate items of the first graphs;
determining cut-off values based on p-th percentiles of density for the first graphs;
identifying recommended variant groups of the candidate item in the nested subclusters of the hierarchy dendrograms below the cut-off values; and
presenting the recommended variant groups on electronic media used for online ordering.
19 . The non-transitory computer-readable medium of claim 18 storing further instructions, wherein the further instructions, upon execution by a processor, cause the processor to, before generating the adjacency list:
generate image embeddings of the candidate items using a second pre-trained Siamese neural network; and
flatten an output from a second pre-trained Siamese neural network to second N-dimensional feature embeddings comprising numeric vector representations used to predict whether the candidate items are variants of one another.
20 . The non-transitory computer-readable medium of claim 19 storing further instructions, wherein the further instructions, upon execution by a processor, cause the processor to index, using a similarity search algorithm, item embeddings in a feature space for retrieval of N nearest neighbors of the candidate items.