Solving sparse data problems in a recommendation system with freezing start
Recommendation services typically struggle with sparse data scenarios. A freezing generated item start technique can use a matrix of external generated items to find a linking generated item. Embeddings can be used to determine distance between items. The technologies are useful for providing recommendations even in scenarios involving little or no transaction data.
1. A computer-implemented method comprising: building, by at least one hardware processor of a recommendation system, a generated—
item-by-generated-item matrix representing correlations between external generated items based on observed transactions for the external generated items from an external database;
pruning, by the at least one hardware processor and a pruner, the generated-item-by-generated-item matrix;
embedding, by the at least one hardware processor, the external generated items into a high-dimensional warm embedded vector space;
for a plurality of freezing generated items, embedding, by the at least one hardware processor, the plurality of freezing generated items into a high-dimensional freezing embedded vector space;
for a freezing generated item out of the plurality of freezing generated items, determining, by the at least one hardware processor, a location of the freezing generated item in the high-dimensional warm embedded vector space;
finding, by the at least one hardware processor, a nearest neighbor external generated item r of the freezing generated item within a complementary threshold distance of the freezing generated item in the high-dimensional warm embedded vector space, comprising: calculating distances between the location of the freezing generated item and positions of the external generated items in the high-dimensional warm embedded vector space;
for a given linking external generated item out of a plurality of possible linking external generated items appearing in the matrix for r, determining, by the at least one hardware processor, whether a linked freezing nearest neighbor generated item exists for the given linking external generated item in the high-dimensional freezing embedded vector space, comprising: calculating distances between a location of the given linking external generated item and other freezing generated items in the high-dimensional freezing embedded vector space;
responsive to determining that a linked freezing nearest neighbor generated item exists, associating, by the at least one hardware processor, the freezing generated item with the linked freezing nearest neighbor generated item using the generated-item-by-generated-item matrix as a bridge to enable the association; and
outputting, to an online portal, a recommendation based on the association.
2. The method of claim 1 , further comprising:
receiving, from the online portal, a request for a recommendation for the freezing generated item; and
responsive to the request for the recommendation and based on association between the freezing generated item and the linked freezing nearest neighbor generated item, responding with the recommendation, wherein the recommendation comprises the linked freezing nearest neighbor generated item.
3. The method of claim 1 , further comprising:
filtering, by the at least one hardware processor, the linked freezing nearest neighbor item with a category tree.
4. The method of claim 1 , further comprising:
finding, by the at least one hardware processor, at least one additional linking generated item out of the plurality of possible linking external generated items; and
associating, by the at least one hardware processor, at least one additional linked freezing nearest neighbor item with the freezing generated item.
5. The method of claim 1 wherein:
the high-dimensional freezing embedded vector space and the high-dimensional warm embedded vector space are a same vector space.
6. The method of claim 5 wherein:
the high-dimensional freezing embedded vector space and the high-dimensional warm embedded vector space are represented by different indexes;
finding a nearest neighbor in the high-dimensional warm embedded vector space comprises searching a first index; and
determining whether a linked freezing nearest neighbor exists comprises searching a second index different from the first index.
7. The method of claim 1 wherein:
determining whether a linked freezing nearest neighbor generated item exists incorporates a threshold.
8. The method of claim 1 wherein:
determining whether a linked freezing nearest neighbor generated item exists comprises searching for an approximate nearest neighbor.
9. The method of claim 1 , further comprising:
wherein: the pruning comprises removing any non-warm generated items from the generated-item-by-generated-item matrix; and the pruned matrix represents warm generated items.
10. The method of claim 1 , further comprising:
normalizing, by the at least one hardware processor, the generated-item-by-generated-item matrix based on observed transactions per represented generated item.
11. The method of claim 1 , further comprising:
representing, by the at least one hardware processor, a number of observed co-occurrence transactions for a generated item in a diagonal of the generated-item-by-generated-item matrix.
12. The method of claim 1 , wherein:
embedding the external generated items into the high-dimensional warm embedded vector space comprises:
for a given item out of the external generated items, calculating vector values of text describing the given item.
13. The method of claim 12 , wherein:
the text describing the given item comprises a title of the given item.
14. The method of claim 12 , wherein:
calculating vector values of text describing the given item comprises incorporating context of words in the text.
15. One or more non-transitory computer-readable media having stored therein computer-executable instructions that when executed by a computing system, cause the computing system to perform a method of outputting a generated item recommendation from a plurality of possible generated items, the method comprising:
building, by at least one hardware processor of a recommendation system, a generated-item-by-generated-item matrix representing correlations between external generated items based on observed transactions for the external generated items from an external database;
pruning, by the at least one hardware processor, the generated-item-by-generated-item matrix;
normalizing, by the at least one hardware processor, the generated-item-by-generated-item matrix;
embedding, by the at least one hardware processor, the external generated items into a high-dimensional warm embedded vector space based on titles of the external generated items;
for a plurality of freezing generated items, embedding, by the at least one hardware processor, the plurality of freezing generated items into a high-dimensional freezing embedded vector space based on titles of the freezing generated items;
for a freezing generated item out of the plurality of freezing generated items, determining, by the at least one hardware processor, a location of the freezing generated item in the high-dimensional warm embedded vector space;
finding, by the at least one hardware processor, a nearest neighbor external generated item r of the freezing generated item within a complementary threshold distance of the freezing generated item in the high-dimensional warm embedded vector space;
for a given linking external generated item out of a plurality of possible linking external generated items appearing in rows and columns for r within the matrix, determining, by the at least one hardware processor, whether a linked freezing nearest neighbor generated item exists for the given linking external generated item in the high-dimensional freezing embedded vector space;
responsive to determining that a linked freezing nearest neighbor generated item exists, associating, by the at least one hardware processor, the freezing generated item with the linked freezing nearest neighbor generated item using the generated-item-by-generated-item matrix as a bridge to enable the association; and
outputting, to an online portal, a recommendation based on the association.