Determining stability for embedding models
An online system determines a stability metric that indicates overlap between the set of entities associated with a particular entity when embeddings have been adjusted due to modifications in the input data of an embedding model. The online system generates a stability score for the embedding model by computing a statistic for one or more stability metrics. The online system determines a stability metric for a particular content provider by identifying a first cluster of content providers in a set of first embeddings, and a second cluster of content providers in a set of second embeddings. The second embeddings are generated after modifications have been made to input data. The online system determines the stability metric based on an overlap between the first cluster and the second cluster of content providers. The stability score can be an indicator of model performance that can be used to select embedding models.
1. A method comprising:
determining, for each embedding model in a plurality of embedding models, a stability score for the embedding model, comprising:
identifying, for each content provider of a set of content providers, a first cluster of content providers associated with the content provider in a set of first embeddings generated by the embedding model, wherein the set of first embeddings are generated based on first input data, and wherein the first cluster of content providers are determined based on distances between the first embeddings;
identifying, for each content provider, a second cluster of content providers associated with the content provider in a set of second embeddings generated by the embedding model, wherein the set of second embeddings are generated based on second input data different from the first input data, and wherein the second cluster of content providers are determined based on distances between the second embeddings; and
determining the stability score for the embedding model based on an overlap between the first cluster of content providers and the second cluster of content providers for one or more content providers; and
selecting an embedding model for use based on the determined stability scores for the plurality of embedding models.
2. The method of claim 1 , wherein the first input data and the second input data correspond to values for a set of features for the set of content providers.
3. The method of claim 1 , wherein the first input data and the second input data correspond to training data for learning one or more parameters of the embedding model.
4. The method of claim 1 , further comprising providing one or more content items to users of the online system based on embeddings generated by the selected embedding model.
5. The method of claim 1 , wherein determining the stability score further comprises:
determining, for each of the one or more content providers, a stability metric based on an overlap between the first cluster of content providers and the second cluster of content providers; and
determining the stability score by computing a statistic of the stability metrics for the one or more content providers.
6. The method of claim 5 , wherein the statistic is an average, a median, or an inverse of variance of the stability metrics for the one or more content providers.
7. The method of claim 5 , wherein the first cluster of content providers correspond to k-nearest neighbors of a first embedding of the content provider, and the second cluster of content providers correspond to k-nearest neighbors of a second embedding of the content provider.
8. The method of claim 7 , wherein the overlap between the first cluster of content providers and the second cluster of content providers is computed by a number of common content providers in the first cluster of content provider and the second cluster of content providers divided by k.
9. The method of claim 1 , wherein the one or more content providers are a subset of the set of content providers, and share at least one common characteristic with one another.
10. A method comprising:
determining, for each embedding model of one or more embedding models, a stability score for the embedding model, comprising:
identifying, for each content provider of a set of content providers, a first cluster of content providers associated with the content provider in a set of first embeddings generated by the embedding model, wherein the set of first embeddings are generated based on first input data, and wherein the first cluster of content providers are determined based on distances between the first embeddings;
identifying, for each content provider, a second cluster of content providers associated with the content provider in a set of second embeddings generated by the embedding model, wherein the set of second embeddings are generated based on second input data different from the first input data, and wherein the second cluster of content providers are determined based on distances between the second embeddings; and
determining the stability score for the embedding model based on an overlap between the first cluster of content providers and the second cluster of content providers for one or more content providers.
11. The method of claim 10 , wherein the first input data and the second input data correspond to values for a set of features for the set of content providers.
12. The method of claim 10 , wherein the first input data and the second input data correspond to training data for learning one or more parameters of the embedding model.
13. The method of claim 10 , further comprising providing one or more content items to users of the online system based on the stability scores determined for the one or more embedding models.
14. The method of claim 13 , wherein providing the one or more content items comprises providing the one or more content items to users of the online system based on embeddings generated by an embedding model associated with a highest stability score.
15. The method of claim 10 , wherein determining the stability score further comprises:
determining, for each of the one or more content providers, a stability metric based on an overlap between the first cluster of content providers and the second cluster of content providers of the content provider; and
determining the stability score by computing a statistic of the stability metrics for the one or more content providers.
16. The method of claim 15 , wherein the statistic is an average, a median, or an inverse of variance of the stability metrics for the one or more content providers.
17. The method of claim 15 , wherein the first cluster of content providers correspond to k-nearest neighbors of a first embedding of the content provider, and the second cluster of content providers correspond to k-nearest neighbors of a second embedding of the content provider.
18. The method of claim 17 , wherein the overlap between the first cluster of content providers and the second cluster of content providers is computed by a number of common content providers in the first cluster of content provider and the second cluster of content providers divided by k.
19. The method of claim 10 , wherein the one or more content providers are a subset of the set of content providers, and share at least one common characteristic with one another.