Method and system of utilizing unsupervised learning to improve text to content suggestions
A method and system for providing an application-specific embedding for an entire text-to-content suggestions service is disclosed. The method includes accessing a dataset containing unlabeled training data collected from an application, the unlabeled training data being collected under user privacy constraints, applying an unsupervised ML model to the dataset to generate a pretrained embedding; and utilizing the pretrained embedding to train the text-to-content suggestion ML model utilized by the application.
1. A data processing system comprising:
a processor; and
a memory in communication with the processor, the memory comprising executable instructions that, when executed by the processor, cause the data processing system to perform functions of:
accessing a dataset containing unlabeled training data collected from users' use of an application that offers text-to-content suggestions, text-to-content suggestions including at least one of text-to-image suggestions, text-to-icon suggestions and text-to-emoticon suggestions, the unlabeled training data including an unordered set of words to ensure privacy; and
applying an unsupervised training process to the dataset to generate a domain-specific pretrained embedding for the application;
wherein the pretrained embedding is configured to be used to train a text-to-content suggestion ML model utilized by the application to suggest content in response to a text query.
2. The data processing system of claim 1 , wherein the unsupervised training process includes an average pooling layer and a fully connected layer.
3. The data processing system of claim 2 , wherein the instructions further cause the processor to apply an activation function to the fully connected layer.
4. The data processing system of claim 2 , wherein the instructions further cause the processor to apply a dropout function to the fully connected layer.
5. The data processing system of claim 2 , wherein the unlabeled training data is used to derive the average pooling layer.
6. The data processing system of claim 1 , wherein:
the unordered set of words includes at least one masked word; and
the unsupervised training process generates a predicted word corresponding to the masked word.
7. The data processing system of claim 1 , wherein the pretrained embedding is a domain specific pretrained word embedding.
8. A method for training a text-to-content suggestion machine-learning (ML) model, the method comprising:
accessing a dataset containing unlabeled training data collected from users' use of an application that offers text-to-content suggestions, text-to-content suggestions including at least one of text-to-image suggestions, text-to-icon suggestions and text-to-emoticon suggestions, the unlabeled training data including an unordered set of words to ensure privacy; and
applying an unsupervised training process to the dataset to generate a domain-specific pretrained embedding for the application;
wherein the pretrained embedding is configured to be used to train a text-to-content suggestion ML model utilized by the application to suggest content in response to a text query.
9. The method of claim 8 , wherein the unsupervised training process includes an average pooling layer and a fully connected layer.
10. The method of claim 9 , further comprising applying an activation function to the fully connected layer.
11. The method of claim 9 , further comprising applying a dropout function to the fully connected layer.
12. The method of claim 9 , wherein the unlabeled training data is used to derive the average pooling layer.
13. The method of claim 8 , wherein:
the unordered set of words includes at least one masked word; and
the unsupervised training process generates a predicted word corresponding to the masked word.
14. A non-transitory computer readable medium on which are stored instructions that, when executed, cause a programmable device to:
access a dataset containing unlabeled training data collected from users' use of an application that offers text-to-content suggestions, text-to-content suggestions including at least one of text-to-image suggestions, text-to-icon suggestions and text-to-emoticon suggestions, the unlabeled training data including an unordered set of words to ensure privacy; and
apply an unsupervised training process to the dataset to generate a domain-specific pretrained embedding for the application;
wherein the pretrained embedding is configured to be used to train a text-to-content suggestion ML model utilized by the application to suggest content in response to a text query.
15. The non-transitory computer readable medium of claim 14 , the unsupervised training process includes an average pooling layer and a fully connected layer.
16. The non-transitory computer readable medium of claim 15 , wherein the instructions further cause the programmable device to apply an activation function to the fully connected layer.
17. The non-transitory computer readable medium of claim 15 , wherein the instructions further cause the programmable device to apply a dropout function to the fully connected layer.
18. The non-transitory computer readable medium of claim 15 , wherein the unlabeled training data is used to derive the average pooling layer.
19. The non-transitory computer readable medium of claim 14 , wherein:
the unordered set of words includes at least one masked word; and
the unsupervised training process generates a predicted word corresponding to the masked word.
20. The non-transitory computer readable medium of claim 14 , wherein the pretrained embedding is a domain specific pretrained word embedding.