Systems and methods for unsupervised neologism normalization of electronic content using embedding space mapping
Systems and methods are disclosed for utilizing a comment moderation bot for detecting and normalizing neologisms in social media. One method comprises transmitting, by a neologism normalization system, a comment moderation bot for detecting neologisms on an online platform maintained by one or more publisher systems. The comment moderation bot may aggregate data related to user comments and transmit the aggregated data to the neologism normalization system for further processing. The neologism normalization system implements unsupervised machine learning models for detecting neologisms in the aggregated data through tokenization and filtering; and normalizing the neologisms through similarity analysis and lattice decoding.
1. A system for neologism normalization, the system comprising:
a processor configured to execute the instructions to perform a method including:
detecting one or more user generated comments on a publisher platform;
tokenizing the one or more user generated comments;
filtering the one or more user generated comments against known lexemes stored in a database and removing the known lexemes from further analysis;
selecting a language and auditing the one or more user generated comments for lexemes in order to identify and remove foreign lexemes from further analysis;
generating a list of remaining lexemes as a result of the filtered and audited one or more user generated comments;
identifying sub-lexemes from the list of remaining lexemes;
normalizing the list of remaining lexemes through lattice decoding; and
storing the normalized list of remaining lexemes in a neologism database.
2. The system of claim 1 ,
wherein detecting one or more user generated comments on a publisher platform further comprises, utilizing a bot to scan content and retrieve lexical data for analysis.
3. The system of claim 1 further comprising:
determining a genre of content associated with the user generated comments.
4. The system of claim 1 ,
wherein filtering the one or more user generated comments against known lexemes, further comprises identifying one or more of social media jargon, named entities, spelling errors, and abbreviations, in the one or more user generated comments.
5. The system of claim 1 , wherein normalizing the list of remaining lexemes through lattice decoding further comprises:
retrieving a list of known lexemes from a corpus and comparing the list of known lexemes to the list of remaining lexemes and assigning a score to the lexemes in the list of known lexemes based on similarity metrics.
6. The system of claim 5 , further comprising:
wherein comparing the list of known lexemes to the list of remaining lexemes further comprises comparing the genre of lexemes in the list of known lexemes to the genre of lexemes in the list of remaining lexemes; and
wherein assigning a score to the lexemes in the list of known lexemes comprises analyzing the lexemes in the list of known lexemes for semantic similarity, lexical similarity and phonetic similarity, to lexemes in the list of remaining lexemes.
7. The system of claim 5 , further comprising:
automatically updating unsupervised machine learning models, based on the scores assigned to the lexemes in the list of known lexemes.
8. A computer-implemented method for neologism normalization, the system comprising:
a processor configured to execute the instructions to perform a method including:
detecting one or more user generated comments on a publisher platform;
tokenizing the one or more user generated comments;
filtering the one or more user generated comments against known lexemes stored in a database and removing the known lexemes from further analysis;
selecting a language and auditing the one or more user generated comments for lexemes in order to identify and remove foreign lexemes from further analysis;
generating a list of remaining lexemes as a result of the filtered and audited one or more user generated comments;
identifying sub-lexemes from the list of remaining lexemes;
normalizing the list of remaining lexemes through lattice decoding; and
storing the normalized list of remaining lexemes in a neologism database.
9. The computer-implemented method of claim 8 ,
wherein detecting one or more user generated comments on a publisher platform further comprises, utilizing a bot to scan content and retrieve lexical data for analysis.
10. The computer-implemented method of claim 8 further comprising:
determining a genre of content associated with the user generated comments.
11. The computer-implemented method of claim 8 ,
wherein filtering the one or more user generated comments against known lexemes, further comprises identifying one or more of social media jargon, named entities, spelling errors, and abbreviations, in the one or more user generated comments.
12. The computer-implemented method of claim 8 , wherein normalizing the list of remaining lexemes through lattice decoding further comprises:
retrieving a list of known lexemes from a corpus and comparing the list of known lexemes to the list of remaining lexemes and assigning a score to the lexemes in the list of known lexemes based on similarity metrics.
13. The computer-implemented method of claim 12 , further comprising:
wherein and comparing the list of known lexemes to the list of remaining lexemes further comprises comparing the genre of lexemes in the list of known lexemes to the genre of lexemes in the list of remaining lexemes; and
wherein assigning a score to the lexemes in the list of known lexemes comprises analyzing the lexemes in the list of known lexemes for semantic similarity, lexical similarity and phonetic similarity, to lexemes in the list of remaining lexemes.
14. The s computer-implemented method of claim 12 , further comprising:
automatically updating unsupervised machine learning models, based on the scores assigned to the lexemes in the list of known lexemes.
15. A non-transitory computer readable medium for neologism normalization, the system comprising:
a processor configured to execute the instructions to perform a method including:
detecting one or more user generated comments on a publisher platform;
tokenizing the one or more user generated comments;
filtering the one or more user generated comments against known lexemes stored in a database and removing the known lexemes from further analysis;
selecting a language and auditing the one or more user generated comments for lexemes in order to identify and remove foreign lexemes from further analysis;
generating a list of remaining lexemes as a result of the filtered and audited one or more user generated comments;
identifying sub-lexemes from the list of remaining lexemes;
normalizing the list of remaining lexemes through lattice decoding; and
storing the normalized list of remaining lexemes in a neologism database.
16. The non-transitory computer readable medium of claim 15 ,
wherein detecting one or more user generated comments on a publisher platform further comprises, utilizing a bot to scan content and retrieve lexical data for analysis.
17. The non-transitory computer readable medium m of claim 15 further comprising:
determining a genre of content associated with the user generated comments.
18. The non-transitory computer readable medium of claim 15 ,
wherein filtering the one or more user generated comments against known lexemes, further comprises identifying one or more of social media jargon, named entities, spelling errors, and abbreviations, in the one or more user generated comments.
19. The non-transitory computer readable medium of claim 18 , wherein normalizing the list of remaining lexemes through lattice decoding further comprises:
retrieving a list of known lexemes from a corpus and comparing the list of known lexemes to the list of remaining lexemes and assigning a score to the lexemes in the list of known lexemes based on similarity metrics; and
automatically updating unsupervised machine learning models, based on the scores assigned to the lexemes in the list of known lexemes.
20. The non-transitory computer readable medium of claim 1 , further comprising:
wherein and comparing the list of known lexemes to the list of remaining lexemes further comprises comparing the genre of lexemes in the list of known lexemes to the genre of lexemes in the list of remaining lexemes; and
wherein assigning a score to the lexemes in the list of known lexemes comprises analyzing the lexemes in the list of known lexemes for semantic similarity, lexical similarity and phonetic similarity, to lexemes in the list of remaining lexemes.