Systems and methods for unsupervised neologism normalization of electronic content using embedding space mapping
Systems and methods are disclosed for utilizing a comment moderation bot for detecting and normalizing neologisms in social media. One method comprises transmitting, by a neologism normalization system, a comment moderation bot for detecting neologisms on an online platform maintained by one or more publisher systems. The comment moderation bot may aggregate data related to user comments and transmit the aggregated data to the neologism normalization system for further processing. The neologism normalization system implements unsupervised machine learning models for detecting neologisms in the aggregated data through tokenization and filtering; and normalizing the neologisms through similarity analysis and lattice decoding.
1. A system for neologism normalization, the system comprising:
a data storage device storing instructions and a processor configured to execute the instructions for:
detecting one or more user generated comments on an electronic platform;
tokenizing the one or more user generated comments;
generating a list of lexemes by filtering the one or more user generated comments against known lexemes and removing the known lexemes from further analysis;
identifying sub-lexemes from the list of lexemes;
normalizing the list of lexemes through lattice decoding; and
storing the normalized list of lexemes in a neologism database.
2. The system of claim 1 , wherein tokenizing the one or more user generated comments includes executing code for implementing one or more of: creating white space splits, identifying Uniform Resource Locators, or identifying specific punctuation patterns.
3. The system of claim 1 , wherein detecting one or more user generated comments on the electronic platform further comprises, scanning content retrieving lexical data for analysis.
4. The system of claim 1 further comprising: determining a type of content associated with the user generated comments.
5. The system of claim 1 , wherein filtering the one or more user generated comments against known lexemes, further comprises identifying one or more of social media jargon, named entities, spelling errors, and abbreviations, in the one or more user generated comments.
6. The system of claim 1 , wherein normalizing the list of lexemes through lattice decoding further comprises:
retrieving a list of known lexemes from a corpus and comparing the list of known lexemes to the list of lexemes and assigning a score to the lexemes in the list of known lexemes based on similarity metrics.
7. The system of claim 6 , further comprising:
wherein comparing the list of known lexemes to the list of remaining lexemes further comprises comparing a type of lexemes in the list of known lexemes to the type of lexemes in the list of lexemes; and
wherein assigning a score to the lexemes in the list of known lexemes comprises analyzing the lexemes in the list of known lexemes for semantic similarity, lexical similarity and phonetic similarity, to lexemes in the list of lexemes.
8. A computer-implemented method for neologism normalization, the computer-implemented method comprising:
detecting one or more user generated comments on an electronic platform;
tokenizing the one or more user generated comments;
filtering the one or more user generated comments against known lexemes stored in a database and removing the known lexemes from further analysis;
generating a list of lexemes as a result of the filtered one or more user generated comments;
identifying sub-lexemes from the list of lexemes;
normalizing the list of lexemes through lattice decoding; and
storing the normalized list of lexemes in a neologism database.
9. The computer-implemented method of claim 8 , wherein tokenizing the one or more user generated comments includes executing code for implementing one or more of: creating white space splits, identifying Uniform Resource Locators, or identifying specific punctuation patterns.
10. The computer-implemented method of claim 8 , wherein detecting one or more user generated comments on the electronic platform further comprises, scanning content retrieving lexical data for analysis.
11. The computer-implemented method of claim 8 , further comprising:
determining a type of content associated with the user generated comments.
12. The computer-implemented method of claim 8 , wherein filtering the one or more user generated comments against known lexemes, further comprises identifying one or more of social media jargon, named entities, spelling errors, and abbreviations, in the one or more user generated comments.
13. The computer-implemented method of claim 8 , wherein normalizing the list of lexemes through lattice decoding further comprises:
retrieving a list of known lexemes from a corpus and comparing the list of known lexemes to the list of lexemes and assigning a score to the lexemes in the list of known lexemes based on similarity metrics.
14. The computer-implemented method of claim 13 , further comprising:
wherein comparing the list of known lexemes to the list of lexemes further comprises comparing a type of lexemes in the list of known lexemes to the type of lexemes in the list of lexemes; and
wherein assigning a score to the lexemes in the list of known lexemes comprises analyzing the lexemes in the list of known lexemes for semantic similarity, lexical similarity and phonetic similarity, to lexemes in the list of lexemes.
15. A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to perform a method for neologism normalization, comprising:
detecting one or more user generated comments on an electronic platform;
tokenizing the one or more user generated comments;
filtering the one or more user generated comments against known lexemes stored in a database and removing the known lexemes from further analysis;
generating a list of lexemes as a result of the filtered one or more user generated comments;
identifying sub-lexemes from the list of lexemes;
normalizing the list of lexemes through lattice decoding; and
storing the normalized list of lexemes in a neologism database.
16. The non-transitory computer readable medium of claim 15 , wherein tokenizing the one or more user generated comments includes executing code for implementing one or more of: creating white space splits, identifying Uniform Resource Locators, or identifying specific punctuation patterns.
17. The non-transitory computer readable medium of claim 15 , wherein detecting one or more user generated comments on the electronic platform further comprises, scanning content retrieving lexical data for analysis.
18. The non-transitory computer readable medium of claim 15 , further comprising: determining a type of content associated with the user generated comments.
19. The non-transitory computer readable medium of claim 15 , wherein normalizing the list of lexemes through lattice decoding further comprises:
retrieving a list of known lexemes from a corpus and comparing the list of known lexemes to the list of lexemes and assigning a score to the lexemes in the list of known lexemes based on similarity metrics.
20. The non-transitory computer readable medium of claim 19 , further comprising:
wherein comparing the list of known lexemes to the list of lexemes further comprises comparing a type of lexemes in the list of known lexemes to the type of lexemes in the list of lexemes; and
wherein assigning a score to the lexemes in the list of known lexemes comprises analyzing the lexemes in the list of known lexemes for semantic similarity, lexical similarity and phonetic similarity, to lexemes in the list of lexemes.