Systems and methods for unsupervised neologism normalization of electronic content using embedding space mapping
Systems and methods are disclosed for utilizing a comment moderation bot for detecting and normalizing neologisms in social media. One method comprises transmitting, by a neologism normalization system, a comment moderation bot for detecting neologisms on an online platform maintained by one or more publisher systems. The comment moderation bot may aggregate data related to user comments and transmit the aggregated data to the neologism normalization system for further processing. The neologism normalization system implements unsupervised machine learning models for detecting neologisms in the aggregated data through tokenization and filtering; and normalizing the neologisms through similarity analysis and lattice decoding.
1 . A system for neologism normalization, the system comprising:
a data storage device storing instructions and a processor configured to execute the instructions for:
detecting user generated comments on an electronic platform;
removing known lexemes of the user generated comments from further analysis;
normalizing, using a machine learning model, the remaining lexemes through lattice decoding by iteratively replacing frequently used pairs of bytes in a sequence with single unused bytes; and
storing the normalized lexemes in a database.
2 . The system of claim 1 further comprising tokenizing the user generated comments, wherein tokenizing includes executing code for implementing one or more of: creating white space splits, identifying Uniform Resource Locators, or identifying specific punctuation patterns.
3 . The system of claim 1 , wherein detecting one or more user generated comments on the electronic platform further comprises, scanning content retrieving lexical data for analysis.
4 . The system of claim 1 further comprising: determining a type of content associated with the user generated comments.
5 . The system of claim 1 , wherein removing known lexemes of the user generated comments from further analysis further comprises identifying one or more of social media jargon, named entities, spelling errors, and abbreviations, in the one or more user generated comments.
6 . The system of claim 1 , wherein normalizing the remaining lexemes through lattice decoding further comprises:
retrieving a list of known lexemes from a corpus and comparing the list of known lexemes to the remaining lexemes and assigning a score to the lexemes in the list of known lexemes based on similarity metrics.
7 . The system of claim 6 , further comprising:
wherein comparing the list of known lexemes to the list of remaining lexemes further comprises comparing a type of lexemes in the list of known lexemes to the type of lexemes in the list of lexemes; and
wherein assigning a score to the lexemes in the list of known lexemes comprises analyzing the lexemes in the list of known lexemes for semantic similarity, lexical similarity and phonetic similarity, to lexemes in the list of lexemes.
8 . A computer-implemented method for neologism normalization, the computer-implemented method comprising:
detecting user generated comments on an electronic platform;
removing known lexemes of the user generated comments from further analysis;
normalizing, using a machine learning model, the remaining lexemes through lattice decoding by iteratively replacing frequently used pairs of bytes in a sequence with single unused bytes; and
storing the normalized lexemes in a database.
9 . The computer-implemented method of claim 8 further comprising tokenizing the user generated comments, wherein tokenizing includes executing code for implementing one or more of: creating white space splits, identifying Uniform Resource Locators, or identifying specific punctuation patterns.
10 . The computer-implemented method of claim 8 , wherein detecting one or more user generated comments on the electronic platform further comprises, scanning content retrieving lexical data for analysis.
11 . The computer-implemented method of claim 8 , further comprising:
determining a type of content associated with the user generated comments.
12 . The computer-implemented method of claim 8 , wherein removing known lexemes of the user generated comments from further analysis further comprises identifying one or more of social media jargon, named entities, spelling errors, and abbreviations, in the one or more user generated comments.
13 . The computer-implemented method of claim 8 , wherein normalizing the remaining lexemes through lattice decoding further comprises:
retrieving a list of known lexemes from a corpus and comparing the list of known lexemes to the remaining lexemes and assigning a score to the lexemes in the list of known lexemes based on similarity metrics.
14 . The computer-implemented method of claim 13 , further comprising:
wherein comparing the list of known lexemes to the list of remaining lexemes further comprises comparing a type of lexemes in the list of known lexemes to the type of lexemes in the list of lexemes; and
wherein assigning a score to the lexemes in the list of known lexemes comprises analyzing the lexemes in the list of known lexemes for semantic similarity, lexical similarity and phonetic similarity, to lexemes in the list of lexemes.
15 . A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to perform a method for neologism normalization, comprising:
detecting user generated comments on an electronic platform;
removing known lexemes of the user generated comments from further analysis;
normalizing, using a machine learning model, the remaining lexemes through lattice decoding by iteratively replacing frequently used pairs of bytes in a sequence with single unused bytes; and
storing the normalized lexemes in a database.
16 . The non-transitory computer readable medium of claim 15 further comprising tokenizing the user generated comments, wherein tokenizing includes executing code for implementing one or more of: creating white space splits, identifying Uniform Resource Locators, or identifying specific punctuation patterns.
17 . The non-transitory computer readable medium of claim 15 , wherein detecting one or more user generated comments on the electronic platform further comprises, scanning content retrieving lexical data for analysis.
18 . The non-transitory computer readable medium of claim 15 , further comprising: determining a type of content associated with the user generated comments.
19 . The non-transitory computer readable medium of claim 15 , wherein normalizing the remaining lexemes through lattice decoding further comprises:
retrieving a list of known lexemes from a corpus and comparing the list of known lexemes to the remaining lexemes and assigning a score to the lexemes in the list of known lexemes based on similarity metrics.
20 . The non-transitory computer readable medium of claim 19 , further comprising:
wherein comparing the list of known lexemes to the list of remaining lexemes further comprises comparing a type of lexemes in the list of known lexemes to the type of lexemes in the list of lexemes; and
wherein assigning a score to the lexemes in the list of known lexemes comprises analyzing the lexemes in the list of known lexemes for semantic similarity, lexical similarity and phonetic similarity, to lexemes in the list of lexemes.