System and methods for generating an enhanced output of relevant content to facilitate content analysis
The present disclosure relates to methods and systems for ingesting content from data feeds and to generate an enhanced output of relevant content for presentation to a user to facilitate analysis of the relevant content. Content is received from data feeds, and filtered to identify relevant content with respect to a particular context. The relevant content is then processed, e.g., using natural language processes, to extract entities involved, and to also identify particular activities detailed in the relevant content. Activity-mining is applied to the identified relevant data to classify and assigned activity tags to the extracted entity. Based on the extracted and identified information, an enhanced output is generated for presentation to facilitate research operations. The enhanced output may include overlaid graphical annotations, indicators, and graphical controls over the relevant articles to provide a means for updating a database based on the relevant content.
1. A method of enhancing an output of relevant content to facilitate analysis of the relevant content, comprising:
receiving content from data feeds;
optimizing the content to be analyzed to generate optimized content, wherein the optimizing includes parsing the content to be analyzed to extract a text portion of the content to be analyzed for further analysis, while removing extraneous information from the content to be analyzed, wherein the extraneous information comprises advertisements;
filtering out non-relevant content from the optimized content, the non-relevant content including content unrelated to a subject associated with the analysis, the filtering including retaining relevant content, wherein:
filtering out the non-relevant content from the optimized content comprises filtering out duplicate content from the optimized content,
in response to determining that an instance of the content is unavailable, retrieving the duplicate content via one or more links to the duplicate content, and
the content to be analyzed includes one or more of a news article, a blog, a social media post, and a long form article;
extracting entity data corresponding to entities mentioned in the relevant content, wherein the entity data includes entity metadata associated with the entities, and wherein extracting the entity data comprises:
determining that a first entity of the entities is related to a second entity of the entities, and
linking the first entity to the second entity;
activity-mining the relevant content based on the extracted entity data and the entity metadata to associate the entities with activity tags corresponding to activities associated with the entities and extracted from the relevant content;
matching the entities to entity profiles in a database to generate matched entity profiles, the matching based on the entity metadata and profile metadata of the entity profiles in the database;
annotating the relevant content with at least one graphical indicator and at least one graphical user interface (GUI) control based at least in part on one or more of the entities, the matched entity profiles, and the activity tags; and
rendering a GUI having at least a first area configured to display a list of the relevant content and a second area configured to display an enhanced output, wherein the enhanced output comprises an item selected from the list of relevant content, wherein original text of the item includes highlighting overlaid on the one or more of the entities to accentuate the one or more of the entities, each entity of the one or more of the entities highlighted with a distinct color to differentiate each entity of the one or more of the entities.
2. The method of claim 1 , wherein the determining that the first entity is related to the second entity comprises calculating a confidence score, wherein the confidence score comprises a relation similarity score and a name similarity score.
3. The method of claim 1 , further comprising:
determining a number of sources that correspond to the duplicate content based on a number of links pointing to the duplicate content.
4. The method of claim 1 , wherein the filtering out the non-relevant content includes applying a keyword filter and a natural expression pattern matching filter to the content to be analyzed, and wherein the rendering the GUI further comprises rendering the GUI having a third area configured to display the entity profiles extracted from the database.
5. The method of claim 1 , wherein the filtering out the non-relevant content further includes clustering content elements based on a similarity of the content elements to each other, wherein the clustering comprises:
generating a plurality of vectors, wherein each vector of the plurality of vectors corresponds to a content element and includes values representing a relative frequency of each word in the content element; and
comparing each vector of the plurality of vectors to determine a similarity of content elements to each other.
6. The method of claim 5 , wherein the clustering further includes applying a Term Frequency-Inverse Document Frequency (TF-IDF) algorithm that includes determining a frequency of a term within a content element, and determining a frequency of the term across all content elements of the content to be analyzed.
7. The method of claim 1 , wherein the extracting the entity data from the relevant content includes:
identifying entities mentioned in a content element; and
obtaining the metadata associated with the mentioned entities.
8. The method of claim 7 , wherein the identifying the entities mentioned in the content element includes applying a natural expression language model to the relevant content, and wherein a type of the entity mentioned in the item corresponds to whether the entity is a person, a country, or a location.
9. The method of claim 2 , wherein the relation similarity score is calculated using a MinHash algorithm, and wherein the name similarity score is calculated using a Levenshtein distance algorithm.
10. The method of claim 7 , wherein the metadata associated with a particular entity includes a location of the mention of the particular entity within the content element claim 1 , the method further comprising aggregating the one or more data feeds of the data feeds by dynamically generating a list of the one or more data feeds likely to include content to be of interest, wherein the list is generated based on metadata included in the content.
11. The method of claim 1 , wherein the matching the entities to entity profiles in the database to generate the matched entity profiles includes:
identifying candidate profiles in the database based on a coarse similarity estimate of the candidate profiles to the extracted entities;
calculating a refined candidate score for each candidate profile, the refined candidate score based on the entity metadata and the profile metadata of each entity profile;
comparing the refined candidate score for each candidate profile with a threshold; and
designating candidate profiles as a match to a particular extracted entity when the refined candidate score of the candidate profiles exceeds the threshold.
12. The method of claim 1 , wherein the at least one graphical indicator includes an indicator indicating that a particular extracted entity is unmatched with a profile entity in the database, and wherein the at least one GUI control includes a GUI control for executing an update to the database to store the particular extracted entity in the database.
13. A system for enhancing an output of relevant articles to facilitate analysis of the relevant articles, comprising:
at least one data feed for receiving one or more articles to be analyzed;
a server comprising a memory and a processor, wherein the memory includes a classifier, the classifier configured to:
optimize the articles to be analyzed to generate optimized content, wherein the classifier configured to optimize the articles to be analyzed further comprises the classifier configured to parse the articles to be analyzed to extract a text portion of the articles to be analyzed for further analysis, while removing extraneous information from the articles to be analyzed, wherein the extraneous information comprises advertisements;
filter out non-relevant articles from the optimized content, the non-relevant articles including articles unrelated to a subject associated with the analysis, wherein relevant articles are retained, wherein the classifier configured to filter out the non-relevant content from the optimized content further comprises the classifier configured to:
filter out duplicate content from the optimized content,
in response to a determination that an instance of the content is unavailable, retrieve the duplicate content via one or more links to the duplicate content, and
the content to be analyzed includes one or more of a news article, a blog, a social media post, and a long form article;
extract entity data corresponding to entities mentioned in the relevant articles, wherein the entity data corresponding to the entities includes entity metadata associated with the entities, and wherein the classifier configured to extract the entity data comprises the classifier further configured to:
generate a confidence score indicative of a correctness of a posited link between a first entity of the entities and a second entity of the entities, wherein the confidence score comprises a relation similarity score and a name similarity score,
determine that a first entity of the entities is related to a second entity of the entities, and
link the first entity to the second entity;
mine the relevant articles for activities based on the entity data and the entity metadata to associate the entities with activity tags corresponding to activities associated with the entities and extracted from the relevant articles;
match the entities to entity profiles in a database to generate matched entity profiles, wherein the matched entity profiles are identified based on the entity metadata and profile metadata of the entity profiles in the database;
at least one user terminal with a graphical user interface configured to display the enhanced output, wherein the enhanced output comprises an article selected from a list of relevant articles, wherein original text of the article includes highlighting overlaid on one or more of the entities to accentuate the one or more of the entities, each entity of the one or more of the entities highlighted with a distinct color to differentiate each entity of the one or more of the entities; and
the database configured to store the extracted entity data for the entities and the associated activity tags.
14. The system of claim 13 , wherein the at least one data feed includes at least one Rich Site Summary (RSS) feed.
15. The system of claim 13 , wherein the extraneous information further comprises external links.
16. The system of claim 13 , wherein the classifier configured to match the entities to the entity profiles in the database includes the classifier further configured to:
identify candidate profiles in the database based on a coarse similarity estimate of the candidate profiles to the extracted entities;
calculate a refined candidate score for each candidate profile, the refined candidate score based on the metadata of the extracted entities and the profile metadata of each entity profile;
compare the refined candidate score for each candidate profile with a threshold; and
designate candidate profiles as a match to a particular extracted entity when the refined candidate score of the candidate profiles exceeds the threshold.
17. A computer-based tool for facilitating analysis of relevant articles, the computer-based tool including non-transitory computer readable media having stored thereon computer code which, when executed by a processor, causes a computing device to perform operations comprising:
receiving an enhanced output from a server of a research system, the enhanced output including one or more relevant articles overlaid with one or more graphical indicators and at least one graphical user interface (GUI) control, wherein the enhanced output comprises an article selected from a list of relevant articles, wherein original text of the article includes highlighting overlaid on one or more entities to accentuate the one or more entities, each entity of the one or more entities highlighted with a distinct color to differentiate each entity of the one or more entities and is generated by the server based on:
receiving at least one data feed of articles to be analyzed;
optimizing the articles to be analyzed to generate optimized content, wherein the optimizing includes parsing the articles to be analyzed to extract a text portion of the articles to be analyzed for further analysis, while removing extraneous information from the articles to be analyzed, wherein the extraneous information comprises advertisements;
filtering out non-relevant content from the optimized content, the filtering including retaining relevant content, wherein:
filtering out the non-relevant content from the optimized content comprises filtering out duplicate content from the optimized content,
in response to determining that an instance of the content is unavailable, retrieving the duplicate content via one or more links to the duplicate content, and
the content to be analyzed includes one or more of a news article, a blog, a social media post, and a long form article;
extracting entity data corresponding to the one or more entities from the relevant articles, wherein the extracting the entity data comprises:
generating a confidence score indicative of a correctness of a posited link between a first entity of the entities and a second entity of the entities, wherein the confidence score comprises a relation similarity score and a name similarity score,
determining that a first entity of the entities is related to a second entity of the entities, and
linking the first entity to the second entity;
mining the relevant articles based on the extracted entity data and entity metadata associated with the one or more entities to associate the entities with activity tags corresponding to activities associated with the one or more entities and extracted from the relevant content;
matching the one or more entities to entity profiles in a database to generated matched entity profiles, the matching based on the entity metadata and profile metadata of the entity profiles in the database; and
annotating the relevant articles with the one or more graphical indicator and the at least one GUI control based at least in part on the one or more entities, the matched entity profiles, and the activity tags; and
displaying the enhanced output on a GUI.
18. The computer-based tool of claim 17 , wherein the optimizing further comprises extracting non-textual elements from the articles.
19. The computer-based tool of claim 17 , wherein the filtering out the non-relevant content comprises:
training a language model;
providing the articles to the language model; and
identifying the relevant articles from the articles using the language model based on a context in which a keyword is used in the articles.