INFORMATION IDENTIFICATION AND EXTRACTION
A computer implemented method of information identification and extraction may include creating an author object in a database for each author of multiple digital documents, each of the digital documents including a topic. For each author object created, the method may additionally include obtaining multiple personal academic web page candidates, obtaining multiple social media account candidates based on a search in the social media for a name of the author in the author object, and cross-validating one of personal academic web page candidates and one of the social media account candidates as a personal academic web page and a social media account associated with the author. The method may also include extracting data from new posts from the social media accounts associated with the authors of each of the author objects, and providing the data in an organization based on the topics of the digital documents.
1 . A computer implemented method of information identification and extraction, the method comprising:
creating an author object in a database for each author of a plurality of digital documents, each of the digital documents including a topic;
for each author object created:
obtaining a plurality of personal academic web page candidates;
obtaining a plurality of social media account candidates based on a search in the social media for a name of the author in the author object; and
cross-validating one of the plurality of personal academic web page candidates and one of the plurality of social media account candidates as a personal academic web page and a social media account associated with the author;
extracting data from new posts from the social media accounts associated with the authors of each of the author objects; and
providing the data in an organization based on the topics of the digital documents.
2 . The method of claim 1 , wherein obtaining a plurality of personal academic web page candidates comprises:
performing a first search for personal academic web pages based on a name of the author;
performing a second search for personal academic web pages based on the name of the author and one or more affiliations of the author;
merging a first number of results from the first search with a second number of results from the second search to create a merged set of results;
identifying social media pages from the merged set of results as part of the plurality of personal academic page candidates;
after identifying social media pages, parsing each result of the merged set of results to identify other parts of the plurality of personal academic page candidates.
3 . The method of claim 2 , wherein parsing each result of the merged set of results to identify the plurality of personal academic page candidates comprises, for each of the results:
analyzing a webpage of the result, comprising:
fetching the webpage;
analyzing code of the webpage to identify one or more information blocks;
extracting keywords from the one or more information blocks; and
generating a keyword score based on the extracted keywords;
analyzing anchor texts of the webpage, comprising:
identifying anchor texts in the webpage;
searching the anchor texts for names; and
generating an anchor text score based on the anchor texts and names in the anchor text that match the author object;
analyzing a uniform resource locator (URL) of the webpage, comprising
splitting the URL into fragments;
searching the fragments for names and keywords; and
generating a URL score based on names and keywords in the fragments;
based on the keyword score, the anchor text score, and the URL score, categorizing the result; and
based on the result being categorized as a personal academic webpage, adding the result to the plurality of personal academic page candidates.
4 . The method of claim 1 , wherein cross-validating one of the plurality of personal academic web page candidates and one of the plurality of social media account candidates comprises:
fetching a profile of the one of the plurality of social media account candidates;
identifying a URL in the profile;
comparing a URL of the one of the plurality of personal academic web page candidates with the URL in the profile;
based on a match between the URL in the profile and the URL of the one of the plurality of personal academic web page candidates, confirming that the one of the plurality of personal academic web page candidates and the one of the plurality of social media account candidates are associated with the author.
5 . The method of claim 1 , wherein cross-validating one of the plurality of personal academic web page candidates and one of the plurality of social media account candidates comprises:
fetching the one of the plurality of personal academic web page candidates;
parsing the one of the plurality of personal academic web page candidates to identify a social media account;
comparing the identified social media account with the one of the plurality of social media account candidates;
based on a match between the identified social media account and the one of the plurality of social media account candidates, confirming that the one of the plurality of personal academic web page candidates and the one of the plurality of social media account candidates are associated with the author.
6 . The method of claim 1 , wherein cross-validating one of the plurality of personal academic web page candidates and one of the plurality of social media account candidates comprises:
fetching the one of the plurality of personal academic web page candidates;
parsing the one of the plurality of personal academic web page candidates to extract first photos from the one of the plurality of personal academic web page candidates;
fetching a profile of the one of the plurality of social media account candidates;
parsing the profile to extract second photos from the profile;
comparing the first photos with the second photos;
based on at least one of the first photos and at least one of the second photos exceeding a similarity threshold, confirming that the one of the plurality of personal academic web page candidates and the one of the plurality of social media account candidates are associated with the author.
7 . The method of claim 1 , wherein cross-validating one of the plurality of personal academic web page candidates and one of the plurality of social media account candidates comprises:
fetching the one of the plurality of personal academic web page candidates;
parsing code of the webpage to identify one or more information blocks;
extracting keywords from the one or more information blocks;
fetching a profile of the one of the plurality of social media account candidates;
comparing the extracted keywords with text in the profile;
based on the extracted keywords and the text in the profile exceeding a similarity threshold, confirming that the one of the plurality of personal academic web page candidates and the one of the plurality of social media account candidates are associated with the author.
8 . The method of claim 1 , wherein cross-validating one of the plurality of personal academic web page candidates and one of the plurality of social media account candidates comprises:
fetching the one of the plurality of personal academic web page candidates;
parsing code of the webpage to identify one or more information blocks;
extracting keywords from the one or more information blocks;
fetching profiles of one or more linked social media accounts, the linked social media accounts linked to the one of the plurality of social media account candidates;
comparing the extracted keywords with text in the profiles of the one or more linked social media accounts;
based on the extracted keywords and the text in the profiles of the one or more linked social media accounts exceeding a similarity threshold, confirming that the one of the plurality of personal academic web page candidates and the one of the plurality of social media account candidates are associated with the author.
9 . The method of claim 1 , wherein cross-validating one of the plurality of personal academic web page candidates and one of the plurality of social media account candidates includes utilizing more than one cross-validation process.
10 . A non-transitory computer-readable medium containing instructions that, when executed by one or more processors, are configured to perform and/or control performance of operations, the operations comprising:
creating an author object in a database for each author of a plurality of digital documents, each of the digital documents including a topic;
for each author object created:
obtaining a plurality of personal academic web page candidates;
obtaining a plurality of social media account candidates based on a search in the social media for a name of the author in the author object; and
cross-validating one of the plurality of personal academic web page candidates and one of the plurality of social media account candidates as a personal academic web page and a social media account associated with the author;
extracting data from new posts from the social media accounts associated with the authors of each of the author objects; and
providing the data in an organization based on the topics of the digital documents.
11 . The computer-readable medium of claim 10 , wherein obtaining a plurality of personal academic web page candidates comprises:
performing a first search for personal academic web pages based on a name of the author;
performing a second search for personal academic web pages based on the name of the author and one or more affiliations of the author;
merging a first number of results from the first search with a second number of results from the second search to create a merged set of results;
identifying social media pages from the merged set of results as part of the plurality of personal academic page candidates;
after identifying social media pages, parsing each result of the merged set of results to identify other parts of the plurality of personal academic page candidates.
12 . The computer-readable medium of claim 11 , wherein parsing each result of the merged set of results to identify the plurality of personal academic page candidates comprises, for each of the results:
analyzing a webpage of the result, comprising:
fetching the webpage;
analyzing code of the webpage to identify one or more information blocks;
extracting keywords from the one or more information blocks; and
generating a keyword score based on the extracted keywords;
analyzing anchor texts of the webpage, comprising:
identifying anchor texts in the webpage;
searching the anchor texts for names; and
generating an anchor text score based on the anchor texts and names in the anchor text that match the author object;
analyzing a uniform resource locator (URL) of the webpage, comprising
splitting the URL into fragments;
searching the fragments for names and keywords; and
generating a URL score based on names and keywords in the fragments;
based on the keyword score, the anchor text score, and the URL score, categorizing the result;
based on the result being categorized as a personal academic webpage, adding the result to the plurality of personal academic page candidates.
13 . The computer-readable medium of claim 10 , wherein cross-validating one of the plurality of personal academic web page candidates and one of the plurality of social media account candidates comprises:
fetching a profile of the one of the plurality of social media account candidates;
identifying a URL in the profile;
comparing a URL of the one of the plurality of personal academic web page candidates with the URL in the profile;
based on a match between the URL in the profile and the URL of the one of the plurality of personal academic web page candidates, confirming that the one of the plurality of personal academic web page candidates and the one of the plurality of social media account candidates are associated with the author.
14 . The computer-readable medium of claim 10 , wherein cross-validating one of the plurality of personal academic web page candidates and one of the plurality of social media account candidates comprises:
fetching the one of the plurality of personal academic web page candidates;
parsing the one of the plurality of personal academic web page candidates to identify a social media account;
comparing the identified social media account with the one of the plurality of social media account candidates;
based on a match between the identified social media account and the one of the plurality of social media account candidates, confirming that the one of the plurality of personal academic web page candidates and the one of the plurality of social media account candidates are associated with the author.
15 . The computer-readable medium of claim 10 , wherein cross-validating one of the plurality of personal academic web page candidates and one of the plurality of social media account candidates comprises:
fetching the one of the plurality of personal academic web page candidates;
parsing the one of the plurality of personal academic web page candidates to extract first photos from the one of the plurality of personal academic web page candidates;
fetching a profile of the one of the plurality of social media account candidates;
parsing the profile to extract second photos from the profile;
comparing the first photos with the second photos;
based on at least one of the first photos and at least one of the second photos exceeding a similarity threshold, confirming that the one of the plurality of personal academic web page candidates and the one of the plurality of social media account candidates are associated with the author.
16 . The computer-readable medium of claim 10 , wherein cross-validating one of the plurality of personal academic web page candidates and one of the plurality of social media account candidates comprises:
fetching the one of the plurality of personal academic web page candidates;
parsing code of the webpage to identify one or more information blocks;
extracting keywords from the one or more information blocks;
fetching a profile of the one of the plurality of social media account candidates;
comparing the extracted keywords with text in the profile;
based on the extracted keywords and the text in the profile exceeding a similarity threshold, confirming that the one of the plurality of personal academic web page candidates and the one of the plurality of social media account candidates are associated with the author.
17 . The computer-readable medium of claim 10 , wherein cross-validating one of the plurality of personal academic web page candidates and one of the plurality of social media account candidates comprises:
fetching the one of the plurality of personal academic web page candidates;
parsing code of the webpage to identify one or more information blocks;
extracting keywords from the one or more information blocks;
fetching profiles of one or more linked social media accounts, the linked social media accounts linked to the one of the plurality of social media account candidates;
comparing the extracted keywords with text in the profiles of the one or more linked social media accounts;
based on the extracted keywords and the text in the profiles of the one or more linked social media accounts exceeding a similarity threshold, confirming that the one of the plurality of personal academic web page candidates and the one of the plurality of social media account candidates are associated with the author.
18 . The computer-readable medium of claim 10 , wherein cross-validating one of the plurality of personal academic web page candidates and one of the plurality of social media account candidates includes utilizing more than one process for cross-validation.
19 . A system comprising:
one or more social media servers;
one or more personal web page servers; and
a computing device including:
one or more processors, and
a non-transitory computer-readable medium containing instructions that, when executed by the one or more processors, are configured to perform and/or control performance of operations, the operations comprising:
creating an author object in a database for each author of a plurality of digital documents, each of the digital documents including a topic;
for each author object created:
obtaining a plurality of personal academic web page candidates from the one or more personal web page servers;
obtaining a plurality of social media account candidates from the one or more social media servers based on a search in the social media for a name of the author in the author object; and
cross-validating one of the plurality of personal academic web page candidates and one of the plurality of social media account candidates as a personal academic web page and a social media account associated with the author;
extracting data from new posts from the social media accounts associated with the authors of each of the author objects; and
providing the data in an organization based on the topics of the digital documents.
20 . The system of claim 19 , wherein cross-validating one of the plurality of personal academic web page candidates and one of the plurality of social media account candidates includes utilizing more than one process for cross-validation.