Systems and methods for facilitating open source intelligence gathering
Systems and methods (e.g., utilities) for use in providing automated, lightweight collection of online, open source data which may be content-based to reduce website source bias. In one aspect, a utility is disclosed for use in extracting content of interest from at least one website or other online data source (e.g., where the extracted content can be used in a subsequent search query). In other aspects, utilities are disclosed that are operable to perform various types of analyzes on such extracted content and present graphical representations of such analyzes on a display of a client device.
1. A system for use in determining a sentiment of a term among a plurality of data sets, the system comprising:
a processing module; and
a memory module logically connected to the processing module and comprising a set of computer readable instructions executable by the processing module to:
receive the x most frequently disclosed terms among a plurality of data sets during a time period, wherein x is a positive integer;
for each of the x most frequently disclosed terms during the time period:
determine a volume of the plurality of data sites disclosing the term; and
obtain a sentiment of the term among the plurality of data sites; and
present, on a display, a first graphical representation illustrating the sentiment and volume of each of the x most frequently disclosed terms during the time period.
2. The system of claim 1 , wherein the computer readable instructions are further executable by the processing module to:
accept at least one stop term;
determine whether one of the x most frequently disclosed terms is the at least one stop term; and
in response to a positive determination that one of the x most frequently disclosed terms is that at least one stop term, replace the one of the x most frequently disclosed terms that is the at least one stop terms with another term that is not the at least one stop term.
3. The system of claim 1 , wherein the computer readable instructions that obtain the sentiment include instructions executable by the processor to:
ascertain whether each of the data sites disclosing the x most frequently disclosed terms is a positive data site or a negative data site; and
for each of the top x most frequently disclosed terms:
sum the number of positive data sites from the plurality of data sites disclosing the term to obtain a count of positive data sites;
sum the number of negative data sites from the plurality of data sites disclosing the term to obtain a negative count of data sites;
subtract the count of positive data sites from the count of negative data sites to obtain a result; and
utilize the result to obtain the sentiment.
4. The system of claim 3 , wherein the computer readable instructions that ascertain whether each of the data sites discloses the x most frequently disclosed terms is a positive or negative data site include instructions executable by the processor to:
mine each of the data sites for pre-documented positively and negatively charged words; and
utilize the mined pre-documented positively and negatively charged words to determine whether the data site is a positive data site or a negative data site.
5. The system of claim 1 , wherein the computer readable instructions that determine, obtain and present are executed by the processor for each of a number of time increments during the time period.
6. The system of claim 1 , wherein the first graphical representation comprises, for each of the x most frequently disclosed terms, a plurality of bars, wherein a height of each bar represents the volume and a color of each bar represents the sentiment.
7. The system of claim 1 , wherein the first graphical illustration is presented on the display with the data sites disclosing the x most frequently disclosed terms.
8. The system of claim 1 , wherein the computer readable instructions include instructions executable by the processor to:
continue to receive, determine, obtain and present for another time period to obtain a second graphical representation that is different than the first graphical representation.
9. The system of claim 1 , wherein each data site comprises at least one of a blog posting and a news article.
10. The system of claim 1 , wherein each of the data sets comprises an object including content of interest that has been extracted from at least one website by way of a set of computer readable instructions executable by a processing module to:
obtain source code used to generate the at least one website on a display, wherein the source code includes a plurality of elements and each element includes at least one tag comprising at least one tag type;
parse the source code to obtain a node tree including a plurality of nodes arranged in a hierarchical structure, wherein each node comprises one of the elements, and wherein one of the plurality of nodes comprises a root node;
determine a tag type of a node under the root node;
assign a heuristic score to the node based at least in part on the tag type of the node;
continue to determine and assign for one or more additional nodes of the node tree; and
generate the object wherein the content of interest is associated with nodes of the node tree having heuristic scores indicating that such content is of interest.
11. The system of claim 5 , wherein the time increments comprise days.
12. A method for determining a sentiment of a term among a plurality of data sets, the method comprising:
receiving the x most frequently disclosed terms among a plurality of data sets during a time period, wherein x is a positive integer;
for each of the x most frequently disclosed terms during the time period:
determining, using a processing engine, a volume of the plurality of data sites disclosing the term; and
obtaining, using the processing engine, a sentiment of the term among the plurality of data sites; and
presenting, on a display, a first graphical representation illustrating the sentiment and volume of each of the x most frequently disclosed terms during the time period.
13. The method of claim 12 , further comprising:
specifying at least one stop term;
determining whether one of the x most frequently disclosed terms is the at least one stop term; and
responsive to a positive determination in the determining, replacing the one of the x most frequently disclosed terms that is the at least one stop term with another term that is not the at least one stop term.
14. The method of claim 12 , wherein the obtaining a sentiment comprises:
ascertaining whether each of the data sites disclosing the x most frequently disclosed terms is a positive data site or a negative data site; and
for each of the x most frequently disclosed terms:
summing the number of positive data sites from the plurality of data sites disclosing the term to obtain a count of positive data sites;
summing the number of negative data sites from the plurality of data sites disclosing the term to obtain a negative count of data sites;
subtracting the count of positive data sites from the count of negative data sites to obtain a result; and
utilizing the result to obtain the sentiment.
15. The method of claim 14 , wherein the ascertaining comprises:
mining each of the data sites for pre-documented positively and negatively charged words; and
utilizing the mined pre-documented positively and negatively charged words in determining whether the data site is a positive data site or a negative data site.
16. The method of claim 12 , wherein the determining, obtaining and presenting are performed for each of a number of time increments during the time period.
17. The method of claim 16 , wherein the time increments comprise days.
18. The method of claim 12 , wherein the first graphical representation comprises, for each of the x most frequently disclosed terms, a plurality of bars, wherein a height of each bar represents the volume and a color of each bar represents the sentiment.
19. The method of claim 12 , wherein the first graphical illustration is presented on the display with the data sites disclosing the x most frequently disclosed terms.
20. The method of claim 12 , further comprising after the presenting:
re-performing the receiving, determining, obtaining and presenting for another time period to obtain a second graphical representation that is different than the first graphical representation.
21. The method of claim 12 , wherein each of the data sets comprises an object including content of interest that has been extracted from at least one website by way of:
obtaining source code used to generate the at least one website on a display, wherein the source code includes a plurality of elements and each element includes at least one tag comprising at least one tag type;
parsing the source code using a processor to obtain a node tree including a plurality of nodes arranged in a hierarchical structure, wherein each node comprises one of the elements, and wherein one of the plurality of nodes comprises a root node;
determining a tag type of a node under the root node;
assigning a heuristic score to the node based at least in part on the tag type of the node;
repeating the determining and assigning for one or more additional nodes of the node tree; and
generating, using the processor, the object, wherein the content of interest is associated with nodes of the node tree having heuristic scores indicating that such content is of interest.