Automated discovery using textual analysis
An example method includes receiving text from a plurality of documents, segmenting text received text of the plurality of documents, calculating a frequency statistic for each segment of each document, determining segments of potential interest of each document based on calculated frequency statistic, calculating distances between each document of the plurality of documents based on a text metric, and storing segments of potential interest of each document and the distances in a search database. The method may further include receiving a search query and performing a search of information contained in the search database, partitioning documents of search results using the distances, for each partition, determining labels of segments of potential interest for documents of that particular partition, the labels being determined based on a plurality of frequency statistics, and providing determined labels of segments of potential interest for documents of each partition.
1. A non-transitory computer readable medium comprising executable instructions, the instructions being executable by a data processing device to perform a method, the method comprising:
receiving text from a plurality of documents;
for each document of the plurality of documents:
segmenting the received text of the particular document of the plurality of documents to create a set of segments;
for each of at least a subset of the set of segments:
calculating a document frequency statistic indicating a frequency of a particular segment of the at least a subset of the segments within the particular document of the plurality of documents;
comparing the document frequency statistic indicating the frequency of the particular segment within the particular document to a frequency threshold;
determining if the particular segment is a segment of potential interest based on the comparison of the document frequency statistic of the particular segment to the frequency threshold; and
calculating a distance between the particular document of the plurality of documents and each of the other documents of the plurality of documents using a text metric;
receiving a search query and performing a search on the received text of the plurality of documents to generate search results, the search results including at least a subset of documents of the plurality of documents;
dividing the at least the subset of the documents of the plurality of documents of the search results between a first set and a guide set;
for each of the documents of the first set, determining a closest document of the guide set using the distances for that particular document to create partitions of documents;
for each partition of documents:
retrieving the document frequency statistics of each segment of potential interest of each document in the particular partition of documents;
selecting a predetermined number of segments of potential interest of the documents in the particular partition of documents based on a highest frequency statistic of the retrieved document frequency statistics; and
determining identifying information associated with at least some content of the predetermined number of segments of potential interest of the documents in the particular partition of documents; and
providing the identified content of at least one partition for display.
2. The non-transitory computer readable medium of claim 1 , the method further comprising providing the identified content for all of the partitions.
3. The non-transitory computer readable medium of claim 1 , the method further comprising displaying the identified content of the at least one partition.
4. The non-transitory computer readable medium of claim 1 , the method further comprising generating a graph of nodes and edges connecting nodes, where each node represents a document of the search results and each edge between nodes is based on a particular distance between documents.
5. The non-transitory computer readable medium of claim 1 , wherein calculating the document frequency statistic utilizes a log likelihood function, wherein calculating the document frequency statistics comprises:
determining an expected segment score based on a total number of times the particular segment is within the plurality of documents relative to a total number of segments in the plurality of documents;
determining an observed score based on a total number of times the particular segment is within the particular document of the plurality of documents; and
generating a log likelihood function score based on the observed score relative to the expected score.
6. The non-transitory computer readable medium of claim 5 , wherein determining if the particular segment is the segment of potential interest comprises comparing the log likelihood function score associated with the particular segment to a likelihood threshold.
7. The non-transitory computer readable medium of claim 1 , wherein the distance is a result of applying a cosine term frequency-inverse document frequency (tf-idf).
8. The non-transitory computer readable medium of claim 1 , the method further comprising filtering the text to remove common or unwanted phrases.
9. The non-transitory computer readable medium of claim 1 , the method further comprising approximating a distance measure between two documents in the search results using a Dijkstra's algorithm.
10. A system comprising:
at least one processor; and
memory, the memory including executable instructions that when executed by the at least one processor performs the steps of:
receiving text from a plurality of documents;
for each document of the plurality of documents:
segmenting the received text of the particular document of the plurality of documents to create a set of segments;
for each of at least a subset of the set of segments:
calculating a document frequency statistic indicating a frequency of a particular segment of the at least a subset of the segments within the particular document of the plurality of documents;
comparing the document frequency statistic indicating the frequency of the particular segment within the particular document to a frequency threshold;
determining if the particular segment is a segment of potential interest based on the comparison of the document frequency statistic of the particular segment to the frequency threshold; and
calculating a distance between the particular document of the plurality of documents and each of the other documents of the plurality of documents using a text metric;
receiving a search query and performing a search on the received text of the plurality of documents to generate search results, the search results including at least a subset of documents of the plurality of documents;
dividing the at least the subset of the documents of the plurality of documents of the search results between a first set and a guide set;
for each of the documents of the first set, determining a closest document of the guide set using the distances for that particular document to create partitions of documents;
for each partition of documents:
retrieving the document frequency statistics of each segment of potential interest of each document in the particular partition of documents;
selecting a predetermined number of segments of potential interest of the documents in the particular partition of documents based on a highest frequency statistic of the retrieved document frequency statistics; and
determining identifying information associated with at least some content of the predetermined number of segments of potential interest of the documents in the particular partition of documents; and
providing the identified content of at least one partition for display.
11. The system of claim 10 , wherein the steps further comprise providing the identified content for all of the partitions.
12. The system of claim 10 , wherein the steps further comprise displaying the identified content of the at least one partition.
13. The system of claim 10 , wherein the steps further comprise generating a graph of nodes and edges connecting nodes, wherein each node represents a document of the search results and each edge between nodes is based on a particular distance between documents.
14. The system of claim 10 , wherein calculating the document frequency statistic utilizes a log likelihood function, wherein calculating the document frequency statistics comprises:
determining an expected segment score based on a total number of times the particular segment is within the plurality of documents relative to a total number of segments in the plurality of documents;
determining an observed score based on a total number of times the particular segment is within the particular document of the plurality of documents; and
generating a log likelihood function score based on the observed score relative to the expected score.
15. The system of claim 14 , wherein determining if the particular segment is the segment of potential interest comprises comparing the log likelihood function score associated with the particular segment to a likelihood threshold.
16. The system of claim 10 , wherein the distance is a result of applying a cosine term frequency-inverse document frequency (tf-idf).
17. The system of claim 10 , wherein the steps further comprise filtering the text to remove common or unwanted phrases.
18. The system of claim 10 , wherein the steps further comprise approximating a distance measure between two documents in the search results using a Dijkstra's algorithm.
19. A method comprising:
receiving text from a plurality of documents;
for each document of the plurality of documents:
segmenting the received text of the particular document of the plurality of documents to create a set of segments;
for each of at least a subset of the set of segments:
calculating a document frequency statistic indicating a frequency of a particular segment of the at least a subset of the segments within the particular document of the plurality of documents;
comparing the document frequency statistic indicating the frequency of the particular segment within the particular document to a frequency threshold;
determining if the particular segment is a segment of potential interest based on the comparison of the document frequency statistic of the particular segment to the frequency threshold; and
calculating a distance between the particular document of the plurality of documents and each of the other documents of the plurality of documents using a text metric;
receiving a search query and performing a search on the received text of the plurality of documents to generate search results, the search results including at least a subset of documents of the plurality of documents;
dividing the at least the subset of the documents of the plurality of documents of the search results between a first set and a guide set;
for each of the documents of the first set, determining a closest document of the guide set using the distances for that particular document to create partitions of documents;
for each partition of documents:
retrieving the document frequency statistics of each segment of potential interest of each document in the particular partition of documents;
selecting a predetermined number of segments of potential interest of the documents in the particular partition of documents based on a highest frequency statistic of the retrieved document frequency statistics; and
determining identifying information associated with at least some content of the predetermined number of segments of potential interest of the documents in the particular partition of documents; and
providing the identified content of at least one partition for display.