Concept indexing among database of documents using machine learning techniques
Systems and techniques for indexing and/or querying a database are described herein. Discrete sections and/or segments from documents may be determined by a concept indexing system. The segments may be indexed by concept and/or higher-level category of interest to a user. A user may query the segments by one or more concepts. The segments may be analyzed to rank the segments by statistical accuracy and/or relatedness to one or more particular concepts. The rankings may be used for presentation of search results in a user interface. Furthermore, segments and/or documents may be ranked based on recency decay functions that distinguish between segments that maintain their relevance over time in contrast with temporal segments whose relevance decays quicker over time, for example.
1. A computing system for identifying segments of interest in documents, the computing system including:
one or more hardware computer processors configured to execute software instructions; and
one or more storage devices storing software instructions configured for execution by the one or more hardware computer processors to cause the computing system to:
identify a plurality of segments within a plurality of documents;
receive, via a user interface, a user selection of a plurality of concepts, wherein each concept is associated with one or more concept keywords;
for each segment:
determine statistical likelihoods that the respective segment is associated with respective concepts of the plurality of concepts;
determine a first set of weights for the plurality of concepts for the respective segment, wherein the first set of weights indicates relative importance of the plurality of concepts to the respective segment; and
determine a second weight for a concept that is excluded from the plurality of concepts for the respective segment;
determine a ranked listing of one or more segments being associated with the plurality of concepts based at least in part on, for each segment, the statistical likelihoods, the first set of weights, and the second weight; and
present the ranked listing in the user interface.
2. The computing system according to claim 1 , wherein the ranked listing is further determined based on a recency score associated with the one or more segments.
3. The computing system of claim 2 , wherein execution of further software instructions by the one or more hardware computer processors further cause the computing system to:
calculate the recency score, wherein calculating the recency score comprises:
determining a time associated with a first segment; and
applying a decay function to the time to determine the recency score.
4. The computing system of claim 1 , wherein the plurality of concepts comprises a first concept and a second concept, wherein determining the first set of weights for the plurality of concepts for the respective segment further comprises:
determining a first quantity of occurrences of the first concept in the respective segment, and a second quantity of occurrences of the second concept in the respective segment;
calculating a first weight based at least in part on the first quantity of occurrences; and
calculating a third weight based at least in part on the second quantity of occurrences.
5. The computing system of claim 4 , wherein determining the first quantity of occurrences is based at least on a quantity of keywords associated with the first concept in the respective segment.
6. A non-transitory computer storage medium storing computer executable instructions that when executed by a computer hardware processor perform operations comprising:
identifying a plurality of segments within a plurality of documents;
receiving, via a user interface, a user selection of a plurality of concepts, wherein each concept is associated with one or more concept keywords;
for each segment:
determining statistical likelihoods that the respective segment is associated with respective concepts of the plurality of concepts;
determining a first set of weights for the plurality of concepts for the respective segment, wherein the first set of weights indicates relative importance of the plurality of concepts to the respective segment; and
determining a second weight for a concept that is excluded from the plurality of concepts for the respective segment;
determining a ranked listing of one or more segments being associated with the plurality of concepts based at least in part on, for each segment, the statistical likelihoods, the first set of weights, and the second weight; and
presenting the ranked listing in the user interface.
7. The non-transitory computer storage medium of claim 6 storing further computer executable instructions that when executed by the computer hardware processor perform further operations comprising:
calculating the recency score, wherein calculating the recency score comprises:
determining a time associated with a first segment; and
applying a decay function to the time to determine the recency score.
8. The non-transitory computer storage medium of claim 6 , wherein the plurality of concepts comprises a first concept and a second concept, wherein determining the first set of weights for the plurality of concepts for the respective segment further comprises:
determining a first quantity of occurrences of the first concept in the respective segment, and a second quantity of occurrences of the second concept in the respective segment;
calculating a first weight based at least in part on the first quantity of occurrences; and
calculating a third weight based at least in part on the second quantity of occurrences.
9. The non-transitory computer storage medium of claim 8 , wherein determining the first quantity of occurrences is based at least on a quantity of keywords associated with the first concept in the respective segment.
10. A computer-implemented method comprising:
identifying a plurality of segments within a plurality of documents;
receiving, via a user interface, a user selection of a plurality of concepts, wherein each concept is associated with one or more concept keywords;
for each segment:
determining statistical likelihoods that the respective segment is associated with respective concepts of the plurality of concepts;
determining a first set of weights for the plurality of concepts for the respective segment, wherein the first set of weights indicates relative importance of the plurality of concepts to the respective segment; and
determining a second weight for a concept that is excluded from the plurality of concepts for the respective segment;
determining a ranked listing of one or more segments being associated with the plurality of concepts based at least in part on, for each segment, the statistical likelihoods, the first set of weights, and the second weight; and
presenting the ranked listing in the user interface.
11. The computer-implemented method according to claim 10 , wherein the ranked listing is further determined based on a recency score associated with the one or more segments.
12. The computer-implemented method according to claim 10 , further comprising:
calculating the recency score, wherein calculating the recency score comprises:
determining a time associated with a first segment; and
applying a decay function to the time to determine the recency score.
13. The computer-implemented method according to claim 10 , wherein the plurality of concepts comprises a first concept and a second concept, wherein determining the first set of weights for the plurality of concepts for the respective segment further comprises:
determining a first quantity of occurrences of the first concept in the respective segment, and a second quantity of occurrences of the second concept in the respective segment;
calculating a first weight based at least in part on the first quantity of occurrences; and
calculating a third weight based at least in part on the second quantity of occurrences.
14. The computer-implemented method according to claim 13 , wherein determining the first quantity of occurrences is based at least on a quantity of keywords associated with the first concept in the respective segment.