Fast-approximate TFIDF
View Patent ↗Our approach seeks to reduce the complexity of this type of calculation through approximation and pre-computation. It is designed to work efficiently with modern relational database constructs for content management. The approach is designed to enable the kinds of highly interactive data-driven visualizations that are the hallmark of third generation business intelligence.
1. A method for generating an ordered list of signature terms occurring in at least a first document and a second document, comprising:
computing a frequency of a first term occurring in a first document;
computing a frequency of a second term occurring in the first document;
sorting the first term and the second term to identify the most frequently occurring term in the first document selected from the group comprising the first term and the second term;
computing a frequency of a third term occurring in a second document;
computing a frequency of a fourth term occurring in the second document;
sorting the third term and the fourth term to identify the most frequently occurring term in the second document selected from the group comprising the third term and the fourth term;
truncating the sorted list for the first document by selecting a number of terms to retain from the sorted list for the first documents wherein the sorted list for the first document includes a plurality of terms;
truncating the sorted list for the second document by selecting a number of terms to retain from the sorted list for the second documents wherein the sorted list for the second document includes a plurality of terms;
reducing the size of at least one of the sorted list for the first document and the sorted list for the second document by utilizing a thesaurus to identify like terms;
combining the sorted list for the first document and the sorted list for the second document utilizing a union operation;
sorting the combined list generated by the union operation to create an ordered list of signature terms;
receiving a signature term request, wherein the signature term request includes a number of top signature terms from the ordered list to be returned; and
returning a number of top signature terms from the ordered list upon receipt of the signature term request.
2. The method of claim 1 , further comprising:
assigning a value of one to at least one term in the first document based upon the frequency of the at least one term occurring in the first document, wherein the at least one term is identified as the most frequently occurring in the first document.
3. The method of claim 1 , further comprising:
assigning a value of one to at least one term in the second document based upon the frequency of the at least one term occurring in the second document, wherein the at least one term is identified as the most frequently occurring in the second document.
4. The method of claim 1 , further comprising:
assigning a value of zero to at least one term in the first document based upon the frequency of the at least one term occurring in the first document.
5. The method of claim 1 , further comprising:
assigning a value of zero to at least one term in the second document based upon the frequency of the at least one term occurring in the second document.
6. The method of claim 1 , further comprising:
retrieving a set of results, the set of results including at least one of the first document or the second document, wherein the set of results is generated from a search query.
7. A computer program product for generating an ordered list of signature terms occurring in at least a first document and a second document, comprising:
computer usable program code stored on a computer readable storage medium, the computer usable program code executed by a device, comprising:
computer usable program code configured to compute a frequency of a first term occurring in a first document;
computer usable program code configured to compute a frequency of a second term occurring in the first document;
computer usable program code configured to sort the first term and the second term to identify the most frequently occurring term in the first document selected from the group comprising the first term and the second term;
computer usable program code configured to compute a frequency of a third term occurring in a second document;
computer usable program code configured to compute a frequency of a fourth term occurring in the second document;
computer usable program code configured to sort the third term and the fourth term to identify the most frequently occurring term in the second document selected from the group comprising the third term and the fourth term;
computer usable program code configured to truncate the sorted list for the first document by selecting a number of terms to retain from the sorted list for the first document, wherein the sorted list for the first document includes a plurality of terms;
computer usable program code configured to truncate the sorted list for the second document by selecting a number of terms to retain from the sorted list for the second document, wherein the sorted list for the second document includes a plurality of terms;
computer usable program code configured to reduce the size of at least one of the sorted list for the first document and the sorted list for the second document by utilizing a thesaurus to identify like terms;
computer usable program code configured to combine the sorted list for the first document and the sorted list for the second document utilizing a union operation;
computer usable program code configured to sort the combined list generated by the union operation to create an ordered list of signature terms;
computer usable program code configured to receive a signature term request, wherein the signature term request includes a number of top signature terms from the ordered list to be returned; and
computer usable program code configured to return the number of top signature terms from the ordered list upon receipt of the signature term request.
8. The computer program product of claim 7 , further comprising:
computer usable program code configured to assign a value of one to at least one term in the first document based upon the frequency of the at least one term occurring in the first document, wherein the at least one term is identified as the most frequently occurring in the first document.
9. The computer program product of claim 7 , further comprising:
computer usable program code configured to assign a value of one to at least one term in the second document based upon the frequency of the at least one term occurring in the second document, wherein the at least one term is identified as the most frequently occurring in the second document.
10. The computer program product of claim 7 , further comprising:
computer usable code configured to assign a value of zero to at least one term in the first document based upon the frequency of the at least one term occurring in the first document.
11. The computer program product of claim 7 , further comprising:
computer usable code configured to assign a value of zero to at least one term in the second document based upon the frequency of the at least one term occurring in the second document.
12. The computer program product of claim 7 , further comprising:
computer usable code configured to retrieve a set of results, the set of results including at least one of the first document or the second document, wherein the set of results is generated from a search query.