Efficiently indexing and searching similar data
Techniques for efficiently indexing and searching similar data are described herein. According to one embodiment, in response to a query for one or more terms received from a client, a query index is accessed to retrieve a list of one or more super files. Each super file is associated with a group of similar files. Each super file includes terms and/or sequences of terms obtained from the associated group of similar files. Thereafter, the super files representing groups of similar files are presented to the client, where each of the super files includes at least one of the queried terms. Other methods and apparatuses are also described.
1. A computer implemented method, the method comprising:
constructing a super file for each group of similar files to include one or more sequences of one or more terms from the similar files of each group, wherein the one or more sequences of the one or more terms are non-repeating terms within the super file;
indexing the super file for each group based on the one or more sequences of the one or more terms contained within the super file, without indexing associated similar files, including
in response to a new file to be indexed, identifying, by a processor, a group of files that the new file belongs to,
updating a super file associated with the identified group to include one or more sequences of one or more terms of the new file,
and
reindexing the updated super file;
in response to a query for one or more terms received from a client, accessing a query index to retrieve a list of one or more super files that have been indexed by an indexing engine to form the query index, wherein each super file includes a super set of terms extracted from content of the associated group of similar files, wherein the super set of terms are generated by parsing and tokenizing each of the associated group of similar files into one or more sequences of one or more terms;
presenting the list of one or more super files to the client, each of the one or more super files including at least one of the one or more queried terms; and
in response to an input from the client for selecting one of the super files in the list, presenting to the client one or more of the similar files associated with the selected super file.
2. The method of claim 1 , further comprising:
performing ranking among the one or more super files; and
presenting to the client one or more super files that are ranked most relevant to the queried terms.
3. The method of claim 1 , wherein presenting the one or more super files comprises determining whether a file represented by one of the one or more super files matches one or more terms of the query, wherein the super file is presented whether the file matches the one or more terms of the query.
4. The method of claim 1 , further comprising:
performing ranking among the similar files associated with the selected super file; and
presenting to the client one or more similar files associated with the selected super file that are ranked most relevant to the queried terms.
5. The method of claim 1 , wherein presenting one or more of the similar files comprises determining whether one of the one or more similar files matches one or more terms of the query, wherein the file is presented whether the file matches the one or more terms of the query.
6. The method of claim 1 , further comprising tracking which similar files are represented by the super file.
7. The method of claim 1 , further comprising tracking which and where of the terms within a super file that each file in the associated group contains.
8. The method of claim 1 , wherein constructing a super file comprises:
in response to a set of files to be indexed, grouping the files into one or more groups based on similarity of the files; and
for each group creating a super file containing all terms or sequences of terms obtained from the associated files.
9. The method of claim 8 , wherein creating a super file comprises:
including in the super file one or more sequences of one or more terms obtained from a file selected from a group of files;
for each of remaining files in the group, determining whether each segment of a remaining file exists in the super file; and
appending a segment of the remaining file to the super file whether the segment does not exist in the super file.
10. The method of claim 9 , wherein determining whether each segment of the remaining file exists in the super file comprises:
determining whether a current segment of a file exists in the super file, the file having a plurality of segment and each segment having a plurality of terms; and
appending the current segment in the super file whether the current segment does not exist in the super file.
11. The method of claim 10 , further comprising:
determining whether a next segment exists as a next segment in the super file if the current segment exists in the super file; and
appending the last predetermined number of terms of the current segment to the super file if the next segment does not exist.
12. The method of claim 11 , further comprising:
determining whether a next segment of the file exist within the super file if the current segment does not exist in the super file; and
appending the first predetermined number of terms of the next segment to the super file if the next segment exists in the super file.
13. The method of claim 1 , further comprising:
creating a new group for the new file whether an existing group is not identifiable;
creating a new super file for the new group; and
indexing the new super file.
14. A non-transitory machine-readable storage medium having instructions stored therein, which when executed by a processor, cause the processor to perform a method, the method comprising:
constructing a super file for each group of similar files to include one or more sequences of one or more terms from the similar files of each group, wherein the one or more sequences of the one or more terms are non-repeating terms within the super file;
indexing the super file for each group based on the one or more sequences of the one or more terms contained within the super file, without indexing associated similar files, including
in response to a new file to be indexed, identifying a group of files that the new file belongs to,
updating a super file associated with the identified group to include one or more sequences of one or more terms of the new file, and
reindexing the updated super file;
in response to a query for one or more terms received from a client, accessing a query index to retrieve a list of one or more super files that have been indexed by an indexing engine to form the query index, wherein each super file includes a super set of terms extracted from content of the associated group of similar files, wherein the super set of terms are generated by parsing and tokenizing each of the associated group of similar files into one or more sequences of one or more terms;
presenting the list of one or more super files to the client, each of the one or more super files including at least one of the one or more queried terms; and
in response to an input from the client for selecting one of the super files, presenting to the client one or more of the similar files associated with the selected super file.
15. The non-transitory machine-readable storage medium of claim 14 , wherein the method further comprises:
performing ranking among the one or more super files; and
presenting to the client one or more super files that are ranked most relevant to the queried terms.
16. The non-transitory machine-readable storage medium of claim 14 , wherein presenting the one or more super files comprises determining whether a file represented by one of the one or more super files matches one or more terms of the query, wherein the super file is presented whether the file matches the one or more terms of the query.
17. The non-transitory machine-readable storage medium of claim 14 , wherein the method further comprises:
performing ranking among the similar files associated with the selected super file; and
presenting to the client one or more similar files associated with the selected super file that are ranked most relevant to the queried terms.
18. The non-transitory machine-readable storage medium of claim 14 , wherein presenting one or more of the similar files comprises determining whether one of the one or more similar files matches one or more terms of the query, wherein the file is presented whether the file matches the one or more terms of the query.
19. A data processing system, the system comprising:
a processor; and
a memory for storing instructions, which when executed by the processor, cause the processor to
construct a super file for each group of similar files to include one or more sequences of one or more terms from the similar files of each group, wherein the one or more sequences of the one or more terms are non-repeating terms within the super file,
index the super file for each group based on the one or more sequences of the one or more terms contained within the super file, without indexing associated similar files, including
in response to a new file to be indexed, identifying a group of files that the new file belongs to,
updating a super file associated with the identified group to include one or more sequences of one or more terms of the new file, and
reindexing the updated super file,
in response to a query for one or more terms received from a client, access a query index to retrieve a list of one or more super files, wherein each super file includes a super set of terms extracted from the associated group of similar files, wherein the super set of terms are generated by parsing and tokenizing each of the associated group of similar files into one or more sequences of one or more terms,
present the list of one or more super files to the client, each of the one or more super files including at least one of the one or more queried terms, and
in response to an input from the client for selecting one of the super files, presenting to the client one or more of the similar files associated with the selected super file.
20. The system of claim 19 , wherein the instructions, which when executed by the processor, cause the processor further to
perform ranking among the one or more super files, and
present to the client one or more super files that are ranked most relevant to the queried terms.
21. The system of claim 19 , wherein presenting the one or more super files comprises determining whether a file represented by one of the one or more super files matches one or more terms of the query, wherein the super file is presented whether the file matches the one or more terms of the query.
22. The system of claim 19 , wherein the instructions, which when executed by the processor, cause the processor further to
perform ranking among the similar files associated with the selected super file, and
present to the client one or more similar files associated with the selected super file that are ranked most relevant to the queried terms.
23. The system of claim 19 , wherein presenting one or more of the similar files comprises determining whether one of the one or more similar files matches one or more terms of the query, wherein the file is presented whether the file matches the one or more terms of the query.