Efficient indexing of documents with similar content

1. A method of processing documents, comprising:

at a computer system having one or more processors and memory storing one or more programs for execution by the one or more processors:

grouping a set of documents into a plurality of clusters, wherein each cluster includes one or more documents of the set of documents and a respective cluster of documents of the plurality of clusters includes respective cluster data corresponding to a plurality of documents including a first document and a second document;

determining that the second document includes duplicate data that is duplicative of corresponding data in the first document;

identifying a respective subset of the respective cluster data that excludes at least a subset of the duplicate data; and

generating an index of the respective subset of the respective cluster data.

2. The method of claim 1 , wherein:

the plurality of clusters includes a first cluster and a second cluster;

a representation of the first cluster is stored at a first computer system; and

a representation of the second cluster is stored at a second computer system different from the first computer system.

3. The method of claim 1 , wherein:

the first document is associated with a plurality of document identifiers including a global document identifier and a local document identifier;

the global document identifier identifies the first document with respect to a document repository; and

the local document identifies the first document with respect to a portion of the document repository.

4. The method of claim 1 , wherein generating the index excludes indexing the duplicate data.

5. The method of claim 1 , wherein identifying the respective subset includes generating respective compressed cluster data that does not include at least a subset of the duplicate data.

6. The method of claim 5 , wherein:

the plurality of clusters include a plurality of single-document clusters and a plurality of multi-document clusters; and

the method further comprises, before generating the respective compressed cluster data, rearranging the plurality of clusters in a sequence of clusters in accordance with the criteria that that single-document clusters precede multi-document clusters in the sequence of clusters.

7. The method of claim 5 , wherein:

the plurality of documents are represented, in the cluster data, as a sequence of tokens;

identifying the respective subset includes storing document reconstruction data for reconstructing documents from the respective compressed cluster data; and

the method further comprises, after generating the index:

receiving a query including one or more query tokens; and

in response to receiving the query:

identifying positions corresponding to occurrences of the one or more query tokens in the respective subset of the respective cluster data based on the index; and

identifying documents matching the query based on the positions corresponding to occurrences of the one or more query tokens and the document reconstruction data.

8. The method of claim 5 , wherein:

the plurality of documents are represented, in the cluster data, as a sequence of tokens;

the method further comprises, after generating the index:

receiving a query including a plurality of query tokens; and

in response to receiving the query:

searching through compressed cluster data, corresponding to a plurality of clusters of documents, for occurrences of the query tokens;

in accordance with a determination that compressed cluster data corresponding to the respective cluster of documents includes all of the plurality of query tokens, determining whether the respective cluster of documents includes a document matching the search query; and

in accordance with a determination that compressed cluster data corresponding to the respective cluster of documents does not include at least one of the plurality of query tokens, eliminating documents in the respective cluster of documents from further consideration.

9. The method of claim 8 , wherein:

identifying the respective subset includes storing document reconstruction data for reconstructing documents from the respective compressed cluster data;

the query specifies a respective sequence for the plurality of query tokens; and

determining whether the respective cluster of documents includes a document matching the search query includes determining, based on the document reconstruction data, whether the respective cluster of documents includes a document in which the plurality of query tokens occur in the respective sequence.

10. The method of claim 1 , wherein:

the respective cluster of documents includes a plurality of documents that are determined to be related to each other; and

a respective document is determined to be related to one or more other documents in the respective cluster of documents based on an analysis of content of the respective document and content of the one or more documents in the respective cluster of documents.

11. The method of claim 1 , wherein:

the respective cluster of documents includes a plurality of documents that are determined to be related to each other; and

a respective document is determined to be related to one or more other documents in the respective cluster of documents based on a resource locator of the respective document and resource locators of the one or more other documents in the respective cluster of documents.

12. The method of claim 11 , wherein:

a plurality of documents in the set of documents each have a resource locator;

grouping the set of documents into a plurality of clusters includes:

ordering the set of documents in accordance with the resource locators; and

selecting a respective plurality of consecutive documents from the ordering for inclusion in the respective cluster of documents.

13. The method of claim 12 , wherein:

a plurality of documents in the set of documents each have a URL including a respective plurality of domains and a respective protocol indicator;

prior to ordering the set of documents, a modified locator is generated for each respective document, wherein generating a respective modified locator for a particular document having a particular URL includes reversing the domains of the particular URL and moving the protocol indicator for the particular URL to the end of the respective modified locator; and

the documents are ordered in accordance with the modified locators.

14. The method of claim 1 , wherein:

the set of documents comprises a historical archive of different versions of documents; and

a respective cluster of documents of the plurality of clusters includes a plurality of different versions of a same document from different times.

15. A computer system, comprising:

one or more processors;

memory; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:

grouping a set of documents into a plurality of clusters, wherein each cluster includes one or more documents of the set of documents and a respective cluster of documents of the plurality of clusters includes respective cluster data corresponding to a plurality of documents including a first document and a second document;

determining that the second document includes duplicate data that is duplicative of corresponding data in the first document;

identifying a respective subset of the respective cluster data that excludes the duplicate data; and

generating an index of the respective subset of the respective cluster data.

16. The system of claim 15 , wherein:

the plurality of clusters includes a first cluster and a second cluster;

a representation of the first cluster is stored at a first computer system; and

a representation of the second cluster is stored at a second computer system different from the first computer system.

17. The system of claim 15 , wherein:

the first document is associated with a plurality of document identifiers including a global document identifier and a local document identifier;

the global document identifier identifies the first document with respect to a document repository; and

the local document identifies the first document with respect to a portion of the document repository.

18. The system of claim 15 , wherein generating the index excludes indexing the duplicate data.

19. The system of claim 15 , wherein identifying the respective subset includes generating respective compressed cluster data that does not include at least a subset of the duplicate data.

20. The system of claim 19 , wherein:

the plurality of clusters include a plurality of single-document clusters and a plurality of multi-document clusters; and

the one or more programs further include instructions for, before generating the respective compressed cluster data, rearranging the plurality of clusters in a sequence of clusters in accordance with the criteria that that single-document clusters precede multi-document clusters in the sequence of clusters.

21. The system of claim 19 , wherein:

the plurality of documents are represented, in the cluster data, as a sequence of tokens;

identifying the respective subset includes storing document reconstruction data for reconstructing documents from the respective compressed cluster data; and

the one or more programs further include instructions for, after generating the index:

receiving a query including one or more query tokens; and

in response to receiving the query:

identifying positions corresponding to occurrences of the one or more query tokens in the respective subset of the respective cluster data based on the index; and

identifying documents matching the query based on the positions corresponding to occurrences of the one or more query tokens and the document reconstruction data.

22. The system of claim 19 , wherein:

the plurality of documents are represented, in the cluster data, as a sequence of tokens;

the one or more programs further include instructions for, after generating the index:

receiving a query including a plurality of query tokens; and

in response to receiving the query:

searching through compressed cluster data, corresponding to a plurality of clusters of documents, for occurrences of the query tokens;

in accordance with a determination that compressed cluster data corresponding to the respective cluster of documents includes all of the plurality of query tokens, determining whether the respective cluster of documents includes a document matching the search query; and

in accordance with a determination that compressed cluster data corresponding to the respective cluster of documents does not include at least one of the plurality of query tokens, eliminating documents in the respective cluster of documents from further consideration.

23. The system of claim 22 , wherein:

identifying the respective subset includes storing document reconstruction data for reconstructing documents from the respective compressed cluster data;

the query specifies a respective sequence for the plurality of query tokens; and

determining whether the respective cluster of documents includes a document matching the search query includes determining, based on the document reconstruction data, whether the respective cluster of documents includes a document in which the plurality of query tokens occur in the respective sequence.

24. The system of claim 15 , wherein:

the respective cluster of documents includes a plurality of documents that are determined to be related to each other; and

a respective document is determined to be related to one or more other documents in the respective cluster of documents based on an analysis of content of the respective document and content of the one or more documents in the respective cluster of documents.

25. The system of claim 15 , wherein:

the respective cluster of documents includes a plurality of documents that are determined to be related to each other; and

a respective document is determined to be related to one or more other documents in the respective cluster of documents based on a resource locator of the respective document and resource locators of the one or more other documents in the respective cluster of documents.

26. The system of claim 25 , wherein:

a plurality of documents in the set of documents each have a resource locator;

grouping the set of documents into a plurality of clusters includes:

ordering the set of documents in accordance with the resource locators; and

selecting a respective plurality of consecutive documents from the ordering for inclusion in the respective cluster of documents.

27. The system of claim 26 , wherein:

a plurality of documents in the set of documents each have a URL including a respective plurality of domains and a respective protocol indicator;

prior to ordering the set of documents, a modified locator is generated for each respective document, wherein generating a respective modified locator for a particular document having a particular URL includes reversing the domains of the particular URL and moving the protocol indicator for the particular URL to the end of the respective modified locator; and

the documents are ordered in accordance with the modified locators.

28. The system of claim 15 , wherein:

the set of documents comprises a historical archive of different versions of documents; and

a respective cluster of documents of the plurality of clusters includes a plurality of different versions of a same document from different times.

29. A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computer system with one or more processors, cause the computer system to:

group a set of documents into a plurality of clusters, wherein each cluster includes one or more documents of the set of documents and a respective cluster of documents of the plurality of clusters includes respective cluster data corresponding to a plurality of documents including a first document and a second document;

determine that the second document includes duplicate data that is duplicative of corresponding data in the first document;

identify a respective subset of the respective cluster data that excludes the duplicate data; and

generate an index of the respective subset of the respective cluster data.