Grouping documents by content similarity

1. A method to facilitate grouping a corpus of documents by content similarity, the method comprising:

determining, by a processing device, a first set of document identifiers (IDs) for a first set of documents of the corpus of documents, the first set of documents identified responsive to a search query;

obtaining at least one shingle corresponding to the first set of document IDs;

identifying a second set of document IDs corresponding to the obtained at least one shingle;

grouping, by the processing device, the second set of document IDs in a grouping data structure based on document IDs;

determining, by the processing device, resulting document IDs from the grouping data structure, wherein each resulting document ID from the resulting document IDs is determined based on a threshold number of occurrences; and

associating, by the processing device, resulting documents of the first set of documents that correspond to the resulting document IDs with one another in search results generated responsive to the search query to create a final search result to send to a user.

2. The method of claim 1 , wherein the search query comprises receiving a keyword search term.

3. The method of claim 1 , wherein the at least one shingle comprises a contiguous subsequences of tokens in a document of the first documents that corresponds to the document ID.

4. The method of claim 1 , wherein the grouping structure is a key-value data structure allowing fast search of elements of the key-value data structure.

5. The method of claim 1 , wherein the grouping structure is at least one of a hashmap container data structure or a tree data structure.

6. The method of claim 1 , wherein each document ID of the second set corresponds to a single record in the grouping structure, and wherein each record comprises a counter to track a number of the occurrences of the document ID in the second set.

7. The method of claim 1 , wherein associating the resulting documents further comprising ordering the resulting documents within the search results so that the resulting documents corresponding to the resulting documents IDs are next to one another.

8. The method of claim 1 , wherein determining the resulting document IDs further comprises disregarding the document ID of the first set for which the at least one shingle was obtained.

9. The method of claim 1 , wherein responsive to one of the resulting document IDs not comprised in the first set of document IDs, providing a document corresponding to the one of the resulting document IDs as a suggested related document in the search results.

10. The method of claim 1 , wherein the at least one shingle is obtained from a shingles data store generated as part of indexing of the corpus of documents.

11. The method of claim 10 , wherein generating shingles records for shingles in the shingles data store comprises:

obtaining an obtained document of the corpus of documents and a corresponding document ID of the document from a document repository;

parsing the obtained document to extract textual content;

removing non-meaning words from the extracted textual content to generate resulting text;

converting the resulting text to a normalized format;

generating the shingles over the normalized format;

for each of the generated shingles:

searching the shingles data store for the generated shingle;

responsive to a record existing for the generated shingle in the shingles data store, adding the corresponding document ID to the record; and

responsive to the generated shingle not existing in the shingles data store, generating a new record for the generated shingle in the shingles data store and adding the corresponding document ID to the record; and

attaching the generated shingles to metadata of the document in the document repository.

12. The method of claim 11 , wherein generating the shingles records occurs during at least one of indexing or re-indexing of the document in the document repository.

13. The method of claim 11 , wherein the second set of documents IDs is obtained from the shingles data store.

14. A server computing system comprising:

a memory; and

a processing device operatively coupled to the memory, the processing device to:

determine a first set of document identifiers (IDs) for a first set of documents of the corpus of documents, the first set of documents identified responsive to a search query;

obtain at least one shingle corresponding to the first set of document IDs;

identify a second set of document IDs corresponding to the obtained at least one shingle;

group the second set of document IDs in a grouping data structure based on document IDs;

determine resulting document IDs from the grouping data structure, wherein each resulting document ID from the resulting document IDs is determined based on a threshold number of occurrences;

associate resulting documents of the first set of documents that correspond to the resulting document IDs with one another in search results generated responsive to the search query to create a final search result to send to a user; and

responsive to one of the resulting document IDs not comprised in the first set of document IDs, provide a document corresponding to the one of the resulting document IDs as a suggested related document in the search results.

15. The server computing system of claim 14 , wherein the at least one shingle comprise contiguous subsequences of tokens in the documents.

16. The server computing system of claim 14 , wherein the grouping structure is a key-value data structure allowing fast search of elements of the key-value data structure.

17. The server computing system of claim 14 , wherein each document ID of the second set corresponds to a single record in the grouping structure, and wherein each record comprises a counter to track a number of the occurrences of the document ID in the second set.

18. The server computing system of claim 14 , wherein the processing device to associate the resulting documents further comprises the processing device to order the resulting documents within the search results so that the resulting documents corresponding to the resulting documents IDs are next to one another.

19. The server computing system of claim 14 , wherein the processing device to determine the resulting document IDs further comprises the processing device to disregard the document ID of the first set for which the at least one shingle was obtained.

20. The server computing system of claim 14 , wherein the at least one shingle is obtained from a shingles data store generated as part of indexing of the corpus of documents.

21. The server computing system of claim 14 , wherein the processing device to generate shingles records for the shingles in the shingles data store further comprises the processing device to:

obtain an obtained document of the corpus of documents and a corresponding document ID of the document from a document repository;

parse the obtained document to extract textual content;

remove non-meaning words from the extracted textual content to generate resulting text;

convert the resulting text to a normalized format;

generate the shingles over the normalized format;

for each of the generated shingles:

search the shingles data store for the generated shingle;

responsive to a record existing for the generated shingle in the shingles data store, add the corresponding document ID to the record; and

responsive to the generated shingle not existing in the shingles data store, generate a new record for the generated shingle in the shingles data store and add the corresponding document ID to the record; and

attach the generated shingles to metadata of the document in the document repository.

22. The server computing system of claim 21 , wherein the processing device to generate the shingles records occurs during at least one of indexing or re-indexing of the document in the document repository.

23. A non-transitory computer readable storage medium storing instructions that, when executed by a processing device of a client device, cause the processing device to:

determine a first set of document identifiers (IDs) for a first set of documents of the corpus of documents, the first set of documents identified responsive to a search query;

obtain at least one shingle corresponding to the first set of document IDs;

identify a second set of document IDs corresponding to the obtained at least one shingle;

group the second set of document IDs in a grouping data structure based on document IDs;

determine resulting document IDs from the grouping data structure, wherein each resulting document ID from the resulting document IDs is determined based on a threshold number of occurrences; and

associate resulting documents of the first set of documents that correspond to the resulting document IDs with one another in search results generated responsive to the search query to create a final search result to send to a user.

24. The non-transitory computer readable storage medium of claim 23 , wherein the at least one shingle comprise contiguous subsequences of tokens in the documents.

25. The non-transitory computer readable storage medium of claim 23 , wherein the grouping structure is a key-value data structure allowing fast search of elements of the key-value data structure.

26. The non-transitory computer readable storage medium of claim 23 , wherein each document ID of the second set corresponds to a single record in the grouping structure, and wherein each record comprises a counter to track a number of the occurrences of the document ID in the second set.

27. The non-transitory computer readable storage medium of claim 23 , wherein the processing device to associate the resulting documents further comprises the processing device to order the resulting documents within the search results so that the resulting documents corresponding to the resulting documents IDs are next to one another.

28. The non-transitory computer readable storage medium of claim 23 , wherein the processing device is further to, responsive to one of the resulting document IDs not comprised in the first set of document IDs, provide a document corresponding to the one of the resulting document IDs as a suggested related document in the search results.

29. The non-transitory computer readable storage medium of claim 23 , wherein the processing device to generate shingles records for the shingles in the shingles data store further comprises the processing device to:

obtain an obtained document of the corpus of documents and a corresponding document ID of the document from a document repository;

parse the obtained document to extract textual content;

remove non-meaning words from the extracted textual content to generate resulting text;

convert the resulting text to a normalized format;

generate the shingles over the normalized format;

for each of the generated shingles:

search the shingles data store for the generated shingle;

responsive to a record existing for the generated shingle in the shingles data store, add the corresponding document ID to the record; and

responsive to the generated shingle not existing in the shingles data store, generate a new record for the generated shingle in the shingles data store and add the corresponding document ID to the record; and

attach the generated shingles to metadata of the document in the document repository.

30. The non-transitory computer readable storage medium of claim 29 , wherein the processing device to generate the shingles records occurs during at least one of indexing or re-indexing of the document in the document repository.