Corpus expansion using lexical signatures
In an approach to corpus expansion using lexical signatures, one or more computer processors retrieve a donor corpus of text, wherein the donor corpus includes a plurality of documents. One or more computer processors generate a document signature for each of the plurality of documents in the donor corpus. One or more computer processors retrieve a target corpus of text for expansion. One or more computer processors generate a corpus signature for the target corpus. One or more computer processors compare each document signature to the corpus signature. Based on the comparison, one or more computer processors determine a similarity score for each document signature. One or more computer processors rank the plurality of documents by the similarity score. One or more computer processors add one or more top-ranked documents of the plurality of documents to the target corpus.
1. A computer-implemented method comprising:
retrieving, by one or more computer processors, a donor corpus of text, wherein the donor corpus includes a plurality of documents;
generating, by one or more computer processors, a document signature for each of the plurality of documents in the donor corpus;
retrieving, by one or more computer processors, a target corpus of text for expansion;
generating, by one or more computer processors, a corpus signature for the target corpus;
comparing, by one or more computer processors, each document signature to the corpus signature;
based on the comparison, determining, by one or more computer processors, a similarity score for each document signature;
ranking, by one or more computer processors, the plurality of documents by the similarity score; and
adding, by one or more computer processors, one or more top-ranked documents of the plurality of documents to the target corpus.
2. The computer-implemented method of claim 1 , further comprising:
determining, by one or more computer processors, whether the similarity score for each document signature exceeds a pre-defined threshold of similarity.
3. The computer-implemented method of claim 2 , wherein the pre-defined threshold of similarity is selected from the group consisting of: a number of terms that overlap between the document signature and the corpus signature, a percentage of terms that overlap between the document signature and the corpus signature, and a minimum number of overlapping terms.
4. The computer-implemented method of claim 1 , wherein generating the document signature for each of the plurality of documents in the donor corpus further comprises:
determining, by one or more computer processors, a document frequency for each term of a plurality of terms in the donor corpus;
clipping, by one or more computer processors, the plurality of terms by a minimum document frequency;
ranking, by one or more computer processors, one or more remaining terms of the plurality of terms;
selecting, by one or more computer processors, a pre-defined quantity of the one or more remaining terms with a lowest document frequency as the document signature; and
storing, by one or more computer processors, the document signature.
5. The computer-implemented method of claim 1 , wherein generating the corpus signature for the target corpus further comprises:
determining, by one or more computer processors, a target document frequency for each term of a plurality of terms in the target corpus;
clipping, by one or more computer processors, the plurality of terms by a minimum target document frequency;
ranking, by one or more computer processors, one or more remaining terms of the plurality of terms;
selecting, by one or more computer processors, a pre-defined quantity of the one or more remaining terms from the target corpus with a lowest document frequency in the donor corpus as the corpus signature; and
storing, by one or more computer processors, the corpus signature.
6. The computer-implemented method of claim 1 , wherein the document signature is a first list of a pre-defined number of first terms with a lowest frequency of occurrence in each of the plurality of documents in the donor corpus, and wherein the corpus signature is a second list of a pre-defined number of second terms with a lowest frequency of occurrence in the target corpus.
7. The computer-implemented method of claim 1 , wherein the donor corpus of text is a public dataset that includes a large number of documents that span diverse facets of knowledge.
8. A computer program product comprising:
one or more computer readable storage media and program instructions collectively stored on the one or more computer readable storage media, the stored program instructions comprising:
program instructions to retrieve a donor corpus of text, wherein the donor corpus includes a plurality of documents;
program instructions to generate a document signature for each of the plurality of documents in the donor corpus;
program instructions to retrieve a target corpus of text for expansion;
program instructions to generate a corpus signature for the target corpus;
program instructions to compare each document signature to the corpus signature;
based on the comparison, program instructions to determine a similarity score for each document signature;
program instructions to rank the plurality of documents by the similarity score; and
program instructions to add one or more top-ranked documents of the plurality of documents to the target corpus.
9. The computer program product of claim 8 , the stored program instructions further comprising:
program instructions to determine whether the similarity score for each document signature exceeds a pre-defined threshold of similarity.
10. The computer program product of claim 9 , wherein the pre-defined threshold of similarity is selected from the group consisting of: a number of terms that overlap between the document signature and the corpus signature, a percentage of terms that overlap between the document signature and the corpus signature, and a minimum number of overlapping terms.
11. The computer program product of claim 8 , wherein the program instructions to generate the document signature for each of the plurality of documents in the donor corpus comprise:
program instructions to determine a document frequency for each term of a plurality of terms in the donor corpus;
program instructions to clip the plurality of terms by a minimum document frequency;
program instructions to rank one or more remaining terms of the plurality of terms;
program instructions to select a pre-defined quantity of the one or more remaining terms with a lowest document frequency as the document signature; and
program instructions to store the document signature.
12. The computer program product of claim 8 , wherein the program instructions to generate the corpus signature for the target corpus comprise:
program instructions to determine a target document frequency for each term of a plurality of terms in the target corpus;
program instructions to clip the plurality of terms by a minimum target document frequency;
program instructions to rank one or more remaining terms of the plurality of terms;
program instructions to select a pre-defined quantity of the one or more remaining terms from the target corpus with a lowest document frequency in the donor corpus as the corpus signature; and
program instructions to store the corpus signature.
13. The computer program product of claim 8 , wherein the document signature is a first list of a pre-defined number of first terms with a lowest frequency of occurrence in each of the plurality of documents in the donor corpus, and wherein the corpus signature is a second list of a pre-defined number of second terms with a lowest frequency of occurrence in the target corpus.
14. The computer program product of claim 8 , wherein the donor corpus of text is a public dataset that includes a large number of documents that span diverse facets of knowledge.
15. A computer system comprising:
one or more computer processors;
one or more computer readable storage media;
program instructions collectively stored on the one or more computer readable storage media for execution by at least one of the one or more computer processors, the stored program instructions comprising:
program instructions to retrieve a donor corpus of text, wherein the donor corpus includes a plurality of documents;
program instructions to generate a document signature for each of the plurality of documents in the donor corpus;
program instructions to retrieve a target corpus of text for expansion;
program instructions to generate a corpus signature for the target corpus;
program instructions to compare each document signature to the corpus signature;
based on the comparison, program instructions to determine a similarity score for each document signature;
program instructions to rank the plurality of documents by the similarity score; and
program instructions to add one or more top-ranked documents of the plurality of documents to the target corpus.
16. The computer system of claim 15 , the stored program instructions further comprising:
program instructions to determine whether the similarity score for each document signature exceeds a pre-defined threshold of similarity.
17. The computer system of claim 16 , wherein the pre-defined threshold of similarity is selected from the group consisting of: a number of terms that overlap between the document signature and the corpus signature, a percentage of terms that overlap between the document signature and the corpus signature, and a minimum number of overlapping terms.
18. The computer system of claim 15 , wherein the program instructions to generate the document signature for each of the plurality of documents in the donor corpus comprise:
program instructions to determine a document frequency for each term of a plurality of terms in the donor corpus;
program instructions to clip the plurality of terms by a minimum document frequency;
program instructions to rank one or more remaining terms of the plurality of terms;
program instructions to select a pre-defined quantity of the one or more remaining terms with a lowest document frequency as the document signature; and
program instructions to store the document signature.
19. The computer system of claim 15 , wherein the program instructions to generate the corpus signature for the target corpus comprise:
program instructions to determine a target document frequency for each term of a plurality of terms in the target corpus;
program instructions to clip the plurality of terms by a minimum target document frequency;
program instructions to rank one or more remaining terms of the plurality of terms;
program instructions to select a pre-defined quantity of the one or more remaining terms from the target corpus with a lowest document frequency in the donor corpus as the corpus signature; and
program instructions to store the corpus signature.
20. The computer system of claim 15 , wherein the document signature is a first list of a pre-defined number of first terms with a lowest frequency of occurrence in each of the plurality of documents in the donor corpus, and wherein the corpus signature is a second list of a pre-defined number of second terms with a lowest frequency of occurrence in the target corpus.