System and method for finding similar documents based on semantic factual similarity
The present disclosure is directed towards systems and methods for finding documents that are similar to a reference text. The inventive systems and methods examine a set of collected documents to determine the facts present in those documents by, for example, extracting triplets and expanding them. A user's input reference text is similarly examined to extract and expand triplets therein and the facts identified with respect to the reference text are used as a basis to find documents having similar facts. The present disclosure is also related to systems and methods for mining facts from documents relating to a primary source such as a piece of legislation and using the mined facts to improve the results of subsequent searches.
1 . A system comprising memory and one or more processors communicatively coupled to the memory, the one or more processors configured to:
receive a reference text string via a user interface of a user computing device;
extract a plurality of reference triples from the reference text string;
expand each of the plurality of reference triples based at least in part on candidate multi-word combinations of a domain-specific semantic corpus to obtain a plurality of expanded reference triples, wherein each of the plurality of reference triples is expanded based at least in part on a normalization of one or more tokens of the associated reference triple to a base form prior to further expansion of the associated reference triple based at least in part on the domain-specific semantic corpus;
access an index based upon each of the plurality of expanded reference triples, wherein the index includes a plurality of library document triples associated with each of a plurality of library documents;
identify, based upon the access of the index, a plurality of library document triples and associated library documents similar to each of the plurality of expanded reference triples; and
scoring the plurality of library documents and generating a scored list of the plurality of library documents based at least in part upon similarity scores between the plurality of expanded reference triples and the library document triples; and
displaying the generated scored list of library documents to the user via the user interface.
2 . The system of claim 1 , wherein the one or more processors are further configured to:
identify one or more fact groups related to each library document based at least in part on the associated expanded reference triples.
3 . The system of claim 1 , wherein the one or more processors are further configured to:
form multi-word tokens as components of the plurality of reference triples based at least in part on the domain-specific semantic corpus; and
normalize the multi-word tokens to the base form prior to further expansion of the plurality of reference triples.
4 . The system of claim 1 , wherein the one or more processors are further configured to:
score the plurality of library documents based at least in part on an aggregation of the similarity scores.
5 . The system of claim 1 , wherein the one or more processors are further configured to:
determine a similarity score of a library document based at least in part on a comparison between a respective library document triple and a corresponding expanded reference triple; and
add the library document to the list in response to a determination that the similarity score satisfies a defined threshold value.
6 . The system of claim 1 , wherein the reference text string is a first reference text string, and wherein the one or more processors are further configured to:
receive a second reference text string via the user interface;
extract at least one second reference triple from the second reference text string;
identify one or more expanded library triples similar to the at least one second reference triple; and
update the list of library documents based at least in part on the one or more expanded library triples.
7 . The system of claim 1 , wherein the normalization of the one or more tokens is based at least in part on a natural language processing technique.
8 . The system of claim 1 , wherein the normalization of the one or more tokens is based at least in part on a stemming and lemmatization normalization technique.
9 . A method, comprising:
receiving a reference text string via a user interface of a user computing device;
extracting a plurality of reference triples from the reference text string;
expanding each of the plurality of reference triples based at least in part on candidate multi-word combinations of a domain-specific semantic corpus to obtain a plurality of expanded reference triples, wherein each of the plurality of reference triples is expanded based at least in part on a normalization of one or more tokens of the associated reference triple to a base form prior to further expansion of the associated reference triple based at least in part on the domain-specific semantic corpus;
accessing an index based upon each of the plurality of expanded reference triples, wherein the index includes a plurality of library document triples associated with each of a plurality library documents;
identifying, based upon the access of the index, a plurality of library document triples and associated library documents similar to each of the plurality of expanded reference triples; and
scoring the plurality of library documents and generating a scored list of the plurality of library documents based at least in part upon similarity scores between the plurality of expanded reference triples and the library document triples; and
displaying the generated scored list of library documents to the user.
10 . The method of claim 9 , further comprising:
identifying one or more fact groups related to each library document based at least in part on the associated expanded reference triples.
11 . The method of claim 9 , further comprising:
forming multi-word tokens as components of the plurality of reference triples based at least in part on the domain-specific semantic corpus; and
normalizing the multi-word tokens to the base form prior to further expansion of the plurality of reference triples.
12 . The method of claim 9 , further comprising:
scoring the plurality of library documents based at least in part on an aggregation of the similarity scores.
13 . The method of claim 9 , further comprising:
determining a similarity score of a library document based at least in part on a comparison between a respective library document triple and a corresponding expanded reference triple; and
adding the library document to the list in response to a determination that the similarity score satisfies a defined threshold value.
14 . The method of claim 9 , wherein the reference text string is a first reference text string, and the method further comprising:
receiving a second reference text string via the user interface;
extracting at least one second reference triple from the second reference text string;
identifying one or more expanded library triples similar to the at least one second reference triple; and
updating the list of scored library documents based at least in part on the one or more expanded library triples.
15 . The method of claim 9 , wherein the normalization of the one or more tokens is based at least in part on a natural language processing technique.
16 . The method of claim 9 , wherein the normalization of the one or more tokens is based at least in part on a stemming and lemmatization normalization technique.
17 . A computer program product, stored on a non-transitory computer readable medium, comprising instructions that when executed by one or more processors cause the one or more processors to:
receive a reference text string via a user interface of a user computing device;
extract a plurality of reference triples from the reference text string;
expand each of the plurality of reference triples based at least in part on candidate multi-word combinations of a domain-specific semantic corpus to obtain a plurality of expanded reference triples, wherein each of the plurality of reference triples is expanded based at least in part on a normalization of one or more tokens of the associated reference triple to a base form prior to further expansion of the associated reference triple based at least in part on the domain-specific semantic corpus;
assess an index based upon each of the plurality of expanded reference triples, wherein the index includes a plurality of library document triples associated with each of the plurality of library documents;
identify, based upon the access of the index, a plurality of library document triples and associated library documents similar to each of the plurality of expanded reference triples; and
score the plurality of library documents and generating a scored list of the plurality of library documents based at least in part upon similarity scores between the plurality of expanded reference triples and the library document triples; and
display the generated scored list of library documents to the user via the user interface.
18 . The computer program product of claim 17 , wherein the one or more processors are further configured to:
identify one or more fact groups related to each library document based at least in part on the associated expanded reference triples.
19 . The computer program product of claim 17 , wherein the one or more processors are further configured to:
form multi-word tokens as components of the plurality of reference triples based at least in part on the domain-specific semantic corpus; and
normalize the multi-word tokens to the base form prior to further expansion of the plurality of reference triples.
20 . The computer program product of claim 17 , wherein the one or more processors are further configured to:
score the plurality of library documents based at least in part on an aggregation of the similarity scores.