Systems and methods for identifying duplicate documents and detecting misrepresentation

1 . A system comprising:

a memory storing instructions; and

at least one processor configured to execute the stored instructions to:

retrieve, via a network, from a first source from a first source comprising a repository of previously submitted documents, a first document;

retrieve, via the network, from a second source distinct from the first source and comprising a repository of newly submitted documents, a second document;

process the first document and the second document, wherein processing comprises:

cleaning each of the first document and the second document by removing personally identifiable metadata and hazardous hidden content comprising malicious scripts or malware,

tokenizing each of the first document and the second document by substituting sensitive data elements with non-sensitive tokens,

removing stop words, and

vectorizing each of the first document and the second document into numeric representations comprising sentence-level vectors and word vectors;

align sentences between the first document and the second document based on cosine similarity between the sentence-level vectors generated during vectorizing, and, within aligned sentence pairs:

determine, based on the cosine similarity, a number W of words in common, and

determine a number C of sentences in common equal to a count of aligned sentence pairs whose pairwise cosine similarity exceeds a sentence-pair threshold stored in the memory;

determine a cosine similarity S between a first set of numbers and a second set of numbers, the first set of numbers corresponding to one or more sentences in the first document and the second set of numbers corresponding to one or more sentences in the second document, wherein each number in the first set of numbers corresponds to a word in the first document and each number in the second set of numbers corresponds to a word in the second document;

determine whether the cosine similarity exceeds a first threshold stored in the memory;

determine, based on the cosine similarity and the sentence alignment, a number of words in common between the first document and the second document equal to W;

determine whether the number of words W in common exceeds a second threshold stored in the memory;

determine, based on the cosine similarity and the sentence alignment, a number of sentences in common between the first document and the second document equal to C;

determine whether the number of sentences in common C exceeds a third threshold stored in the memory; and

responsive to a determination that the cosine similarity S exceeds the first threshold, and at least one of: the number of words W in common exceeds the second threshold, or the number of sentences in common C exceeds the third threshold:

persistently write, to a record associated with the second document in a vector repository, a duplicate flag field stored as at least one bit that indicates that the second document is a duplicate.

2 . The system of claim 1 , wherein the at least one processor is further configured to:

iterate the processing, determining, and flag writing steps for each of a plurality of documents retrieved from the second source, until the second source no longer contains any documents to process.

3 . The system of claim 2 , wherein the at least one processor is further configured to provide for display, on a graphical user interface, a number of duplicate flags.

4 . The system of claim 1 , wherein the first source includes a repository of previously submitted documents.

5 . The system of claim 1 , wherein the second source includes a repository of newly submitted documents.

6 . The system of claim 1 , wherein processing further includes extracting and storing text and metadata on a distributed computer network.

7 . The system of claim 6 , wherein the at least one processor is further configured to extract text and metadata using at least one of:

natural language processing;

optical character recognition;

a k-nearest neighbors (KNN) algorithm; or

an Artificial Neural Network (ANN) algorithm.

8 . The system of claim 1 , wherein the at least one processor is further configured to load one or more batches of processed first documents into a vector repository.

9 . The system of claim 1 , wherein the first threshold is between 0.5 and 1.

10 . The system of claim 9 , wherein the first threshold is 0.85.

11 . The system of claim 1 , wherein the second threshold is between 50 and 150.

12 . The system of claim 11 , wherein the second threshold is 100.

13 . The system of claim 1 , wherein the third threshold is between 5 and 10.

14 . The system of claim 13 , wherein the third threshold is 6.

15 . The system of claim 1 , wherein cleaning further includes:

removing malicious scripts;

removing personal identifying information; or

removing malware from the first document and the second document.

16 . The system of claim 1 , wherein tokenizing further includes substituting a sensitive data element with a non-sensitive data element using at least one of: word tokenization, character tokenization, or subword tokenization.

17 . A method comprising:

retrieving, via a network, from a first source from a first source comprising a repository of previously submitted documents, a first document;

retrieving, via the network, from a second source distinct from the first source and comprising a repository of newly submitted documents, a second document;

processing the first document and the second document, wherein processing comprises:

cleaning each of the first document and the second document by removing personally identifiable metadata and hazardous hidden content comprising malicious scripts or malware,

tokenizing each of the first document and the second document by substituting sensitive data elements with non-sensitive tokens,

removing stop words, and

vectorizing each of the first document and the second document into numeric representations comprising sentence-level vectors and word vectors;

align sentences between the first document and the second document based on cosine similarity between the sentence-level vectors generated during vectorizing, and, within aligned sentence pairs:

determine, based on the cosine similarity, a number W of words in common, and

determine a number C of sentences in common equal to a count of aligned sentence pairs whose pairwise cosine similarity exceeds a sentence-pair threshold stored in the memory;

determining a cosine similarity S between a first set of numbers and a second set of numbers, the first set of numbers corresponding to one or more sentences in the first document and the second set of numbers corresponding to one or more sentences in the second document, wherein each number in the first set of numbers corresponds to a word in the first document and each number in the second set of numbers corresponds to a word in the second document;

determining whether the cosine similarity exceeds a first threshold stored in the memory;

determining, based on the cosine similarity and the sentence alignment, a number of words in common between the first document and the second document equal to W;

determining whether the number of words W in common exceeds a second threshold stored in the memory;

determining, based on the cosine similarity and the sentence alignment, a number of sentences in common between the first document and the second document equal to C;

determining whether the number of sentences in common C exceeds a third threshold stored in the memory; and

responsive to a determination that the cosine similarity exceeds S the first threshold, and at least one of: the number of words in common W exceeds the second threshold, or the number of sentences in common C exceeds the third threshold:

persistently write, to a record associated with the second document in a vector repository, a duplicate flag field stored as at least one bit that indicates that the second document is a duplicate.

18 . The method of claim 17 , further including iterating the processing, determining, and flag writing steps for each of a plurality of documents retrieved from the second source, until the second source no longer contains any documents to process.

19 . A non-transitory computer readable medium having stored instructions, which when executed, cause at least one processor to perform instructions comprising:

retrieving, via a network, from a first source from a first source comprising a repository of previously submitted documents, a first document;

retrieving, via the network, from a second source distinct from the first source and comprising a repository of newly submitted documents, a second document;

processing the first document and the second document, wherein processing comprises:

cleaning each of the first document and the second document by removing personally identifiable metadata and hazardous hidden content comprising malicious scripts or malware,

tokenizing each of the first document and the second document by substituting sensitive data elements with non-sensitive tokens, removing stop words, and

vectorizing each of the first document and the second document into numeric representations comprising sentence-level vectors and word vectors;

align sentences between the first document and the second document based on cosine similarity between the sentence-level vectors generated during vectorizing, and, within aligned sentence pairs:

determine, based on the cosine similarity, a number W of words in common, and

determine a number C of sentences in common equal to a count of aligned sentence pairs whose pairwise cosine similarity exceeds a sentence-pair threshold stored in the memory;

determining a cosine similarity S between a first set of numbers and a second set of numbers, the first set of numbers corresponding to one or more sentences in the first document and the second set of numbers corresponding to one or more sentences in the second document, wherein each number in the first set of numbers corresponds to a word in the first document and each number in the second set of numbers corresponds to a word in the second document;

determining whether the cosine similarity exceeds a first threshold stored in the memory;

determining, based on the cosine similarity and the sentence alignment, a number of words in common between the first document and the second document equal to W;

determining whether the number of words W in common exceeds a second threshold stored in the memory;

determining, based on the cosine similarity and the sentence alignment, a number of sentences in common between the first document and the second document equal to C;

determining whether the number of sentences in common C exceeds a third threshold stored in the memory; and

responsive to a determination that the cosine similarity exceeds S the first threshold, and at least one of: the number of words in common W exceeds the second threshold, or the number of sentences in common C exceeds the third threshold:

persistently write, to a record associated with the second document in a vector repository, a duplicate flag field stored as at least one bit that indicates that the second document is a duplicate.

20 . The non-transitory computer readable medium of claim 19 ,

wherein the at least one processor is further configured to:

iterate the processing, determining, and flag writing steps for each of a plurality of documents retrieved from the second source, until the second source no longer contains any documents to process.