Pre-processing of files for normalization of fixed length fragment comparison

1. A method of pre-processing a file by creating that file's derivative that contains data or the purpose of comparing its fragments of fixed size N to other fragments of size N in other files, method comprising the steps of:

obtaining the file for pre-processing;

creating a derivative of the file that is different from the file by

de-compiling a source code of the file, using a decompiler corresponding to the compiler used to compile the file,

re-compiling the file with the compiler specific to a digital fingerprint library,

identifying a language of the text of the file, using Artificial Intelligence, for translating the text of the file from the identified language to a pre-defined language using an automatic translation tool,

translating at least a portion of the text of the file to the pre-defined language using the automatic translation tool,

translating at least a portion of the text of the file to the pre-defined language using a manual translation method,

implementing an automatic text enhancement tool to enhance the text of the file for spell check, thesaurus, grammar check, sentence correction, and language optimization;

creating a collection of strings of size greater than fixed size N from the derivative of the file;

storing the collection of strings of size greater than fixed size N;

adding fingerprints of fragments of fixed size N from the derivative of the file to the digital fingerprint library;

comparing fingerprints of certain fragments of fixed size N from the derivative of the file to the digital fingerprint library;

calculating, for each of the strings in the collection of strings, the number of fragments of fixed size N from the derivative of the file that are substrings of the string and have a matching fingerprint in the digital fingerprint library; and

when the number of fragments of fixed size N that are substrings of the string have matches in the digital fingerprint library, determining the string has a common source with the matches in the digital fingerprint library.

2. The method of claim 1 , wherein the step of obtaining file for pre-processing further

comprises the step of determining parameters for classification associated with the file, such as file format; and the step of creating a derivative is based on the determination of these parameters.

3. The method of claim 1 , wherein the step of creating a derivative file further comprises

the steps of:

a. extracting text from a file containing text;

b. converting the text to a pre-defined codepage;

c. implementing Unicode normalization forms;

d. replacing symbols with modifiers to Latin symbols without modifiers;

e. decomposing diacritics and digraphs, and removing combining marks from text;

f. removing double spaces from the text;

g. removing spaces from text;

h. removing punctuation marks from the text;

i. removing certain characters from the text, e.g., vowels;

j. removing words, indicating “noise” expressions, from the text;

k. removing footers, headers, and standard segments including salutations and signatures from the text;

l. replacing non-Latin characters according to predefined rules; or

m. decompiling binary files to obtain source code in text format.

4. A system implemented to pre-process a file by creating its derivative for the purpose of comparison of the fingerprints of its fragments of length N to fragments of length N from other files, wherein the system comprises:

a digital fingerprint library including a memory for storing fingerprints of fragments;

a processor;

a file receiver configured to obtain a file for pre-processing;

a pre-processor for pre-processing the file, wherein the pre-processor is configured to create a derivative of the file by;

de-compiling a source code of the file, using a decompiler corresponding to the compiler used to compile the file, and

re-compiling the file with the compiler specific to the digital fingerprint library,

identifying a language of the text of the file, using Artificial Intelligence, for translating the text of the file from the identified language to a pre-defined language using an automatic translation tool,

translating at least a portion of the text of the file to a pre-defined language using the automatic translation tool,

translating at least a portion of the text of the file to the pre-defined language using a manual translation method, and

implementing an automatic text enhancement tool to enhance the text of the file for spell check, thesaurus, grammar check, sentence correction, and language optimization;

a digital fingerprint library manager configured to add fingerprints of certain fragments of size N of the derivative of the file to the digital fingerprint library by creating a collection of strings of size greater than fixed size N from the derivative of the file, and storing the collection of strings of size greater than fixed size N;

a comparison engine that compares fingerprints of certain fragments of size N of the derivative of the file to the digital fingerprint library including:

calculating, for each of the strings in the collection of strings, the number of fragments of fixed size N from the derivative of the file that are substrings of the string and have a matching fingerprint in the digital fingerprint library; and

when the number of fragments of fixed size N that are substrings of the string have matches in the digital fingerprint library, determining the string has a common source with the matches in the digital fingerprint library.

5. The system of claim 4 , wherein the file receiver is further configured to determine

parameters for classification associated with the file and the pre-processor for preprocessing the file is configured to perform pre-processing based on the determination of these parameters.

6. The system of claim 4 , wherein the pre-processor configured to perform the symmetric processing is further configured to:

a. extract text from a file containing protected information;

b. convert the text to a pre-defined codepage;

c. implement Unicode normalization forms;

d. replace symbols with modifiers to Latin symbols without modifiers;

e. decompose diacritics and digraphs, and removing combining marks from the text;

f. remove spaces from the text;

g. remove punctuation marks from the text;

h. remove certain characters from the text;

i. remove words, indicating noise expressions, from the text;

j. remove footers, headers, and standard segments including salutations and signatures from the text;

k. replace non-Latin characters according to predefined rules; and

l. decompile binary files to obtain source code in text format.