Data watermarking and fingerprinting system and method
A system for applying fingerprinting/watermarking of consumer data, and analyzing “wild files” of consumer data to assign a guilt score for a particular party who may have leaked the data, allows the owner of data sources (“Data Owners”) to identify and assert ownership of textual data that has been distributed outside of their firewall in the clear (i.e., without encryption), either intentionally or unintentionally, and assign guilt to parties misusing the data. The system can be used by Data Owners who transmit, lease, or sell data to individuals or organizations (“Trusted Third Parties” or “TTPs”) to recognize and assert ownership of their data in the case where one or more TTPs leaks the data (the leaked file is defined as a “Leaked Subset”) into the hands of others (“Bad Actors”) who either knowingly or unknowingly use the data illegally.
1. A method for watermarking and fingerprinting a data set, the method comprising the steps of:
reading the data set from an input data file or feed into a computing system memory;
while the data set still resides in the computing system memory, applying a watermark to the data set, wherein the watermark in the data set is not detectable by a third party after the watermark is applied, and wherein a watermarking algorithm chosen to apply the watermark to the data set is chosen from a plurality of potential watermarking algorithms;
while the data set still resides in the computing system memory, extracting a metadata set from the data set;
while the data set still resides in the computing system memory, creating a fingerprint for the data, wherein the fingerprint comprises a subset of data smaller than the data set but that is sufficient to identify the data set and wherein the fingerprint comprises the metadata set, further wherein the fingerprint is a visual fingerprint comprising statistical patterns in fields to form a statistical image, and further wherein a fingerprinting algorithm chosen to apply the fingerprint is chosen from a plurality of potential fingerprinting algorithms; and
after the step of reading data set from an input data file or feed, processing the data set at a file loader to extract taxonomy information if taxonomy information is available in the data set.
2. The method of claim 1 , further comprising the step of assigning each field of the data set to a field category based on the extracted taxonomy if taxonomy information is available, or using statistical methods to assign each field of the data set to a field category if extracted taxonomy information is not available.
3. The method of claim 2 , further comprising the steps of:
assigning to each field category a standard field name; and
converting each field value in the data set to a field value taken from a standard list of field values.
4. The method of claim 3 , further comprising the step of, if a field is missing, executing a call to a comprehensive commercial database with consumer information to append information into the missing field.