System and method for real-time transactional data obfuscation
A system and method for providing transactional data privacy while maintaining data usability, including the use of different obfuscation functions for different data types to securely obfuscate the data, in real-time, while maintaining its statistical characteristics. In accordance with an embodiment, the system comprises an obfuscation process that captures data while it is being received in the form of data changes at a first or source system, selects one or more obfuscation techniques to be used with the data according to the type of data captured, and obfuscates the data, using the selected one or more obfuscation techniques, to create an obfuscated data, for use in generating a trail file containing the obfuscated data, or applying the data changes to a target or second system.
1. A system, implemented on a computer comprising one or more processors, for providing transactional data privacy while maintaining data usability, comprising:
a capture process, executing on the one or more processors, that
monitors a first or source system, and
captures a transaction containing one or more changes to data while the transaction is being received and committed at the first or source system; and
an obfuscation process that
receives a signal from the capture process when the capture process detects that the transaction is being committed,
wherein meta-data associated with the data contained in the transaction includes a data structure describing a distribution of data values that is incrementally maintained,
obfuscates the transaction using an obfuscation technique to create an obfuscated transaction,
wherein the obfuscation technique includes a nearest neighbor data substitution process combined with anonymization whereby
a distance between the data and an origin point is calculated,
a nearest neighbor point in the data structure is determined, and
a geometric transformation is applied to the nearest neighbor point, generating an obfuscated value,
wherein at least some of the data contained in the transaction cannot be determined from the obfuscated transaction, and
sends the obfuscated transaction back to the capture process for use in generating a trail file or other information to be sent to a target or second system, wherein the trail file or other information reflects the obfuscated transaction.
2. The system of claim 1 , wherein the first or source system and the target or second system are transaction-based systems, and wherein one of more of the transactions are to replicate data changes at the first or source system in real-time to the target or second system.
3. The system of claim 1 , wherein the first or source system includes a first database and the target or second system includes a second database, and wherein the data changes are replicated between the first and second databases in real-time.
4. The system of claim 1 , wherein the obfuscation process is configured according to one or more parameter files that include dictionary names, and the histogram that are used to classify the data.
5. A method for providing transactional data privacy while maintaining data usability, comprising the steps of:
monitoring a first or source system with a capture process;
capturing a transaction containing one or more changes to data while the transaction is being received and committed at the first or source system;
sending a signal, from the capture process to an obfuscation process, when the capture process detects that the transaction is being committed;
wherein meta-data associated with the data contained in the transaction includes a data type and a data structure describing a distribution of data values that is incrementally maintained;
obfuscating the transaction, using an obfuscation technique, to create an obfuscated transaction;
wherein the obfuscation technique includes a nearest neighbor data substitution process combined with anonymization whereby
a distance between the data and an origin point is calculated,
a nearest neighbor point in the data structure is determined, and
a geometric transformation is applied to the nearest neighbor point, generating an obfuscated value;
wherein at least some of the data contained in the transaction cannot be determined from the obfuscated transaction;
sending the obfuscated transaction back to the capture process; and
generating a trail file or other information to be sent to a target or second system, wherein the trail file or other information reflects the obfuscated transaction.
6. The method of claim 5 , wherein the first or source system and the target or second system are transaction-based systems, and wherein one of more of the transactions are to replicate data changes at the first or source system in real-time to the target or second system.
7. The method of claim 5 , wherein the first or source system includes a first database and the target or second system includes a second database, and wherein the data changes are replicated between the first and second databases in real-time.
8. The method of claim 5 , wherein the obfuscation process is configured according to one or more parameter files that include dictionary names, and the histograms that are used to classify the data.
9. A non-transitory computer readable storage medium, including instructions stored thereon which when read and executed by a computer cause the computer to perform the steps comprising:
monitoring a first or source system with a capture process;
capturing a transaction containing one or more changes to data while the transaction is being received and committed at the first or source system;
sending a signal, from the capture process to an obfuscation process, when the capture process detects that the transaction is being committed;
wherein meta-data associated with the data contained in the transaction includes a data structure describing a distribution of data values that is incrementally maintained;
obfuscating the transaction, using an obfuscation technique, to create an obfuscated transaction;
wherein the obfuscation technique includes a nearest neighbor data substitution process combined with anonymization whereby
a distance between the data and an origin point is calculated,
a nearest neighbor point in the data structure is determined, and
a geometric transformation is applied to the nearest neighbor point, generating an obfuscated value;
wherein at least some of the data contained in the transaction cannot be determined from the obfuscated transaction;
sending the obfuscated transaction back to the capture process; and
generating a trail file or other information to be sent to a target or second system, wherein the trail file or other information reflects the obfuscated transaction.
10. The non-transitory computer readable storage medium of claim 9 , wherein the first or source system and the target or second system are transaction-based systems, and wherein one of more of the transactions are to replicate data changes at the first or source system in real-time to the target or second system.
11. The non-transitory computer readable storage medium of claim 9 , wherein the first or source system includes a first database and the target or second system includes a second database, and wherein the data changes are replicated between the first and second databases in real-time.
12. The non-transitory computer readable storage medium of claim 9 , wherein the obfuscation process is configured according to one or more parameter files that include dictionary names, and histograms that are used to classify the data.