System and method for on-demand data cleansing
A system and method for on-demand data cleansing is disclosed. The system includes a processor and a volatile memory including a data object having a plurality of data fields, each field having a tag and a value. The system also includes a cleansing module stored in the volatile memory and executed by the processor. The cleansing module includes a library having a plurality of tag-operation pairs. The cleansing module is configured to receive a pointer locating the data object within volatile memory, and further configured to, for each data field, look up the tag of the data field among the library tags, and execute the operation paired with the matching library tag on the value of the field, modifying the value of the data field while it is stored in volatile memory. The cleansing module is configured to execute the operation in near real-time, and at runtime.
1 . A system for on-demand data cleansing, comprising:
a processor;
a network interface communicatively coupled to the processor and a network;
an API executed by the processor that performs actions including:
receiving a new tag-operation pair through the network interface,
identifying an existing tag-operation pair within a library having a plurality of tag-operation pairs, each tag-operation pair comprising a library tag and an associated operation, the existing tag-operation pair having a library tag that matches a tag of the new tag-operation pair,
in response to identifying the existing tag-operation pair, updating an operation of the existing tag-operation pair with an operation of the new tag-operation pair, and
receiving, into volatile memory, a data object having a data field; and
a cleansing module executed by the processor that performs actions including:
executing an operation paired with the library tag of the existing tag-operation pair on a value of the data field in response to determining that the data field is associated with a tag that matches the library tag of the existing tag-operation pair, the operation modifying the value of the data field while it is stored in the volatile memory.
2 . The system of claim 1 , wherein executing the operation on the value of the data field comprises one of overwriting at least some of the value within the volatile memory and replacing the value within the data object with a reformatted value.
3 . The system of claim 1 , wherein executing the operation on the value of the data field comprises:
generating a new value based upon the value, the new value having less specificity than the value; and
replacing the value with the new value within the data object stored in the volatile memory.
4 . The system of claim 3 , wherein the value comprises an image, and wherein generating the new value based upon the value comprises:
using machine vision to detect any faces within the image; and
generating a new image having all detected faces obscured.
5 . The system of claim 3 , wherein the value comprises a geolocation, and wherein generating the new value based upon the value comprises resolving the geolocation within a geographic information system and generating a geographic descriptor of lesser specificity than the geolocation.
6 . The system of claim 1 , wherein the operation defines logic whose execution brings information described by the paired library tag into compliance with a policy.
7 . The system of claim 1 , wherein the actions further include:
receiving a pointer locating the data object within the volatile memory; and
receiving the data object through the network interface, store the data object in the volatile memory, and pass the pointer locating the data object within the volatile memory to the cleansing module.
8 . The system of claim 6 , wherein the policy is HIPAA.
9 . The system of claim 1 , wherein the library is a first library from a plurality of libraries, each library implementing a different policy.
10 . A method for on-demand data cleansing, comprising:
receiving a tag-operation pair via an API;
identifying an existing tag-operation pair within a library having a plurality of tag-operation pairs, each tag-operation pair comprising a library tag and an associated operation, the existing tag-operation pair having a library tag that matches a library tag of the tag-operation pair received via the API;
in response to identifying the existing tag-operation pair, replacing the operation of the existing tag-operation pair with the operation of the tag-operation pair received via the API;
receiving, via the API, a data object comprising a data field having a tag and a value; and
using the replaced operation stored in the library to update the value of the data field in response to determining that the library tag matches the tag of the data field,
wherein, for each tag-operation pair of the library, the operation defines logic whose execution brings information described by the paired library tag into compliance with a policy.
11 . The method of claim 10 , wherein using the replaced operation comprises evaluating, at runtime, a string defining operation logic and stored in the tag-operation pair.
12 . The method of claim 10 , wherein using the replaced operation to update the value of the data field is performed in near real-time.
13 . The method of claim 10 , wherein using the replaced operation to update the value of the data field comprises one of overwriting at least some of the value within a volatile memory and replacing the value within the data object with a reformatted value.
14 . The method of claim 10 , wherein using the replaced operation to update the value of the data field comprises:
generating a new value based upon the value yet having less specificity; and
replacing the value with the new value within the data object stored in volatile memory.
15 . The method of claim 14 , wherein the value comprises an image, and wherein generating the new value based upon the value comprises:
using machine vision to detect any faces within the image; and
generating a new image having all detected faces obscured.
16 . The method of claim 14 , wherein the value comprises a geolocation, and wherein generating the new value based upon the value comprises resolving the geolocation within a geographic information system and determining a geographic descriptor of lesser specificity than the geolocation.
17 . The method of claim 10 , further comprising:
receiving a new tag-operation pair through the API; and
storing the new tag-operation pair within the library.
18 . The method of claim 10 , further comprising storing the data object in a volatile memory.
19 . The method of claim 10 , wherein the policy is HIPAA.
20 . The method of claim 10 , further comprising selecting the library from a plurality of libraries, each library implementing a different policy.