Information compartmentalizing data store
A computer-implemented method for extracting data from separate databases and loading the data into a central data store without introducing additional security vulnerabilities is disclosed. The method begins by receiving a configuration comprising one or more transformation functions to be applied to one or more data fields. Values of the one or more data fields are extracted from the databases for a plurality of records, the values being previously tagged with metadata indicating at least a datatype for the values. Based at least in part on the configuration and the metadata tagging, a function from the one or more transformation functions is selected for each value. A second database with a central data store is populated with transformed data. Requestors are not permitted to access to pre-transformed data in the first one or more databases while being allowed access to at least some post-transformed data in the central data store.
1. A system for extracting data from separate databases and loading the data into a central data store without introducing additional security vulnerabilities, comprising:
a first one or more databases containing one or more data fields;
an extract transform and load server comprising one or more processors;
a second database acting as a central data store for data from the first one or more databases; and
non-transitory memory storing instructions that, when executed by the one or more processors, cause the one or more processors to:
receive a configuration comprising one or more transformation functions to be applied to the one or more data fields;
extract, from the first one or more databases, values of the one or more data fields for a plurality of records, the values being previously tagged with metadata indicating at least a datatype for the values;
based at least in part on the configuration and the metadata tagging, select for each value a function from the one or more transformation functions to be performed on that value;
populate the central data store with transformed data; and
enforce, by limiting query access, a constraint that a requestor is not permitted to access to pre-transformed data in the first one or more databases while allowing access to at least some post-transformed data in the central data store.
2. The system of claim 1 , further comprising one or more servers that host microservices, such that when the requestor makes a second query for access, the second query does not obtain data from the central data store, but rather receives information through a microservice that delineates a subset of data to be provided to the requestor.
3. The system of claim 1 , wherein the one or more transformation functions include a function for obfuscating data by masking.
4. The system of claim 1 , wherein the one or more transformation functions include a function for obfuscating data by tokenization.
5. The system of claim 1 , wherein the one or more transformation functions include a function for transforming data by bucketing.
6. The system of claim 1 , wherein the one or more transformation functions include a function for transforming data by a change in datatype.
7. The system of claim 1 , wherein the one or more transformation functions include a function for transforming data by aggregation.
8. The system of claim 1 , wherein the one or more transformation functions include a function for transforming data by deletion.
9. The system of claim 1 , wherein an application programming interface is provided to authenticate requests to access data from the data store and to provide a subset of data from the data store to an authenticated requestor.
10. The system of claim 1 , wherein the configuration comprises a first transformation function to be used on values of a first datatype when those values originate from a first application, and the configuration comprises a second transformation to be used on values of the first datatype when those values originate from a second application different from the first application.
11. A computer-implemented method for extracting data from separate databases and loading the data into a central data store without introducing additional security vulnerabilities, comprising:
receiving a configuration comprising one or more transformation functions to be applied to one or more data fields;
extracting, from a first one or more databases, values of the one or more data fields for a plurality of records, the values being previously tagged with metadata indicating at least a datatype for the values;
based at least in part on the configuration and the metadata tagging, selecting for each value a function from the one or more transformation functions to be performed on that value;
populating a second database with a central data store with transformed data; and
enforcing, by limiting query access, a constraint that a requestor is not permitted to access to pre-transformed data in the first one or more databases while allowing access to at least some post-transformed data in the central data store.
12. The method of claim 11 , wherein when the requestor makes a second query for access, the second query does not obtain data from the central data store, but rather receives information through a microservice that delineates a subset of data to be provided to the requestor.
13. The method of claim 11 , wherein the one or more transformation functions include a function for obfuscating data by masking.
14. The method of claim 11 , wherein the one or more transformation functions include a function for obfuscating data by tokenization.
15. The method of claim 11 , wherein the one or more transformation functions include a function for transforming data by bucketing.
16. The method of claim 11 , wherein the one or more transformation functions include a function for transforming data by a change in datatype.
17. The method of claim 11 , wherein the one or more transformation functions include a function for transforming data by aggregation.
18. The method of claim 11 , wherein the one or more transformation functions include a function for transforming data by deletion.
19. The method of claim 11 , wherein an application programming interface is provided to authenticate requests to access data from the data store and to provide a subset of data from the data store to an authenticated requestor.
20. The method of claim 11 , wherein the configuration comprises a first transformation function to be used on values of a first datatype when those values originate from a first application, and the configuration comprises a second transformation to be used on values of the first datatype when those values originate from a second application different from the first application.