Secure deployment of de-risked confidential data within a distributed computing environment
In some examples, computer-implemented systems and processes deploy securely de-risked elements of confidential data within a distributed computing environment. For example, an apparatus may obtain configuration data associated with a source data table. The configuration data may specify an identifier of a column of the source data table that includes elements of confidential data, and based on the configuration data, the apparatus perform operations that anonymize the elements of confidential data within the column of the source data table and generate an anonymized column within the source data table. The apparatus may also perform operations that provision an anonymized data table that includes the anonymized column to at least one computing system, which may process the anonymized data table and generate an output data table that includes the anonymized column and maintains a referential integrity between the source data table and the output data table.
1. An apparatus, comprising:
a communications interface;
a memory storing instructions; and
at least one processor coupled to the communications interface and the memory, the at least one processor being configured to execute the instructions to:
access a source data table, and obtain configuration data and a delta table associated with the source data table, the configuration data comprising an identifier of a column of the source data table that includes elements of confidential data, and the delta table comprising data that identifies a de-risking operation associated with the column of the source data table;
based on the configuration data, perform operations that apply the de-risking operation to the elements of confidential data within the column of the source data table and generate anonymized elements of confidential data, the anonymized elements of confidential data establishing an anonymized column within the source data table; and
perform operations that provision an anonymized data table that includes the anonymized column to at least one computing system via the communications interface, the at least one computing system being configured to process the anonymized data table and generate an output data table that includes the anonymized column, the anonymized column maintaining a referential integrity between the source data table and the output data table.
2. The apparatus of claim 1 , wherein:
the anonymized elements of confidential data preserve at least one of a format, a structure, or a composition of the elements of confidential data; and
the at least one computing system is configured to apply a trained machine-learning or artificial intelligence process to portions of the anonymized data table.
3. The apparatus of claim 1 , wherein the at least one processor is further configured to:
store, within a portion of the delta table, update data associated with a performance of the operations that anonymize the elements of confidential data within the column of the source data table.
4. The apparatus of claim 3 , wherein:
the source data table comprises a plurality of temporal partitions;
the de-risking operation is associated with one or more of the temporal partitions;
the delta table comprises data that identifies each of the temporal partitions and the de-risking operation associated with the one or more of the temporal partitions, each of the temporal partitions being associated with a corresponding temporal interval; and
the at least one processor is further configured to execute the instructions to:
select a corresponding one of the temporal partitions based on the delta table; and
based on the configuration data, perform the operations that anonymize the elements of confidential data within the column of the corresponding temporal partition of the source data table.
5. The apparatus of claim 1 , wherein:
the configuration data further comprises an identifier of an additional column of the source data table and information characterizing a pre-processing operation associated with the additional column, the pre-processing operations comprising a full-redaction operation or a partial-redaction operation; and
the at least one processor is further configured to execute the instructions to apply the pre-processing operation to elements of data within the additional column based on the configuration data.
6. The apparatus of claim 5 , wherein the at least one processor is further configured to execute the instructions to:
store, within a portion of the delta table, update data associated with an application of the pre-processing operation to the elements of data within the additional column.
7. The apparatus of claim 1 , wherein the at least one processor is further configured to execute the instructions to
obtain elements of table schema data associated with the source data table, the elements of table schema data comprising the identifier of the column and information characterizing at least one of a format, a structure, or a composition of the elements of confidential data; and
access the elements of confidential data within the column of the source data table based on the elements of table schema data.
8. The apparatus of claim 7 , wherein the at least one processor is further configured to execute the instructions to:
obtain type-preserving mapping data that includes a first one of the elements of confidential data;
obtain, from the elements of type-preserving mapping data, a first element of anonymized data associated with the first element of confidential data; and
perform operations that replace the first element of confidential data within the column of the source data table with the first element of anonymized data.
9. The apparatus of claim 8 , wherein the at least one processor is further configured to execute the instructions to:
determine that the type-preserving mapping data fails to include a second one of the elements of confidential data;
based on the elements of table schema data, generate a second element of anonymized data associated with the second element of confidential data; and
perform operations that replace the second element of confidential data within the column of the source data table with the second element of anonymized data.
10. The apparatus of claim 8 , wherein the anonymized elements of confidential data comprise at least one of an element of fully anonymized data or an element of partially anonymized data.
11. The apparatus of claim 1 , wherein the at least one processor is further configured to execute the instructions to generate the anonymized data table based on an application of a join operation to the anonymized column within the source data table and to a corresponding anonymized column within one or more additional source data tables.
12. The apparatus of claim 1 , wherein the at least one processor is further configured to execute the instructions to:
obtain additional configuration data associated with an additional source data table, the configuration data comprising an identifier of a column of the additional source data table that includes additional elements of confidential data;
based on the additional configuration data, access the additional elements of confidential data within the column of the additional source data table;
obtain elements of token-based mapping data that associate each of the additional elements of confidential data with a corresponding digital token;
obtain, from the elements of token-based mapping data, the corresponding digital token associated with each of the additional elements of confidential data, and perform operations that replace each of the additional elements of confidential data within the column with the corresponding digital token, the digital tokens establishing a tokenized column within the additional source data table; and
perform operations that provision a tokenized data table that includes the tokenized column to the at least one computing system via the communications interface.
13. A computer-implemented method, comprising:
using at least one processor, accessing a source data table, and obtaining configuration data and a delta table associated with the source data table, the configuration data comprising an identifier of a column of the source data table that includes elements of confidential data, and the delta table comprising data that identifies a de-risking operation associated with the column of the source data table;
based on the configuration data, performing operations, using the at least one processor, that apply the de-risking operation to the elements of confidential data within the column of the source data table and generate anonymized elements of confidential data, the anonymized elements of confidential data establishing an anonymized column within the source data table; and
performing operations, using at least one processor, that provision an anonymized data table that includes the anonymized column to at least one computing system, the at least one computing system being configured to process the anonymized data table and generate an output data table that includes the anonymized column, the anonymized column maintaining a referential integrity between the source data table and the output data table.
14. The computer-implemented method of claim 13 , wherein:
the anonymized elements of confidential data preserve at least one of a format, structure, or composition of the elements of confidential data;
the anonymized elements of confidential data comprise at least one of an element of fully anonymized data or an element of partially anonymized data; and
the at least one computing system being configured to apply a trained machine-learning or artificial-intelligence process to portions of the anonymized data table.
15. The computer-implemented method of claim 13 , wherein:
the source data table comprises a plurality of temporal partitions;
the de-risking operation is associated with one or more of the temporal partitions; and
the computer-implemented method further comprises:
obtaining, using the at least one processor, the delta table associated with the source data table, the delta table comprising data that identifies each of the temporal partitions and the de-risking operation associated with the one or more of the temporal partitions, each of the temporal partitions being associated with a corresponding temporal interval;
selecting, using the at least one processor, a corresponding one of the temporal partitions based on the delta table; and
performing the operations that anonymize the elements of confidential data within the column of the corresponding temporal partition.
16. The computer-implemented method of claim 13 , wherein:
the configuration data further comprises an identifier of an additional column of the source data table and information characterizing a pre-processing operation associated with the additional column, the pre-processing operations comprising a full-redaction operation or a partial-redaction operation; and
the computer-implemented method further comprises applying, using the at least one processor, the pre-processing operation to elements of data within the additional column based on the configuration data.
17. The computer-implemented method of claim 13 , wherein performing the operations that anonymize the elements of confidential data comprises:
obtaining type-preserving mapping data that includes a first one of the elements of confidential data;
obtaining, from the elements of type-preserving mapping data, a first element of anonymized data associated with the first element of confidential data; and
performing operations that replace the first element of confidential data within the column of the source data table with the first element of anonymized data.
18. The computer-implemented method of claim 17 , wherein performing the operations that anonymize the elements of confidential data comprises:
determining that the type-preserving mapping data fails to include a second one of the elements of confidential data;
based on the elements of table schema data, generating a second element of anonymized data associated with the second element of confidential data; and
performing operations that replace the second element of confidential data within the column of the source data table with the second element of anonymized data.
19. The computer-implemented method of claim 13 , further comprising generating, using the at least one processor, the anonymized data table based on an application of a join operation to the anonymized column within the source data table and to a corresponding anonymized column within one or more additional source data tables.
20. A tangible, non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform a method, comprising:
accessing a source data table, and obtaining configuration data and a delta table associated with the source data table, the configuration data comprising an identifier of a column of the source data table that includes elements of confidential data, and the delta table comprising data that identifies a de-risking operation associated with the columns of the source data table;
based on the configuration data, performing operations that apply the de-risking operation to the elements of confidential data within the column of the source data table and generate anonymized elements of confidential data, the anonymized elements of confidential data establishing an anonymized column within the source data table; and
performing operations that provision an anonymized data table that includes the anonymized column to at least one computing system, the at least one computing system being configured to process the anonymized data table and generate an output data table that includes the anonymized column, the anonymized column maintaining a referential integrity between the source data table and the output data table.