System and method for federated execution of anonymization tasks for large databases

1 . A system for data anonymization, comprising:

a memory operable to store a plurality of database tables comprising sensitive data and a plurality of scripts configured to anonymize the database tables; and

a plurality of subsystems, each comprising one or more processors operably coupled to the memory;

wherein the processors of a first subsystem of the subsystems are configured to:

generate a plurality of tasks to execute the scripts stored in the memory;

generate a plurality of task groups for the plurality of tasks based at least in part upon characteristics associated with the database tables, wherein the characteristics comprises at least size, partition state, underlying database files, location, and drive or disk space;

predict, based at least in part upon the characteristics associated with the database tables, that one or more performance measurements for executing the plurality of task groups using the processors of the first subsystem do not meet required performance levels; and

assign one or more sets of the task groups to one or more second subsystems; and

wherein the processors of at least one of the one or more second subsystems are configured to:

predict the performance measurements for executing the assigned set of task groups using the processors of the second subsystem;

execute the assigned set of task groups using the processors of the second subsystem if the predicted performance measurements meet the required performance levels; and

assign one or more subsets of the assigned set of task groups to one or more third subsystems if the predicted performance measurements do not meet the required performance levels.

2 . The system of claim 1 , wherein predicting the performance measurements is further based on capacity and availability associated with the processors.

3 . The system of claim 1 , wherein predicting the performance measurements is further based on application constraints associated with the database tables.

4 . The system of claim 1 , wherein the first subsystem is configured to control the one or more second subsystems.

5 . The system of claim 1 , wherein the one or more second subsystems are configured to execute the assigned task groups.

6 . The system of claim 1 , wherein the one or more second subsystems comprise a second subsystem configured to control the one or more third subsystems and another second subsystem configured to execute a subset of the assigned task groups.

7 . The system of claim 1 , wherein the processors of the first subsystem are further configured to:

predict the performance measurements for executing the task groups using the processors of the first subsystem meet the required performance levels based on current characteristics associated with the database tables and status of the execution of the task groups; and

reassign all sets of the task groups from the second subsystems to the first subsystem for execution.

8 . The system of claim 1 , wherein the plurality of subsystems are associated with each other based on a tree structure, wherein a root node of the tree structure corresponds to a subsystem configured to control one or more subsystems of the remaining subsystems, and wherein one or more leaf nodes of the tree structure correspond to one or more subsystems configured to execute the task groups.

9 . The system of claim 1 , wherein executing the assigned set of task groups by the processors of one of the second subsystems comprises:

assigning the assigned set of task groups to one or more first processors of the processors;

executing the assigned set of task groups by the first processors;

determining, by one or more second processors of the processors, current capacity and current availability associated with the processors, current characteristics associated with the database tables associated with the assigned set of task groups, and status of the execution of the assigned set of task groups; and

adjusting the execution by the second processors based on one or more of current capacity associated with the processors, current availability associated with the processors, current characteristics associated with the database tables associated with the assigned set of task groups, or the status of the execution of the assigned set of task groups.

10 . A method for data anonymization comprising, by one or more computing systems:

generating, by a first subsystem of the computing systems, a plurality of tasks to execute a plurality of scripts configured to anonymize a plurality of database tables;

generating, by the first subsystem, a plurality of task groups for the plurality of tasks based at least in part upon characteristics associated with the database tables, wherein the characteristics comprises at least size, partition state, underlying database files, location, and drive or disk space;

predicting, by the first subsystem based at least in part upon the characteristics associated with the database tables, that one or more performance measurements for executing the plurality of task groups using processors of the first subsystem do not meet required performance levels;

assigning, by the first subsystem, one or more sets of the task groups to one or more second subsystems associated with the computing systems;

predicting, by at least one of the second subsystems, the performance measurements for executing the assigned set of task groups using processors of the second subsystem;

executing, by the at least one of the second subsystems, the assigned set of task groups using the processors of the second subsystem if the predicted performance measurements meet the required performance levels; and

assigning, by the at least one of the second subsystems, one or more subsets of the assigned set of task groups to one or more third subsystems associated with the computing systems if the predicted performance measurements do not meet the required performance levels.

11 . The method of claim 10 , wherein the first subsystem is configured to control the one or more second subsystems.

12 . The method of claim 10 , wherein the one or more second subsystems comprise a second subsystem configured to control the one or more third subsystems and another second subsystem configured to execute a subset of the assigned task groups.

13 . The method of claim 10 , further comprising:

predicting, by the first subsystem, the performance measurements for executing the task groups using the processors of the first subsystem meet the required performance levels based on current characteristics associated with the database tables and status of the execution of the task groups; and

reassigning, by the first subsystem, all sets of the task groups from the second subsystems to the first subsystem for execution.

14 . A non-transitory computer-readable medium storing instructions that when executed by one or more processors cause the processors to:

generate, by a first subsystem associated with the processors, a plurality of tasks to execute a plurality of scripts configured to anonymize a plurality of database tables;

generate, by the first subsystem, a plurality of task groups for the plurality of tasks based at least in part upon characteristics associated with the database tables, wherein the characteristics comprises at least size, partition state, underlying database files, location, and drive or disk space;

predict, by the first subsystem based at least in part upon the characteristics associated with the database tables, that one or more performance measurements for executing the plurality of task groups using processors of the first subsystem do not meet required performance levels;

assign, by the first subsystem, one or more sets of the task groups to one or more second subsystems associated with the processors;

predict, by at least one of the second subsystems, the performance measurements for executing the assigned set of task groups using processors of the second subsystem;

execute, by the at least one of the second subsystems, the assigned set of task groups using the processors of the second subsystem if the predicted performance measurements meet the required performance levels; and

assign, by the at least one of the second subsystems, one or more subsets of the assigned set of task groups to one or more third subsystems associated with the computing systems if the predicted performance measurements do not meet the required performance levels.

15 . The non-transitory computer-readable medium of claim 14 , wherein the first subsystem is configured to control the one or more second subsystems.

16 . The non-transitory computer-readable medium of claim 14 , wherein the one or more second subsystems comprise a second subsystem configured to control the one or more third subsystems and another second subsystem configured to execute a subset of the assigned task groups.

17 . The non-transitory computer-readable medium of claim 14 , wherein the instructions further cause the processors to:

predict, by the first subsystem, the performance measurements for executing the task groups using the processors of the first subsystem meet the required performance levels based on current characteristics associated with the database tables and status of the execution of the task groups; and

reassign, by the first subsystem, all sets of the task groups from the second subsystems to the first subsystem for execution.