Distributed dataset modification, retention, and replication
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for data retention and modification. One of the methods includes dividing partitions into a set of generations according to a retention policy; accumulating modification and deletion events that define changes to be applied to data of the distributed dataset; and when a triggering event occurs for a triggered generation in the set of generations, rolling an oldest partition out of the triggered generation, the rolling comprising: if the oldest partition has reached the end of a retention period for the dataset, marking the oldest partition for deletion in the triggered generation; otherwise: creating a new partition corresponding to the data of the oldest partition, wherein the data is cleaned using a scrubbing process; adding the new partition to a next generation in the set of generations; and marking the oldest partition for deletion in the triggered generation.
1. A method of managing data in a distributed dataset stored in partitions, the method comprising:
dividing the partitions into a plurality of generations;
accumulating events that define changes to be applied to data of the distributed dataset; and
in response to a triggering event occurring for a first generation of the plurality of generations, rolling a first partition out of the first generation, the rolling comprising:
cleaning data of the first partition using a scrubbing process based on the defined changes to be applied to the data in the first partition,
creating a second partition and adding the second partition to a second generation of the plurality of generations of the dataset,
writing the cleaned data of the first partition to the second partition in the second generation, and
marking the first partition for deletion in the first generation.
2. The method of claim 1 , wherein the plurality of generations includes a number of generations determined based on one or more of: (i) an overall retention period of the dataset or (ii) an active size of a generation.
3. The method of claim 1 , wherein marking the first partition for deletion comprises marking the first partition inactive in the generation for a time period limited by a specified deletion period.
4. The method of claim 1 , wherein dividing the partitions into the plurality of generations comprises associating each partition with metadata, the metadata for each partition comprising one or more of: (i) generation information of the partition or (ii) information about when the partition was created.
5. The method of claim 1 , wherein adding the second partition to the second generation comprises associating the second partition with metadata comprising one or more of: (i) a time at which the data of the second partition was last cleaned or (ii) generation information of the second partition.
6. The method of claim 1 , further comprising, for each of the accumulated events:
identifying one or more partitions containing data associated with the event;
generating one or more respective scrubbing tasks for modifying the data in the one or more partitions; and
executing the respective scrubbing tasks on a schedule based on a rolling window of partition scrubbing.
7. The method of claim 1 , further comprising, responsive to a subsequent triggering event:
determining that a problem has occurred in creating the second partition; and
performing a partition roll back on first partition to restore the data prior to creating the second partition.
8. One or more non-transitory computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations for managing data in a distributed dataset stored in partitions, the operations comprising:
dividing the partitions into a plurality of generations;
accumulating events that define changes to be applied to data of the distributed dataset; and
in response to a triggering event occurring for a first generation of the plurality of generations, rolling a first partition out of the first generation, the rolling comprising:
cleaning data of the first partition using a scrubbing process based on the defined changes to be applied to the data in the first partition,
creating a second partition and adding the second partition to a second generation of the plurality of generations of the dataset,
writing the cleaned data of the first partition to the second partition in the second generation, and
marking the first partition for deletion in the first generation.
9. The non-transitory computer storage media of claim 8 , wherein the plurality of generations includes a number of generations determined based on one or more of: (i) an overall retention period of the dataset or (ii) an active size of a generation.
10. The non-transitory computer storage media of claim 8 , wherein marking the first partition for deletion comprises marking the first partition inactive in the generation for a time period limited by a specified deletion period.
11. The non-transitory computer storage media of claim 8 , wherein dividing the partitions into the plurality of generations comprises associating each partition with metadata, the metadata for each partition comprising one or more of: (i) generation information of the partition or (ii) information about when the partition was created.
12. The non-transitory computer storage media of claim 8 , wherein adding the second partition to the second generation comprises associating the second partition with metadata comprising one or more of: (i) a time at which the data of the second partition was last cleaned or (ii) generation information of the second partition.
13. The non-transitory computer storage media of claim 8 , the operations further comprising, for each of the accumulated events:
identifying one or more partitions containing data associated with the event;
generating one or more respective scrubbing tasks for modifying the data in the one or more partitions; and
executing the respective scrubbing tasks on a schedule based on a rolling window of partition scrubbing.
14. The non-transitory computer storage media of claim 8 , the operations further comprising, responsive to a subsequent triggering event:
determining that a problem has occurred in creating the second partition; and
performing a partition roll back on first partition to restore the data prior to creating the second partition.
15. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations for managing data in a distributed dataset stored in partitions, the operations comprising:
dividing the partitions into a plurality of generations;
accumulating events that define changes to be applied to data of the distributed dataset; and
in response to a triggering event occurring for a first generation of the plurality of generations, rolling a first partition out of the first generation, the rolling comprising:
cleaning data of the first partition using a scrubbing process based on the defined changes to be applied to the data in the first partition,
creating a second partition and adding the second partition to a second generation of the plurality of generations of the dataset,
writing the cleaned data of the first partition to the second partition in the second generation, and
marking the first partition for deletion in the first generation.
16. The system of claim 15 , wherein the plurality of generations includes a number of generations determined based on one or more of: (i) an overall retention period of the dataset or (ii) an active size of a generation.
17. The system of claim 15 , wherein marking the first partition for deletion comprises marking the first partition inactive in the generation for a time period limited by a specified deletion period.
18. The system of claim 15 , wherein dividing the partitions into the plurality of generations comprises associating each partition with metadata, the metadata for each partition comprising one or more of: (i) generation information of the partition or (ii) information about when the partition was created.
19. The system of claim 15 , wherein adding the second partition to the second generation comprises associating the second partition with metadata comprising one or more of: (i) a time at which the data of the second partition was last cleaned or (ii) generation information of the second partition.
20. The system of claim 15 , the operations further comprising, for each of the accumulated events:
identifying one or more partitions containing data associated with the event;
generating one or more respective scrubbing tasks for modifying the data in the one or more partitions; and
executing the respective scrubbing tasks on a schedule based on a rolling window of partition scrubbing.
21. The system of claim 15 , the operations further comprising, responsive to a subsequent triggering event:
determining that a problem has occurred in creating the second partition; and
performing a partition roll back on first partition to restore the data prior to creating the second partition.