Distributed dataset modification, retention, and replication
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for data retention and modification. One of the methods includes dividing partitions into a set of generations according to a retention policy; accumulating modification and deletion events that define changes to be applied to data of the distributed dataset; and when a triggering event occurs for a triggered generation in the set of generations, rolling an oldest partition out of the triggered generation, the rolling comprising: if the oldest partition has reached the end of a retention period for the dataset, marking the oldest partition for deletion in the triggered generation; otherwise: creating a new partition corresponding to the data of the oldest partition, wherein the data is cleaned using a scrubbing process; adding the new partition to a next generation in the set of generations; and marking the oldest partition for deletion in the triggered generation.
1. A method for data retention and modification, according to a retention policy, of a distributed dataset stored in partitions, the method comprising:
dividing the partitions into an ordered set of generations according to the retention policy;
accumulating modification and deletion events that define changes to be applied to data of the distributed dataset; and
when a triggering event occurs for a triggered generation in the set of generations, rolling an oldest partition out of the triggered generation, the rolling comprising:
if the oldest partition has reached the end of a retention period for the dataset, marking the oldest partition for deletion in the triggered generation;
otherwise:
creating a new partition corresponding to the data of the oldest partition, wherein the data is cleaned using a scrubbing process based on the defined changes to be applied to data in the oldest partition;
adding the new partition to a next generation in the ordered set of generations of the dataset; and
marking the oldest partition for deletion in the triggered generation.
2. The method of claim 1 , wherein the set of generations includes a number of generations determined based on an overall retention period of the dataset and an active size of a generation.
3. The method of claim 1 , wherein marking the oldest partition for deletion includes marking the oldest partition inactive in the generation for a time period limited by a specified deletion period.
4. The method of claim 1 , wherein dividing the dataset partitions into the set of generations structures includes associating each dataset partition with metadata that includes generation information and information about when the dataset partition was created.
5. The method of claim 3 , wherein adding the new partition to a next generation in the set of generations of the dataset includes associating the new partition with metadata that represents the clean time and generation information of the new partition.
6. The method of claim 1 , wherein the scrubbing process is associated with at least one of the accumulated modification and deletion events.
7. The method of claim 1 , further comprising:
determining whether the oldest partition reached the end of a retention period for the dataset by:
determining whether the oldest partition has reached the maximum age in the triggered generation and the triggered generation is a last generation in the set of generations.
8. The method of claim 1 , wherein for each of the accumulated modification and deletion events:
identifying one or more partitions containing data associated with the event;
generating one or more respective scrubbing tasks for modifying the data in the one or more partitions; and
executing the respective scrubbing tasks on a schedule based on a rolling window of partition scrubbing.
9. The method of claim 1 , wherein responsive to a subsequent triggering event:
determining that a problem has occurred in creating a subsequent new partition corresponding to a current oldest partition; and
performing a partition roll back on the current oldest partition to restore the data prior to creating the subsequent new partition.
10. One or more non-transitory computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations for data retention and modification, according to a retention policy, of a distributed dataset stored in partitions comprising:
dividing the partitions into an ordered set of generations according to the retention policy;
accumulating modification and deletion events that define changes to be applied to data of the distributed dataset; and
when a triggering event occurs for a triggered generation in the set of generations, rolling an oldest partition out of the triggered generation, the rolling comprising:
if the oldest partition has reached the end of a retention period for the dataset, marking the oldest partition for deletion in the triggered generation;
otherwise:
creating a new partition corresponding to the data of the oldest partition, wherein the data is cleaned using a scrubbing process based on the defined changes to be applied to data in the oldest partition;
adding the new partition to a next generation in the ordered set of generations of the dataset; and
marking the oldest partition for deletion in the triggered generation.
11. The one or more non-transitory computer storage media of claim 10 , wherein the set of generations includes a number of generations determined based on an overall retention period of the dataset and an active size of a generation.
12. The one or more non-transitory computer storage media of claim 10 , wherein marking the oldest partition for deletion includes marking the oldest partition inactive in the generation for a time period limited by a specified deletion period.
13. The one or more non-transitory computer storage media of claim 10 , wherein dividing the dataset partitions into the set of generations structures includes associating each dataset partition with metadata that includes generation information and information about when the dataset partition was created.
14. The one or more non-transitory computer storage media of claim 13 , wherein adding the new partition to a next generation in the set of generations of the dataset includes associating the new partition with metadata that represents the clean time and generation information of the new partition.
15. The one or more non-transitory computer storage media of claim 10 , wherein the scrubbing process is associated with at least one of the accumulated modification and deletion events.
16. The one or more non-transitory computer storage media of claim 10 , further comprising instructions that, when executed, cause the one or more computers to perform operations comprising:
determining whether the oldest partition reached the end of a retention period for the dataset by:
determining whether the oldest partition has reached the maximum age in the triggered generation and the triggered generation is a last generation in the set of generations.
17. The one or more non-transitory computer storage media of claim 10 , wherein for each of the accumulated modification and deletion events:
identifying one or more partitions containing data associated with the event;
generating one or more respective scrubbing tasks for modifying the data in the one or more partitions; and
executing the respective scrubbing tasks on a schedule based on a rolling window of partition scrubbing.
18. The one or more non-transitory computer storage media of claim 10 , wherein responsive to a subsequent triggering event:
determining that a problem has occurred in creating a subsequent new partition corresponding to a current oldest partition; and
performing a partition roll back on the current oldest partition to restore the data prior to creating the subsequent new partition.
19. A system comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations for data retention and modification, according to a retention policy, of a distributed dataset stored in partitions comprising:
dividing the partitions into an ordered set of generations according to the retention policy;
accumulating modification and deletion events that define changes to be applied to data of the distributed dataset; and
when a triggering event occurs for a triggered generation in the set of generations, rolling an oldest partition out of the triggered generation, the rolling comprising:
if the oldest partition has reached the end of a retention period for the dataset, marking the oldest partition for deletion in the triggered generation;
otherwise:
creating a new partition corresponding to the data of the oldest partition, wherein the data is cleaned using a scrubbing process based on the defined changes to be applied to data in the oldest partition;
adding the new partition to a next generation in the ordered set of generations of the dataset; and
marking the oldest partition for deletion in the triggered generation.
20. The system of claim 19 , wherein the set of generations includes a number of generations determined based on an overall retention period of the dataset and an active size of a generation.
21. The system of claim 19 , wherein marking the oldest partition for deletion includes marking the oldest partition inactive in the generation for a time period limited by a specified deletion period.
22. The system of claim 19 , wherein dividing the dataset partitions into the set of generations structures includes associating each dataset partition with metadata that includes generation information and information about when the dataset partition was created.
23. The system of claim 22 , wherein adding the new partition to a next generation in the set of generations of the dataset includes associating the new partition with metadata that represents the clean time and generation information of the new partition.
24. The system of claim 19 , wherein the scrubbing process is associated with at least one of the accumulated modification and deletion events.
25. The system of claim 19 , further storing instructions that are operable to cause the one or more computers to perform operations comprising:
determining whether the oldest partition reached the end of a retention period for the dataset by:
determining whether the oldest partition has reached the maximum age in the triggered generation and the triggered generation is a last generation in the set of generations.
26. The system of claim 19 , wherein for each of the accumulated modification and deletion events:
identifying one or more partitions containing data associated with the event;
generating one or more respective scrubbing tasks for modifying the data in the one or more partitions; and
executing the respective scrubbing tasks on a schedule based on a rolling window of partition scrubbing.
27. The system of claim 19 , wherein responsive to a subsequent triggering event:
determining that a problem has occurred in creating a subsequent new partition corresponding to a current oldest partition; and
performing a partition roll back on the oldest partition to restore the data prior to creating the subsequent new partition.