Big data retention management system

1 . A method for data retention, comprising:

parsing, by a partition semantics system, a metastore table to extract a first table property, the metastore table having one or more time-related partitions, the first table property corresponding to a retention policy having a retention action and a first retention period;

in response to determining that the first retention period applies to a first time-related partition of the metastore table, evaluating, by the partition semantics system, the first time-related partition against the first retention period;

in response to determining that the first time-related partition is out-of-retention, creating, by the partition semantics system, a retention event specifying the first time-related partition and the retention action;

storing, by the partition semantics system, the retention event in a retention events database; and

consuming, by an event consumption system, the retention event by performing operations in the following order:

(1) serializing partition metadata of the first time-related partition as a string and copying the string to an archive storage bucket;

(2) creating, by the event consumption system, a storage transfer job to copy data of the first time-related partition from a source storage to a further archive storage bucket, wherein the storage transfer job deletes the data of the first time-related partition at the further archive storage bucket when the data of the first time-related partition at the further archive storage bucket differs from the source storage or when the data of the first time-related partition only exists at the further archive storage bucket;

(3) dropping, by the event consumption system, the first time-related partition from the metastore table; and

(4) deleting, by the event consumption system, the first time-related partition at the source storage.

2 . The method of claim 1 , wherein evaluating, by the partition semantics system, the first time-related partition against the first retention period comprises:

determining, by the partition semantics system, a time based on one or more time-related partition values of the first time-related partition; and

evaluating, by the partition semantics system, the time against a current time.

3 . The method of claim 1 , further comprising:

parsing, by the partition semantics system, the metastore table to extract a second table property specifying a custom date partition key to which the retention policy applies.

4 . The method of claim 3 , further comprising:

parsing, by the partition semantics system, the metastore table to extract a third table property specifying a custom date format used by the custom date partition key.

5 . The method of claim 1 , wherein the retention action is an archive action.

6 . The method of claim 1 , wherein the retention action is a delete action.

7 . The method of claim 1 , wherein the first retention period specifies a number of days.

8 . The method of claim 1 , wherein the first table property comprises a unique identifier identifying the retention policy in a configuration file that associates different unique identifiers to different retention policies.

9 . The method of claim 1 , wherein the retention event is consumable by a consumer and causes the consumer to perform the retention action on the first time-related partition.

10 . The method of claim 1 , wherein:

creating the retention event is performed in response to determining that no active restore request is present in a restore requests database for the first time-related partition; and

a restore request in the restore requests database requests archived data to be restored and is considered active when the restore request has not yet expired.

11 . The method of claim 1 , further comprising:

parsing, by the partition semantics system, the metastore table to extract a fourth table property specifying a second retention period overriding the first retention period; and

in response to determining that the second retention period applies to the first time-related partition of the metastore table, evaluating, by the partition semantics system, the first time-related partition against the second retention period.

12 . The method of claim 1 , further comprising:

parsing, by the partition semantics system, the metastore table to extract a fifth table property specifying a filter criterion and a third retention period overriding the first retention period; and

in response to determining that the third retention period applies to the first time-related partition of the metastore table based on the first time-related partition meeting the filter criterion, evaluating, by the partition semantics system, the first time-related partition against the third retention period.

13 . The method of claim 12 , wherein the filter criterion specifies a partition key and a partition value.

14 . The method of claim 12 , wherein the filter criterion specifies a path prefix.

15 . The method of claim 1 , further comprising:

determining, by the partition semantics system, an applicable retention period for the first time-related partition based on an order of precedence and one or more table properties of the metastore table, the order of precedence including, in the following order, a path prefix retention period override or a partition-level retention period override, a table-level retention period override, and a table-level retention period.

16 . One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to:

parse a metastore table to extract a first table property, the metastore table having one or more time-related partitions, the first table property corresponding to a retention policy having a retention action and a first retention period;

in response to determining that the first retention period applies to a first time-related partition of the metastore table, evaluate the first time-related partition against the first retention period;

in response to determining that the first time-related partition is out-of-retention, create a retention event specifying the first time-related partition and the retention action;

store the retention event in a retention events database; and

consume the retention event by performing operations in the following order:

(1) serializing partition metadata of the first time-related partition as a string and copying the string to an archive storage bucket;

(2) creating a storage transfer job to copy data of the first time-related partition from a source storage to a further archive storage bucket, wherein the storage transfer job deletes the data of the first time-related partition at the further archive storage bucket when the data of the first time-related partition at the further archive storage bucket differs from the source storage or when the data of the first time-related partition only exists at the further archive storage bucket;

(3) dropping the first time-related partition from the metastore table; and

(4) deleting the first time-related partition at the source storage.

17 . The one or more non-transitory computer-readable media of claim 16 , wherein evaluating the first time-related partition against the first retention period comprises:

determining a time based on one or more time-related partition values of the first time-related partition; and

evaluating the time against a current time.

18 . The one or more non-transitory computer-readable media of claim 16 , wherein the instructions further cause the one or more processors to:

parse the metastore table to extract a second table property specifying a custom date partition key to which the retention policy applies.

19 . A computer-implemented system, comprising:

one or more processors, and

one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to:

parse a metastore table to extract a first table property, the metastore table having one or more time-related partitions, the first table property corresponding to a retention policy having a retention action and a first retention period;

in response to determining that the first retention period applies to a first time-related partition of the metastore table, evaluate the first time-related partition against the first retention period;

in response to determining that the first time-related partition is out-of-retention, create a retention event specifying the first time-related partition and the retention action;

store the retention event in a retention events database; and

consume the retention event by performing operations in the following order:

(1) serializing partition metadata of the first time-related partition as a string and copying the string to an archive storage bucket;

(2) creating a storage transfer job to copy data of the first time-related partition from a source storage to a further archive storage bucket, wherein the storage transfer job deletes the data of the first time-related partition at the further archive storage bucket when the data of the first time-related partition at the further archive storage bucket differs from the source storage or when the data of the first time-related partition only exists at the further archive storage bucket;

(3) dropping the first time-related partition from the metastore table; and

(4) deleting the first time-related partition at the source storage.

20 . The computer-implemented system of claim 19 , wherein the first retention period specifies a number of days.