Hadoop distributed file system (HDFS) express bulk file deletion
Hadoop Distributed File System (HDFS) express bulk file deletion is described. A list of files in a Hadoop Distributed File System (HDFS) is generated in order of an associated date of files in the HDFS. A predetermined date for retaining selected files in the HDFS is received. A count of files in the list with the associated date of the files in the HDFS older than the predetermined date is identified. Files for deletion from the HDFS having the associated date older than the predetermined date are identified, a number of the files identified for deletion being equal to the count of files. The number of the files in the HDFS identified for deletion equal to the count of files are deleted.
1 . A method for performing bulk deletion of files in a Hadoop Distributed File System (HDFS), comprising:
identifying files across multiple distributed data nodes in an HDFS for inclusion in a list of files in the HDFS based on a first quantity of days indicating a retention period extending backward in time from a current date for the first quantity of days to a first cutoff date, and based on a second quantity of days indicating a lookback window extending backward in time from the current date for the second quantity of days to a second cutoff date older than the first cutoff date, wherein the lookback window defines a temporal range that extends beyond the retention period to capture additional files for potential deletion;
generating the list of files in the HDFS, the list of files comprising names of the files included in the list of files and creation dates of the files included in the list of files, wherein the files included in the list of files are sorted based on the creation date of each file in the list of files in order from oldest to newest in the list of files;
identifying a count of files in the list of files in the HDFS older than the first cutoff date and within the lookback window;
in response to identifying the count of files in the list of files older than the first cutoff date and within the lookback window is greater than zero, identifying a last occurrence of a closest date to the first cutoff date among the creation dates of the files in the list of files in the HDFS older than the first cutoff date and within the lookback window;
determining a deletion boundary by using the count of files in the list of files older than the first cutoff date and within the lookback window to identify a position in the sorted list of files corresponding to the count, wherein the deletion boundary separates files to be retained from files to be deleted without requiring individual date comparisons for each file;
identifying files in the list of files having the closest date as the creation date and older than the closest date based on the count of files in the list of files older than the first cutoff date and within the lookback window as files for deletion by selecting files from the sorted list up to the deletion boundary;
determining whether a file name associated with each of the identified files for deletion is to be manipulated based on a set of predefined conditions;
manipulating the file name associated with at least one of the identified files for deletion; and
bulk deleting the files in the HDFS identified for deletion based on the manipulated file name associated with the at least one of the identified files for deletion.
2 . The method of claim 1 , further comprising:
in response to the count of files in the list older than the first cutoff date and within the lookback window being equal to zero, increasing the lookback window by receding the second cutoff date until the count of files in the list of files in the HDFS older than the first cutoff date and within the lookback window is greater than zero.
3 . The method of claim 2 , wherein increasing the lookback window by receding the second cutoff date until the count of files in the list of files in the HDFS older than the first cutoff date and within the lookback window is greater than zero is an iterative process by which the second cutoff date is moved backward in time by a predetermined quantity of days.
4 . The method of claim 3 , wherein the predetermined quantity of days is 30 or more.
5 . The method of claim 4 , wherein the predetermined quantity of days is 60 or more.
6 . The method of claim 1 , wherein manipulating the file name associated with the at least one of the identified files comprises:
manipulating text identifying the names of the files in the HDFS identified for deletion to generate manipulated file names having unwanted characters in the text removed.
7 . The method of claim 1 , wherein manipulating the file name associated with the at least one of the identified files further comprises:
passing the manipulated file names as arguments to deletion commands for deleting the files identified for deletion.
8 . The method of claim 1 , wherein manipulating the file name associated with the at least one of the identified files comprises:
identifying the at least one of the identified files for deletion having one or more predefined characters; and
executing a command in association with the file name of the at least one of the identified files for deletion having the one or more predefined characters.
9 . A device for performing bulk deletion of files in a Hadoop Distributed File System (HDFS), comprising:
a memory storing computer-readable instructions; and
a processor connected to the memory, wherein the processor is configured to execute the computer-readable instructions to:
identify files across multiple distributed data nodes in an HDFS for inclusion in a list of files in the HDFS based on a first quantity of days indicating a retention period extending backward in time from a current date for the first quantity of days to a first cutoff date, and based on a second quantity of days indicating a lookback window extending backward in time from the current date for the second quantity of days to a second cutoff date older than the first cutoff date, wherein the lookback window defines a temporal range that extends beyond the retention period to capture additional files for potential deletion;
generate the list of files in the HDFS, the list of files comprising names of the files included in the list of files and creation dates of the files included in the list of files, wherein the files included in the list of files are sorted based on the creation date of each file in the list of files in order from oldest to newest in the list of files;
identify a count of files in the list of files in the HDFS older than the first cutoff date and within the lookback window;
in response to identifying the count of files in the list of files older than the first cutoff date and within the lookback window is greater than zero, identifying a last occurrence of a closest date to the first cutoff date among the creation dates of the files in the list of files in the HDFS older than the first cutoff date and within the lookback window;
determine a deletion boundary by using the count of files in the list of files older than the first cutoff date and within the lookback window to identify a position in the sorted list of files corresponding to the count, wherein the deletion boundary separates files to be retained from files to be deleted without requiring individual date comparisons for each file;
identify files in the list of files having the closest date as the creation date and older than the closest date based on the count of files in the list of files older than the first cutoff date and within the lookback window as files for deletion by selecting files from the sorted list up to the deletion boundary;
determine whether a file name associated with each of the identified files for deletion is to be manipulated based on a set of predefined conditions;
manipulate the file name associated with at least one of the identified files for deletion; and
bulk delete the files in the HDFS identified for deletion based on the manipulated file name associated with the at least one of the identified files for deletion.
10 . The device of claim 9 , wherein, in response to the count of files in the list older than the first cutoff date and within the lookback window being equal to zero, the processor increases the lookback window by receding the second cutoff date until the count of files in the list of files in the HDFS older than the first cutoff date and within the lookback window is greater than zero.
11 . The device of claim 9 , wherein to manipulate the file name associated with the at least one of the identified files, the processor is further configured to manipulate text identifying the names of the files in the HDFS identified for deletion to generate manipulated file names having unwanted characters in the text removed.
12 . The device of claim 11 , wherein to manipulate the file name associated with the at least one of the identified files, the processor is further configured to pass the manipulated file names as arguments to deletion commands for deleting the files identified for deletion.
13 . The device of claim 9 , wherein the processor increases the lookback window by receding the second cutoff date until the count of files in the list of files in the HDFS older than the first cutoff date and within the lookback window is greater than zero by way of an iterative process by which the second cutoff date is moved backward in time by a predetermined quantity of days.
14 . The device of claim 13 , wherein the predetermined quantity of days is 30 or more.
15 . A non-transitory computer-readable media having computer-readable instructions stored thereon, which when executed by a processor causes the processor to perform operations comprising:
identifying files across multiple distributed data nodes in an HDFS for inclusion in a list of files in the HDFS based on a first quantity of days indicating a retention period extending backward in time from a current date for the first quantity of days to a first cutoff date, and based on a second quantity of days indicating a lookback window extending backward in time from the current date for the second quantity of days to a second cutoff date older than the first cutoff date, wherein the lookback window defines a temporal range that extends beyond the retention period to capture additional files for potential deletion;
generating the list of files in the HDFS, the list of files comprising names of the files included in the list of files and creation dates of the files included in the list of files, wherein the files included in the list of files are sorted based on the creation date of each file in the list of files in order from oldest to newest in the list of files;
identifying a count of files in the list of files in the HDFS older than the first cutoff date and within the lookback window;
in response to identifying the count of files in the list of files older than the first cutoff date and within the lookback window is greater than zero, identifying a last occurrence of a closest date to the first cutoff date among the creation dates of the files in the list of files in the HDFS older than the first cutoff date and within the lookback window;
determining a deletion boundary by using the count of files in the list of files older than the first cutoff date and within the lookback window to identify a position in the sorted list of files corresponding to the count, wherein the deletion boundary separates files to be retained from files to be deleted without requiring individual date comparisons for each file;
identifying files in the list of files having the closest date as the creation date and older than the closest date based on the count of files in the list of files older than the first cutoff date and within the lookback window as files for deletion by selecting files from the sorted list up to the deletion boundary;
determining whether a file name associated with each of the identified files for deletion is to be manipulated based on a set of predefined conditions;
manipulating the file name associated with at least one of the identified files for deletion; and
bulk deleting the files in the HDFS identified for deletion based on the manipulated file name associated with the at least one of the identified files for deletion.
16 . The non-transitory computer-readable media of claim 15 , wherein the operations further comprise:
in response to the count of files in the list older than the first cutoff date and within the lookback window being equal to zero, increasing the lookback window by receding the second cutoff date until the count of files in the list of files in the HDFS older than the first cutoff date and within the lookback window is greater than zero.
17 . The non-transitory computer-readable media of claim 15 , wherein manipulating the file name associated with the at least one of the identified files comprises:
manipulating text identifying the names of the files in the HDFS identified for deletion to generate manipulated file names having unwanted characters in the text removed.
18 . The non-transitory computer-readable media of claim 15 , wherein manipulating the file name associated with the at least one of the identified files further comprises:
passing the manipulated file names as arguments to deletion commands for deleting the files identified for deletion.
19 . The non-transitory computer-readable media of claim 15 , wherein increasing the lookback window by receding the second cutoff date until the count of files in the list of files in the HDFS older than the first cutoff date and within the lookback window is greater than zero is an iterative process by which the second cutoff date is moved backward in time by a predetermined quantity of days.
20 . The non-transitory computer-readable media of claim 19 , wherein the predetermined quantity of days is 30 or more.
21 . The non-transitory computer-readable media of claim 20 , wherein the predetermined quantity of days is 60 or more.