Optimizing files stored in a distributed file system
Methods and systems for optimizing a number of files stored in a distributed file system are provided. The method includes determining whether the size of a plurality of input files in a directory are less than a block size. The method further includes determining whether the size of the input files in the directory divided by a desired file size is greater than a threshold. Additionally, the method includes combining the collection of input files into a reduced set of files.
1. A method for optimizing a number of files stored in a distributed file system, the method comprising:
determining that the size of a plurality of input files in a directory is less than a block size, wherein each file in the plurality of input files comprises a plurality of file parts;
in response to determining that the size of a plurality of input files in the directory is less than a block size, combining the plurality of input files into a reduced set of files;
in response to determining that the size of a second plurality of input files in a second directory is not less than a block size, determining that the size of the second plurality of input files in the second directory divided by a desired file size is greater than a threshold, wherein each file in the second plurality of input files comprises a plurality of file parts;
in response to determining that the size of the second plurality of input files in the second directory divided by a desired file size is greater than a threshold, increasing the desired file size by a block size; and
in response to determining that the size of the second plurality of input files in the second directory divided by a desired file size is not greater than a threshold, combining the second plurality of input files into a second reduced set of files, wherein combining the second plurality of input files comprises:
assigning a random number to each of the plurality of file parts for each file in the second plurality of input files;
grouping the plurality of file parts based on the assigned random numbers to generate a shuffled plurality of file parts; and
reducing the shuffled plurality of file parts based on the assigned random number, wherein all file parts with the same random number are reduced to a same reduced file, and the reduced file has a block size equal to the size of the second plurality of input files divided by the desired file size.
2. The method of claim 1 wherein the threshold results in a minimum file system utilization of 75%.
3. The method of claim 1 wherein the random number is between one and a desired number of files.
4. The method of claim 1 further comprising writing the reduced plurality of file parts to a plurality of output files.
5. The method of claim 1 wherein the block size is 128 MB.
6. The method of claim 1 wherein the distributed file system stores files on a plurality of servers.
7. The method of claim 1 further comprising storing metadata for the reduced set of files in a namenode.
8. A distributed file system comprising one or more processors which, alone or in combination, are configured to provide for performance of the following steps:
determining that the size of a plurality of input files in a directory is less than a block size, wherein each file in the plurality of input files comprises a plurality of file parts;
in response to determining that the size of a plurality of input files in the directory is less than a block size, combining the plurality of input files into a reduced set of files;
in response to determining that the size of a second plurality of input files in a second directory is not less than a block size, determining that the size of the second plurality of input files in the second directory divided by a desired file size is greater than a threshold, wherein each file in the second plurality of input files comprises a plurality of file parts;
in response to determining that the size of the second plurality of input files in the second directory divided by a desired file size is greater than a threshold, increasing the desired file size by a block size; and
in response to determining that the size of the second plurality of input files in the second directory divided by a desired file size is not greater than a threshold, combining the second plurality of input files into a second reduced set of files, wherein combining the second plurality of input files comprises:
assigning a random number to each of the plurality of file parts for each file in the second plurality of input files;
grouping the plurality of file parts based on the assigned random numbers to generate a shuffled plurality of file parts; and
reducing the shuffled plurality of file parts based on the assigned random number, wherein all file parts with the same random number are reduced to a same reduced file, and the reduced file has a block size equal to the size of the second plurality of input files divided by the desired file size.
9. The system according to claim 8 further comprising steps for writing the reduced plurality of file parts to a plurality of output files.
10. A non-transitory computer readable medium storing instructions that when executed by a processor perform the following steps:
determining that the size of a plurality of input files in a directory is less than a block size, wherein each file in the plurality of input files comprises a plurality of file parts;
in response to determining that the size of a plurality of input files in the directory is less than a block size, combining the plurality of input files into a reduced set of files:
in response to determining that the size of a second plurality of input files in a second directory is not less than a block size, determining that the size of the second plurality of input files in the second directory divided by a desired file size is greater than a threshold, wherein each file in the second plurality of input files comprises a plurality of file parts;
in response to determining that the size of the second plurality of input files in the second directory divided by a desired file size is greater than a threshold, increasing the desired file size by a block size; and
in response to determining that the size of the second plurality of input files in the second directory divided by a desired file size is not greater than a threshold, combining the second plurality of input files into a second reduced set of files, wherein combining the second plurality of input files comprises:
assigning a random number to each of the plurality of file parts for each file in the second plurality of input files;
grouping the plurality of file parts based on the assigned random numbers to generate a shuffled plurality of file parts; and
reducing the shuffled plurality of file parts based on the assigned random number, wherein all file parts with the same random number are reduced to a same reduced file, and the reduced file has a block size equal to the size of the second plurality of input files divided by the desired file size.
11. The non-transitory computer readable medium according to claim 10 further comprising instructions for writing the reduced plurality of file parts to a plurality of output files.
12. The non-transitory computer readable medium according to claim 10 further comprising instructions for storing metadata for the reduced set of files in a namenode.