Rebalancing engine for use in rebalancing files in a distributed storage systems
Redistribution of files in a containerized distributed file system is disclosed. Containers each have an engine and a scanner and each of the containers stores files and parameters for characteristics of files stored on the container. A first engine in a first container monitors characteristics of files stored on the first container and, responsive to determining that the parameters for files on the first container exceed one or more predetermined thresholds, communicates with a second engine in a second container to determine a destination container for one or more files from the first container. The second engine in the second container indicates to the first engine in the first container whether the second container is available to receive one or more files from the first container. The first engine triggers file system scanning by the scanner of the first container to identify files to be moved to the second container.
1. A rebalancing engine in a local data container of a distributed file system having a plurality of data containers, each data container having its own rebalancing engine to facilitate rebalancing of files within the distributed file system, the rebalancing engine in the local data container to:
monitor characteristics associated with files stored in a local data container with respect to container-level parameters for the characteristics;
responsive to determining that the container-level parameters exceed one or more predetermined thresholds,
communicate with one or more remote rebalancing engines in remote data containers to determine a selected destination data container to which one or more files from the local data container are to be moved from the local data container;
generate a query to identify the one or more files from the local data container to be moved to the selected destination data container;
receive a response to the query indicating the one or more files to be moved from the local data container;
store results of the query indicating the one or more files to be moved in candidate database; and
cause movement of the one or more files as indicated by the candidate database to the selected data destination container, wherein the one or more files are stored in the selected destination container.
2. The rebalancing engine of claim 1 , wherein determining whether the container-level parameters exceed one or more predetermined thresholds comprises determining if a current size of a file system of the first container is greater than an optimal usage size of the file system of the first container by determining whether an excess size of the file system of the first container is greater that the optimal usage size of the file system of the first container is greater than a threshold.
3. The rebalancing engine of claim 1 , wherein the response to the query identifying the one or more files from the local container to be moved comprises identifiers corresponding to the one or more files from a candidate database.
4. The rebalancing engine of claim 1 , further comprising transmitting the query to a rebalancing scanner in the local container, wherein the rebalancing scanner inserts entries for candidate files into a candidate file database based on analysis of metadata corresponding to the files of the local container.
5. The rebalancing engine of claim 1 , further comprising selecting one or more files to transfer from a set of candidate files based at least on capacity available on the second container and a determination of storage capacity to be gained in the first container in response to movement of a selected file.
6. The method of claim 1 , wherein the rebalancing engine of the local container shares file system characteristic information with at least one rebalancing engine of a remote container on a periodic schedule.
7. The rebalancing engine of claim 1 , wherein the rebalancing engine determines one or more of:
an Optimal Usage Size, which refers to a preselected usage level corresponding to a storage capacity of a container;
a Current Usage Size, which refers to current storage capacity utilized by a container;
a Maximum Usage Size, which refers to a storage capacity value that is greater than the Optimal Usage Size and is used to trigger rebalancing operations; and
a Balancing Threshold, which refers to a threshold value that can be utilized to terminate rebalancing operations.
8. A non-transitory computer readable medium having stored thereon instructions that, when executed, cause a rebalancing engine in a local container of a distributed file system to facilitate rebalancing of files within the distributed file system, the instructions to cause the rebalancing engine to:
monitor characteristics associated with files stored in a local data container with respect to container-level parameters for the characteristics;
responsive to determining that the container-level parameters exceed one or more predetermined thresholds,
communicate with one or more remote rebalancing engines in remote data containers to determine a selected destination data container to which one or more files from the local data container are to be moved from the local data container;
generate a query to identify the one or more files from the local data container to be moved to the selected destination data container;
receive a response to the query indicating the one or more files to be moved from the local data container;
store results of the query indicating the one or more files to be moved in candidate database;
cause movement of the one or more files as indicated by the candidate database to the selected data destination container, wherein the one or more files are stored in the selected destination container.
9. The non-transitory computer readable medium of claim 8 , wherein the instructions that cause the rebalancing engine to determine whether the container-level parameters exceed one or more predetermined thresholds comprises instructions that, when executed, cause the rebalancing engine to determine if a current size of a file system of the first container is greater than an optimal usage size of the file system of the first container by determining whether an excess size of the file system of the first container is greater that the optimal usage size of the file system of the first container is greater than a threshold.
10. The non-transitory computer readable medium of claim 8 , wherein the response to the query identifying the one or more files from the local container to be moved comprises identifiers corresponding to the one or more files from a candidate database.
11. The non-transitory computer readable medium of claim 8 , further comprising instructions that, when executed, cause the engine to transmit the query to a rebalancing scanner in the local container, wherein the rebalancing scanner inserts entries for candidate files into a candidate file database based on analysis of metadata corresponding to the files of the local container.
12. The non-transitory computer readable medium of claim 8 , further comprising instructions that, when executed, cause the engine to select one or more files to transfer from a set of candidate files based at least on capacity available on the second container and a determination of storage capacity to be gained in the first container in response to movement of a selected file.
13. The non-transitory computer readable medium of claim 8 , wherein the rebalancing engine of the local container shares file system characteristic information with at least one engine of a remote container on a periodic schedule.
14. The non-transitory computer readable medium of claim 8 , wherein the rebalancing engine determines one or more of:
an Optimal Usage Size, which refers to a preselected usage level corresponding to a storage capacity of a container;
a Current Usage Size, which refers to current storage capacity utilized by a container;
a Maximum Usage Size, which refers to a storage capacity value that is greater than the Optimal Usage Size and is used to trigger rebalancing operations; and
a Balancing Threshold, which refers to a threshold value that can be utilized to terminate rebalancing operations.
15. A distributed file system comprising:
a plurality of containers each having at least a rebalancing engine to facilitate rebalancing of files within the distributed file system and a scanner, each of the plurality of containers to store one or more files and each of the plurality of containers having corresponding container-level parameters for characteristics of files stored on the container;
wherein a rebalancing first engine in a first container monitors characteristics associated with files stored in a local data container with respect to container-level parameters for the characteristics and, responsive to determining that the parameters for files on the first container exceed one or more predetermined thresholds, communicates with one or more remote rebalancing engines in remote data containers to determine a selected destination data container to which one or more files from the local data container are to be moved from the local data container and generates a query to identify the one or more files from the local data container to be moved to the selected destination data container;
wherein a second rebalancing engine in a second container indicates to the first rebalancing engine in the first container whether the second container is available to receive one or more files from the first container;
wherein the first rebalancing engine, responsive to an indication from the second rebalancing engine that the second container is available to receive one or more files from the first container, receives a response to the query indicating the one or more files to be moved from the local data container, stores results of the query indicating the one or more files to be moved in candidate database, and causes movement of the one or more files as indicated by the candidate database to the selected data destination container, wherein the one or more files are stored in the selected destination container.
16. The distributed file system of claim 15 wherein the container-level parameters comprise one or more of:
an Optimal Usage Size, which refers to a preselected usage level corresponding to a storage capacity of a container;
a Current Usage Size, which refers to current storage capacity utilized by a container;
a Maximum Usage Size, which refers to a storage capacity value that is greater than the Optimal Usage Size and is used to trigger rebalancing operations; and
a Balancing Threshold, which refers to a threshold value that can be utilized to terminate rebalancing operations.
17. The distributed file system of claim 15 further comprising selecting one or more files to transfer from a set of candidate files based at least on capacity available on the second container and a determination of storage capacity to be gained in the first container in response to movement of a selected file.
18. The distributed file system of claim 15 wherein the first rebalancing engine transmits the query to a rebalancing scanner in the local container, wherein the rebalancing scanner inserts entries for candidate files into a candidate file database based on analysis of metadata corresponding to the files of the local container.
19. The distributed file system of claim 15 wherein the first rebalancing engine of the first container shares file system characteristic information with at least the second rebalancing engine on a periodic schedule.
20. The distributed file system of claim 15 wherein determining whether the container-level parameters exceed one or more predetermined thresholds comprises determining if a current size of a file system of the first container is greater than an optimal usage size of the file system of the first container by determining whether an excess size of the file system of the first container is greater that the optimal usage size of the file system of the first container is greater than a threshold.