Scale out deduplicated file system as microservices
A deduplicated file system includes a set of microservices including front-ends and back-ends. Assignments of files are balanced across front-ends. The files are represented by segment trees including multiple segment levels. Assignments of similarity groups are balanced across back-ends. Similarity groups are associated with segments at a lower-level of the segment trees that form the files. Front-ends are responsible for operations involving an upper-level of the trees. Back-ends are responsible for operations involving the lower-level of the trees. A mapping of file assignments to front-ends and of similarity group assignments to back-ends is stored. A request to perform a file system operation is received. The mapping is consulted to identify particular front and back-ends that should be responsible for handling and processing the request.
1. A method for a deduplicated file system as a plurality of microservices comprising front-ends and back-ends, the method comprising:
balancing assignments of files in the file system across the front-ends, the files being represented by segment trees;
balancing assignments of similarity groups across the back-ends, the similarity groups comprising segments at a lower-level of the segment trees that form the files, the front-ends being responsible for operations involving an upper-level of the segment trees, and the back-ends being responsible for operations involving the lower-level of the segment trees;
storing a mapping of file assignments to the front-ends, and of similarity group assignments to the back-ends;
receiving, by a load balancer, a request to perform a file system operation;
consulting, by the load balancer, the file assignments to identify a particular front-end that should be assigned the request;
consulting, by the particular front-end, the similarity group assignments to identify a particular back-end that should be responsible for processing segments associated with the request, the segments belonging to a similarity group that has been assigned to the particular back-end; and
sending, by the particular front-end, the segments to the particular back-end.
2. The method of claim 1 wherein the request to perform the file system operation comprises an identification of a file and the method further comprises:
receiving, by the load balancer, a subsequent request involving the same file; and
assigning, by the load balancer, the subsequent request to the same particular front-end.
3. The method of claim 1 wherein the request to perform the file system operation comprises incoming data for a file, and the method further comprises:
segmenting, by the particular front-end, the data into a plurality of segments corresponding to the lower-level of segments in the segment trees;
representing each of one or more subsets of the lower-level of segments by one or more respective higher-level segments, each of the one or more higher-level segments being associated with one or more respective similarity groups; and
sending each of the one or more higher-level segments, including the one or more subsets of the lower-level segments, to one or more back-ends according to the similarity group assignments.
4. The method of claim 1 further comprising:
receiving an indication from a container orchestration service that an initial number of the plurality of microservices has been scaled to a new number, different from the initial number; and
based on the new number of microservices, rebalancing one or more of the file assignments to the front-ends or the similarity group assignments to the back-ends.
5. The method of claim 1 further comprising:
receiving an indication from a container orchestration service that an initial number of the front-ends has been scaled to a new number, different from the initial number; and
replacing previous assignments of files with new assignments of files,
wherein when the scaling comprises increasing a number of the front-ends, the new assignments of files comprise new ranges of files for which additional front-ends and previous front-ends are now responsible, a number of files in a new range of files now being less than a number of files in a previous range of files, each previous front-end thereby now being responsible for a fewer number of files, and
wherein when the scaling comprises decreasing the number of front-ends, the new assignments of files comprise new ranges of files for which remaining front-ends are now responsible, a number of files in the new range of files now being greater than the number of files in the previous range of files, each remaining front-end thereby now being responsible for a greater number of files.
6. The method of claim 1 further comprising:
receiving an indication from a container orchestration service that an initial number of the back-ends has been scaled to a new number, different from the initial number; and
replacing previous assignments of similarity groups with new assignments of similarity groups,
wherein when the scaling comprises increasing a number of the back-ends, the new assignments of back-ends comprise new ranges of similarity groups for which additional back-ends and previous back-ends are now responsible, a number of similarity groups in a new range of similarity groups now being less than a number of similarity groups in a previous range of similarity groups, each previous back-end thereby now being responsible for a fewer number of similarity groups, and
wherein when the scaling comprises decreasing the number of back-ends, the new assignments of similarity groups comprise new ranges of similarity groups for which remaining back-ends are now responsible, a number of similarity groups in the new range of similarity groups now being greater than the number of similarity groups in the previous range of similarity groups, each remaining back-end thereby now being responsible for a greater number of similarity groups.
7. A system for a deduplicated file system as a plurality of microservices comprising front-ends and back-ends, the system comprising: a processor; and memory configured to store one or more sequences of instructions which, when executed by the processor, cause the processor to carry out the steps of:
balancing assignments of files in the file system across the front-ends, the files being represented by segment trees;
balancing assignments of similarity groups across the back-ends, the similarity groups comprising segments at a lower-level of the segment trees that form the files, the front-ends being responsible for operations involving an upper-level of the segment trees, and the back-ends being responsible for operations involving the lower-level of the segment trees;
storing a mapping of file assignments to the front-ends, and of similarity group assignments to the back-ends;
receiving, by a load balancer, a request to perform a file system operation;
consulting, by the load balancer, the file assignments to identify a particular front-end that should be assigned the request;
consulting, by the particular front-end, the similarity group assignments to identify a particular back-end that should be responsible for processing segments associated with the request, the segments belonging to a similarity group that has been assigned to the particular back-end; and
sending, by the particular front-end, the segments to the particular back-end.
8. The system of claim 7 wherein the request to perform the file system operation comprises an identification of a file, and the processor further carries out the steps of:
receiving, by the load balancer, a subsequent request involving the same file; and
assigning, by the load balancer, the subsequent request to the same particular front-end.
9. The system of claim 7 wherein the request to perform the file system operation comprises incoming data for a file, and the processor further carries out the steps of:
segmenting, by the particular front-end, the data into a plurality of segments corresponding to the lower-level of segments in the segment trees;
representing each of one or more subsets of the lower-level of segments by one or more respective higher-level segments, each of the one or more higher-level segments being associated with one or more respective similarity groups; and
sending each of the one or more higher-level segments, including the one or more subsets of the lower-level segments, to one or more back-ends according to the similarity group assignments.
10. The system of claim 7 wherein the processor further carries out the steps of:
receiving an indication from a container orchestration service that an initial number of the plurality of microservices has been scaled to a new number, different from the initial number; and
based on the new number of microservices, rebalancing one or more of the file assignments to the front-ends or the similarity group assignments to the back-ends.
11. The system of claim 7 wherein the processor further carries out the steps of:
receiving an indication from a container orchestration service that an initial number of the front-ends has been scaled to a new number, different from the initial number; and
replacing previous assignments of files with new assignments of files,
wherein when the scaling comprises increasing a number of the front-ends, the new assignments of files comprise new ranges of files for which additional front-ends and previous front-ends are now responsible, a number of files in a new range of files now being less than a number of files in a previous range of files, each previous front-end thereby now being responsible for a fewer number of files, and
wherein when the scaling comprises decreasing the number of front-ends, the new assignments of files comprise new ranges of files for which remaining front-ends are now responsible, a number of files in the new range of files now being greater than the number of files in the previous range of files, each remaining front-end thereby now being responsible for a greater number of files.
12. The system of claim 7 wherein the processor further carries out the steps of:
receiving an indication from a container orchestration service that an initial number of the back-ends has been scaled to a new number, different from the initial number; and
replacing previous assignments of similarity groups with new assignments of similarity groups,
wherein when the scaling comprises increasing a number of the back-ends, the new assignments of back-ends comprise new ranges of similarity groups for which additional back-ends and previous back-ends are now responsible, a number of similarity groups in a new range of similarity groups now being less than a number of similarity groups in a previous range of similarity groups, each previous back-end thereby now being responsible for a fewer number of similarity groups, and
wherein when the scaling comprises decreasing the number of back-ends, the new assignments of similarity groups comprise new ranges of similarity groups for which remaining back-ends are now responsible, a number of similarity groups in the new range of similarity groups now being greater than the number of similarity groups in the previous range of similarity groups, each remaining back-end thereby now being responsible for a greater number of similarity groups.
13. A computer program product, comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein, the computer-readable program code adapted to be executed by one or more processors to implement a method for a deduplicated file system as a plurality of microservices comprising front-ends and back-ends, the method comprising:
balancing assignments of files in the file system across the front-ends, the files being represented by segment trees;
balancing assignments of similarity groups across the back-ends, the similarity groups comprising segments at a lower-level of the segment trees that form the files, the front-ends being responsible for operations involving an upper-level of the segment trees, and the back-ends being responsible for operations involving the lower-level of the segment trees;
storing a mapping of file assignments to the front-ends, and of similarity group assignments to the back-ends;
receiving, by a load balancer, a request to perform a file system operation;
consulting, by the load balancer, the file assignments to identify a particular front-end that should be assigned the request;
consulting, by the particular front-end, the similarity group assignments to identify a particular back-end that should be responsible for processing segments associated with the request, the segments belonging to a similarity group that has been assigned to the particular back-end; and
sending, by the particular front-end, the segments to the particular back-end.
14. The computer program product of claim 13 wherein the request to perform the file system operation comprises an identification of a file, and the method further comprises:
receiving, by the load balancer, a subsequent request involving the same file; and
assigning, by the load balancer, the subsequent request to the same particular front-end.
15. The computer program product of claim 13 wherein the request to perform the file system operation comprises incoming data for a file, and the method further comprises:
segmenting, by the particular front-end, the data into a plurality of segments corresponding to the lower-level of segments in the segment trees;
representing each of one or more subsets of the lower-level of segments by one or more respective higher-level segments, each of the one or more higher-level segments being associated with one or more respective similarity groups; and
sending each of the one or more higher-level segments, including the one or more subsets of the lower-level segments, to one or more back-ends according to the similarity group assignments.
16. The computer program product of claim 13 wherein the method further comprises:
receiving an indication from a container orchestration service that an initial number of the plurality of microservices has been scaled to a new number, different from the initial number; and
based on the new number of microservices, rebalancing one or more of the file assignments to the front-ends or the similarity group assignments to the back-ends.
17. The computer program product of claim 13 wherein the method further comprises:
receiving an indication from a container orchestration service that an initial number of the front-ends has been scaled to a new number, different from the initial number; and
replacing previous assignments of files with new assignments of files,
wherein when the scaling comprises increasing a number of the front-ends, the new assignments of files comprise new ranges of files for which additional front-ends and previous front-ends are now responsible, a number of files in a new range of files now being less than a number of files in a previous range of files, each previous front-end thereby now being responsible for a fewer number of files, and
wherein when the scaling comprises decreasing the number of front-ends, the new assignments of files comprise new ranges of files for which remaining front-ends are now responsible, a number of files in the new range of files now being greater than the number of files in the previous range of files, each remaining front-end thereby now being responsible for a greater number of files.
18. The computer program product of claim 13 wherein the method further comprises:
receiving an indication from a container orchestration service that an initial number of the back-ends has been scaled to a new number, different from the initial number; and
replacing previous assignments of similarity groups with new assignments of similarity groups,
wherein when the scaling comprises increasing a number of the back-ends, the new assignments of back-ends comprise new ranges of similarity groups for which additional back-ends and previous back-ends are now responsible, a number of similarity groups in a new range of similarity groups now being less than a number of similarity groups in a previous range of similarity groups, each previous back-end thereby now being responsible for a fewer number of similarity groups, and
wherein when the scaling comprises decreasing the number of back-ends, the new assignments of similarity groups comprise new ranges of similarity groups for which remaining back-ends are now responsible, a number of similarity groups in the new range of similarity groups now being greater than the number of similarity groups in the previous range of similarity groups, each remaining back-end thereby now being responsible for a greater number of similarity groups.