Techniques for resource utilization in replication pipeline processing
Techniques are described for ensuring end-to-end fair-share resource utilization during cross-region replication. In certain embodiments, a fair-share architecture is used for communication among pipeline stages performing a cross-region replication between different cloud infrastructure regions. Cross-region replication-related jobs are distributed evenly from a pipeline stage into a temporary buffer in the fair-share architecture, and then further distributed evenly form the fair-share architecture to parallel running threads of next pipeline stage for execute. Techniques for static and dynamic resource allocations are also disclosed.
1. A method, comprising:
receiving, by a computing system, a request for a cross-region replication of a file system;
processing data, by a first pipeline stage of the computing system and a second pipeline stage of the computing system, each pipeline stage of the first pipeline stage and the second pipeline stage comprising a thread pool with a plurality of parallel running threads; and
communicating, by the computing system, between the first pipeline stage and the second pipeline stage through an architecture for enqueuing and dequeuing replication-related jobs for execution, each replication-related job for execution being configured to use an equal amount of resources of the file system, the architecture comprising:
distributing, by the computing system, the replication-related jobs evenly into a temporary buffer;
distributing, by the computing system, the replication-related jobs substantially evenly from the temporary buffer to the plurality of parallel running threads for execution; and
executing each of the replication-related jobs, by one of the plurality of parallel running threads, using an equal amount of resources of the file system.
2. The method of claim 1 , further comprising adding the architecture between the second pipeline stage and a third pipeline stage for scaling the file system while maintaining fair-share utilization of resources.
3. The method of claim 1 , wherein the temporary buffer comprises a linked list of queues holding the replication-related jobs for the cross-region replication to be executed by the plurality of parallel running threads.
4. The method of claim 3 , wherein a queue of the linked list of queues is dynamically created upon receiving one of the replication-related jobs.
5. The method of claim 3 , wherein a queue of the linked list of queues is dynamically deleted when no replication-related job exists in the queue.
6. The method of claim 3 , further comprising placing, by a processing thread of the first pipeline stage, one of the replication-related jobs into one queue of the linked list of queues in a round-robin fashion, the processing thread being one of the plurality of parallel running threads of the first pipeline stage.
7. The method of claim 3 , further comprising extracting one of the replication-related jobs from one queue of the linked list of queues into a first-in-first-out queue in a round-robin fashion.
8. The method of claim 7 , further comprising extracting, by a processing thread of the second pipeline stage, one of the replication-related jobs from the first-in-first-out queue for execution by the processing thread of the second pipeline stage, the processing thread being one of the plurality of parallel running threads of the second pipeline stage.
9. A non-transitory computer-readable medium storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving, by a computing system, a request for a cross-region replication of a file system;
processing data, by a first pipeline stage of the computing system and a second pipeline stage of the computing system, each pipeline stage of the first pipeline stage and the second pipeline stage comprising a thread pool with a plurality of parallel running threads; and
communicating, by the computing system, between the first pipeline stage and the second pipeline stage through an architecture for enqueuing and dequeuing replication-related jobs for execution, each replication-related job for execution being configured to use an equal amount of resources of the file system, the architecture comprising:
distributing, by the computing system, the replication-related jobs evenly into a temporary buffer;
distributing, by the computing system, the replication-related jobs evenly from the temporary buffer to the plurality of parallel running threads for execution; and
executing each of the replication-related jobs, by one of the plurality of parallel running threads, using an equal amount of resources of the file system.
10. The non-transitory computer-readable medium of claim 9 , wherein the temporary buffer comprises a linked list of queues holding the replication-related jobs for the cross-region replication to be executed by the plurality of parallel running threads.
11. The non-transitory computer-readable medium of claim 10 , wherein a queue of the linked list of queues is dynamically created upon receiving one of the replication-related jobs.
12. The non-transitory computer-readable medium of claim 10 , wherein a queue of the linked list of queues is dynamically deleted when no replication-related job exists in the queue.
13. The non-transitory computer-readable medium of claim 10 , the operations further comprising placing, by a processing thread of the first pipeline stage, one of the replication-related jobs into one queue of the linked list of queues in a round-robin fashion, the processing thread being one of the plurality of parallel running threads of the first pipeline stage.
14. The non-transitory computer-readable medium of claim 10 , the operations further comprising extracting, one of the replication-related jobs from one queue of the linked list of queues into a first-in-first-out queue in a round-robin fashion.
15. The non-transitory computer-readable medium of claim 14 , the operations further comprising extracting, by a processing thread of the second pipeline stage, one of the replication-related jobs from the first-in-first-out queue for execution by the processing thread of the second pipeline stage, the processing thread being one of the plurality of parallel running threads of the second pipeline stage.
16. A system, comprising:
one or more processors; and
one or more non-transitory computer readable media storing computer-executable instructions that, when executed by the one or more processors, cause the system to:
receive, by a computing system, a request for a cross-region replication of a file system;
process data by a first pipeline stage of the computing system and a second pipeline stage of the computing system, each pipeline stage of the first pipeline stage and the second pipeline stage comprising a thread pool with a plurality of parallel running threads; and
communicate, by the computing system, between the first pipeline stage and the second pipeline stage through an architecture for enqueuing and dequeuing replication-related jobs for execution, each replication-related job for execution being configured to use an equal amount of resources of the file system, the architecture comprising:
distribute, by the computing system, the replication-related jobs evenly into a temporary buffer;
distribute, by the computing system, the replication-related jobs evenly from the temporary buffer to the plurality of parallel running threads for execution; and
execute each of the replication-related jobs, by one of the plurality of parallel running threads, using an equal amount of resources of the file system.
17. The system of claim 16 , wherein the temporary buffer comprises a linked list of queues holding the replication-related jobs for the cross-region replication to be executed by the plurality of parallel running threads.
18. The system of claim 17 , wherein a queue of the linked list of queues is dynamically created upon receiving one of the replication-related jobs, and is dynamically deleted when no replication-related job exists in the queue.
19. The system of claim 17 , wherein the system is further caused to place, by a processing thread of the first pipeline stage, one of the replication-related jobs into one queue of the linked list of queues in a round-robin fashion, the processing thread being one of the plurality of parallel running threads of the first pipeline stage.
20. The system of claim 17 , wherein the system is further caused to
extract one of the replication-related jobs from one queue of the linked list of queues into a first-in-first-out queue in a round-robin fashion; and
extract, by a processing thread of the second pipeline stage, one of the replication-related jobs from the first-in-first-out queue for execution by the processing thread of the second pipeline stage, the processing thread being one of the plurality of parallel running threads of the second pipeline stage.