Techniques for resource utilization in replication pipeline processing
Techniques are described for ensuring end-to-end fair-share resource utilization during cross-region replication. In certain embodiments, a fair-share architecture is used for communication among pipeline stages performing a cross-region replication between different cloud infrastructure regions. Cross-region replication-related jobs are distributed evenly from a pipeline stage into a temporary buffer in the fair-share architecture, and then further distributed evenly form the fair-share architecture to parallel running threads of next pipeline stage for execute. Techniques for static and dynamic resource allocations are also disclosed.
1 . A method, comprising:
executing a set of jobs, by a first pipeline stage and a second pipeline stage of a computing system, the first pipeline stage comprising a first set of parallel running threads, and the second pipeline stage comprising a second set of parallel running threads; and
communicating, by the computing system, between the first pipeline stage and the second pipeline stage through an architecture for enqueuing and dequeuing, the communicating comprising:
enqueuing the set of jobs from the first pipeline stage evenly to the architecture; and
dequeuing the set of jobs from the architecture evenly to the second pipeline stage, wherein executing the set of jobs by the first pipeline stage uses an equivalent amount of resources of the computing system as executing the set of jobs by the second pipeline stage.
2 . The method of claim 1 , wherein the architecture comprises a linked list of queues holding the set of jobs having completed execution by the first set of parallel running threads in the first pipeline stage, and to be executed by the second set of parallel running threads in the second pipeline stage.
3 . The method of claim 2 , wherein enqueuing the set of jobs evenly to the architecture comprises placing, by a processing thread of the first set of parallel running threads in the first pipeline stage, one of the set of jobs into one queue of the linked list of queues in a round-robin fashion.
4 . The method of claim 2 , wherein dequeuing the set of jobs from the architecture evenly comprises extracting one of the set of jobs from one queue of the linked list of queues in a round-robin fashion.
5 . The method of claim 2 , wherein a queue of the linked list of queues is created upon receiving one of the set of jobs.
6 . The method of claim 2 , wherein a queue of the linked list of queues is deleted when no job exists in the queue.
7 . The method of claim 1 , wherein the first pipeline stage and the second pipeline stage are replication pipeline stages of a file system.
8 . The method of claim 1 ,
wherein each job of the set of jobs is executed by one or more of the first set of parallel running threads in the first pipeline stage,
wherein each job of the set of jobs is executed by one or more of the second set of parallel running threads in the second pipeline stage, and
wherein each parallel running thread of the first set and the second set uses an equal share of resources of the computing system.
9 . The method of claim 1 , further comprising a job failure in the first pipeline stage does not impact any jobs being executed by the second pipeline stage.
10 . The method of claim 1 , wherein the resources of the computing system comprise computing devices, memory, storage, network, database, and bandwidth.
11 . A non-transitory computer-readable medium storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
executing a set of jobs, by a first pipeline stage and a second pipeline stage of a computing system, the first pipeline stage comprising a first set of parallel running threads, and the second pipeline stage comprising a second set of parallel running threads; and
communicating, by the computing system, between the first pipeline stage and the second pipeline stage through an architecture for enqueuing and dequeuing, the communicating comprising:
enqueuing the set of jobs from the first pipeline stage evenly to the architecture; and
dequeuing the set of jobs from the architecture evenly to the second pipeline stage, wherein executing the set of jobs by the first pipeline stage uses an equivalent amount of resources of the computing system as executing the set of jobs by the second pipeline stage.
12 . The non-transitory computer-readable medium of claim 11 , wherein the architecture comprises a linked list of queues holding the set of jobs having completed execution by the first set of parallel running threads in the first pipeline stage, and to be executed by the second set of parallel running threads in the second pipeline stage.
13 . The non-transitory computer-readable medium of claim 12 , wherein enqueuing the set of jobs evenly to the architecture comprises placing, by a processing thread of the first set of parallel running threads in the first pipeline stage, one of the set of jobs into one queue of the linked list of queues in a round-robin fashion, and wherein dequeuing the set of jobs from the architecture evenly comprises extracting one of the set of jobs from one queue of the linked list of queues in a round-robin fashion.
14 . The non-transitory computer-readable medium of claim 12 , wherein a queue of the linked list of queues is created upon receiving one of the set of jobs, and a queue of the linked list of queues is deleted when no job exists in the queue.
15 . The non-transitory computer-readable medium of claim 11 ,
wherein each job of the set of jobs is executed by one or more of the first set of parallel running threads in the first pipeline stage,
wherein each job of the set of jobs is executed by one or more of the second set of parallel running threads in the second pipeline stage, and
wherein each parallel running thread of the first set and the second set uses an equal share of resources of the computing system.
16 . A computing system, comprising:
one or more processors; and
one or more non-transitory computer readable media storing computer-executable instructions that, when executed by the one or more processors, cause the computing system to:
execute a set of jobs, by a first pipeline stage and a second pipeline stage, the first pipeline stage comprising a first set of parallel running threads, and the second pipeline stage comprising a second set of parallel running threads; and
communicate between the first pipeline stage and the second pipeline stage through an architecture for enqueuing and dequeuing, the communicating comprising:
enqueue the set of jobs from the first pipeline stage evenly to the architecture; and
dequeue the set of jobs from the architecture evenly to the second pipeline stage, wherein executing the set of jobs by the first pipeline stage uses an equivalent amount of resources of the computing system as executing the set of jobs by the second pipeline stage.
17 . The computing system of claim 16 , wherein the architecture comprises a linked list of queues holding the set of jobs having completed execution by the first set of parallel running threads in the first pipeline stage, and to be executed by the second set of parallel running threads in the second pipeline stage.
18 . The computing system of claim 17 , wherein enqueuing the set of jobs evenly to the architecture comprises placing, by a processing thread of the first set of parallel running threads in the first pipeline stage, one of the set of jobs into one queue of the linked list of queues in a round-robin fashion, and wherein dequeuing the set of jobs from the architecture evenly comprises extracting one of the set of jobs from one queue of the linked list of queues in a round-robin fashion.
19 . The computing system of claim 17 ,
wherein a queue of the linked list of queues is created upon receiving one of the set of jobs, and a queue of the linked list of queues is deleted when no job exists in the queue.
20 . The computing system of claim 16 ,
wherein each job of the set of jobs is executed by one or more of the first set of parallel running threads in the first pipeline stage,
wherein each job of the set of jobs is executed by one or more of the second set of parallel running threads in the second pipeline stage, and
wherein each parallel running thread of the first set and the second set uses an equal share of resources of the computing system.