Method and system for processing data in a replication system
In general, embodiments of the invention relate to a method, non-transitory computer readable medium, and system for managing replication system processing. More specifically, embodiments of the invention relate to managing the dedicated set of computational resources in a replication system during runtime in order to, among other things, improve the performance of the replication system.
1. A method for managing replication system processing, the method comprising:
allocating a first set of cores to a replication application in a replication system;
obtaining, by the replication system, data from a client device for processing by the replication application;
processing the data by the replication application using the first set of cores;
determining, based on the processing, a processing rate of the data by the first set of cores;
making a determination that in the processing rate is greater than a breakpoint associated with the first set of cores, wherein the breakpoint corresponds to a modeled processing rate of the first set of cores, wherein processing the data obtained from client devices at a rate above the breakpoint results in an exponential drop in output of the replication application compared to incoming data rate; and
based on the determination, updating the first set of cores to a second set of cores, wherein the second set of cores is allocated to the replication application.
2. The method of claim 1 , further comprising:
prior to allocating the first set of cores:
selecting the first set of cores to evaluate;
evaluating, by the replication system, the first set of cores to obtain performance data; and
determining the breakpoint associated with the first set of cores using, at least in part, the performance data.
3. The method of claim 2 , wherein the performance data is stored in a core profile in the replication system, wherein the core profile specifies a cardinality of the first set of cores.
4. The method of claim 3 , wherein a core profiler, executing on the replication system, obtains the performance data and determines the breakpoint using, at least in part, the performance data, wherein the core profiler generates the core profile.
5. The method of claim 1 , wherein determining the processing rate comprises logging a plurality of processing rates for the first set of cores over a period of time, wherein the processing rate is one of the plurality of processing rates.
6. The method of claim 1 , wherein processing the data comprises:
creating a copy of the data from the client device; and
transmitting copy of the data from the replication system to a second replication system.
7. The method of claim 1 , wherein a cardinality of the first set of cores is less than a cardinality of cores on the replication system.
8. The method of claim 7 , wherein a cardinality of the first set of cores is less than a cardinality of the second set of cores.
9. The method of claim 1 , further comprising:
obtaining, by the replication system, second data from the client device for processing by the replication application;
processing the second data by the replication application using the second set of cores;
determining, based on the processing of the second data, a second processing rate of the data by the second set of cores;
making a second determination that the second processing rate is less than the breakpoint associated with the second set of cores; and
based on the second determination, not modifying a cardinality of the second set of cores.
10. A non-transitory computer readable medium (CRM) storing instructions for managing replication system processing, the instructions comprising functionality for:
allocating a first set of cores to a replication application in a replication system;
obtaining, by the replication system, data from a client device for processing by the replication application;
processing the data by the replication application using the first set of cores;
determining, based on the processing, a processing rate of the data by the first set of cores;
making a determination that in the processing rate is greater than a breakpoint associated with the first set of core, wherein the breakpoint corresponds to a modeled processing rate of the first set of cores, wherein processing the data obtained from client devices at a rate above the breakpoint results in an exponential drop in output of the replication application compared to incoming data rate; and
based on the determination, updating the first set of cores to a second set of cores, wherein the second set of cores is allocated to the replication application.
11. The CRM of claim 10 , further comprising functionality for:
prior to allocating the first set of cores:
selecting the first set of cores to evaluate;
evaluating, by the replication system, the first set of cores to obtain performance data; and
determining the breakpoint associated with the first set of cores using, at least in part, the performance data.
12. The CRM of claim 11 , wherein the performance data is stored in a core profile in the replication system, wherein the core profile specifies a cardinality of the first set of cores.
13. The CRM of claim 12 , wherein a core profiler, executing on the replication system, obtains the performance data and determines the breakpoint using, at least in part, the performance data, wherein the core profiler generates the core profile.
14. The CRM of claim 10 , wherein determining the processing rate comprises logging a plurality of processing rates for the first set of cores over a period of time, wherein the processing rate is one of the plurality of processing rates.
15. The CRM of claim 10 , wherein processing the data comprises:
creating a copy of the data from the client device; and
transmitting copy of the data from the replication system to a second replication system.
16. The CRM of claim 10 , wherein a cardinality of the first set of cores is less than a cardinality of cores on the replication system.
17. The CRM of claim 16 , wherein a cardinality of the first set of cores is less than a cardinality of the second set of cores.
18. A replication system, comprising:
a replication application and
a processor,
wherein the replication application comprises threads for replicating input data, which when executed by the processor enables the processor to perform a method for managing replication system processing, the method comprising:
allocating a first set of cores to the replication application in a replication system;
obtaining, by the replication system, data from a client device for processing by the replication application;
processing the data by the replication application using the first set of cores;
determining, based on the processing, a processing rate of the data by the first set of cores;
making a determination that in the processing rate is greater than a breakpoint associated with the first set of cores, wherein the breakpoint corresponds to a modeled processing rate of the first set of cores, wherein processing the input data obtained from client devices at a rate above the breakpoint results in an exponential drop in output of the replication application compared to incoming data rate; and
based on the determination, updating the first set of cores to a second set of cores, wherein the second set of cores is allocated to the replication application.