Methods and systems for dynamic recovery with adaptive latency injection
Embodiments of the disclosure provide systems and methods for reducing the capacity used to provide High Availability (HA) and Disaster Recovery (DR) in a distributed computing environment. According to one embodiment, dynamic recovery of a cloud-based resource can comprise setting a current latency value to an initial latency value and handling received requests with the current latency value. Current resource utilization can be detected while requests are being processed and a determination can be made as to whether the detected current resource utilization exceeds a predetermined threshold amount of resource utilization. In response to determining the detected current resource utilization does not exceed the threshold, the current latency amount can be maintained at the initial latency value. In response to determining the detected current resource utilization exceeds the threshold, the current latency value can be adjusted and injected into handling of received client requests.
1 . A method for dynamic recovery of a cloud-based resource, the method comprising:
setting, by a processor of a distributed cloud-based storage system, a current latency value to an initial latency value;
handling, by the processor of a distributed cloud-based storage system, received requests with the current latency value;
detected, by the processor of a distributed cloud-based storage system, current resource utilization while requests are being processed;
determining, by the processor of a distributed cloud-based storage system, whether the detected current resource utilization exceeds a predetermined threshold amount of resource utilization;
in response to determining the detected current resource utilization does not exceed the predetermined threshold amount of resource utilization, maintaining, by the processor of a distributed cloud-based storage system, the current latency amount at the initial latency value; and
in response to determining the detected current resource utilization exceeds the predetermined threshold amount of resource utilization, adjusting, by the processor of a distributed cloud-based storage system, the current latency value and injecting the adjusted current latency value into handling of received client requests, wherein injecting the adjusted current latency value into handling of received requests introduces a period of latency equal to the adjusted current latency value into the handling of the received requests.
2 . The method of claim 1 , wherein adjusting the current latency value comprises:
identifying a context for a received request;
determining a latency increment value and a maximum latency value for the identified context for the received request; and
incrementing the current latency value by the latency increment value for the identified request type for up to the maximum latency value for the identified context for the received request.
3 . The method of claim 2 , wherein identifying the context for the received request comprises determining whether the received request is from a client device with an interactive user or a client device with a non-interactive user.
4 . The method of claim 2 , wherein identifying the context for the received request comprises determining whether the received request is from a client device with a paid user or a client device with a non-paid user.
5 . The method of claim 4 , wherein identifying the context for the received request comprises, in response to determining the received request is from a client device with a paid user, determining a tier of a plurality of tiers for the paid user.
6 . The method of claim 2 , wherein identifying the context for the received request comprises determining a request type for the received request.
7 . The method of claim 2 , wherein identifying the context for the received request comprises determining a level of traffic for the distributed cloud-based storage system.
8 . A system comprising:
a processor; and
a memory coupled with and readable by the processor and storing therein a set of instructions which, when executed by the processor, causes the processor to:
set a current latency value to an initial latency value;
handle received requests with the current latency value;
detect current resource utilization while requests are being processed;
determine whether the detected current resource utilization exceeds a predetermined threshold amount of resource utilization;
in response to determining the detected current resource utilization does not exceed the predetermined threshold amount of resource utilization, maintain the current latency amount at the initial latency value; and
in response to determining the detected current resource utilization exceeds the predetermined threshold amount of resource utilization, adjust the current latency value and inject the adjusted current latency value into handling of received client requests, wherein injecting the adjusted current latency value into handling of received requests introduces a period of latency equal to the adjusted current latency value into the handling of the received requests.
9 . The system of claim 8 , wherein adjusting the current latency value comprises:
identifying a context for a received request;
determining a latency increment value and a maximum latency value for the identified context of the received request; and
incrementing the current latency value by the latency increment value for the identified context for the received request for up to the maximum latency value for the identified context for the received request.
10 . The system of claim 8 , wherein identifying the context for the received request comprises determining whether the received request is from a client device with an interactive user or a client device with a non-interactive user.
11 . The system of claim 8 , wherein identifying the context for the received request comprises determining whether the received request is from a client device with a paid user or a client device with a non-paid user.
12 . The system of claim 11 , wherein identifying the context for the received request comprises, in response to determining the received request is from a client device with a paid user, determining a tier of a plurality of tiers for the paid user.
13 . The system of claim 8 , wherein identifying the context for the received request comprises determining a request type for the received request.
14 . The system of claim 8 , wherein identifying the context for the received request comprises determining a level of traffic for the distributed cloud-based storage system.
15 . A non-transitory, computer-readable medium comprising a set of instructions stored therein which, when executed by a processor, causes the processor to:
set a current latency value to an initial latency value;
handle received requests with the current latency value;
detect current resource utilization while requests are being processed;
determine whether the detected current resource utilization exceeds a predetermined threshold amount of resource utilization;
in response to determining the detected current resource utilization does not exceed the predetermined threshold amount of resource utilization, maintain the current latency amount at the initial latency value; and
in response to determining the detected current resource utilization exceeds the predetermined threshold amount of resource utilization, adjust the current latency value and inject the adjusted current latency value into handling of received client requests, wherein injecting the adjusted current latency value into handling of received requests introduces a period of latency equal to the adjusted current latency value into the handling of the received requests.
16 . The non-transitory, computer-readable medium of claim 15 , wherein adjusting the current latency value comprises:
identifying a context for a received request;
determining a latency increment value and a maximum latency value for the identified context for the received request; and
incrementing the current latency value by the latency increment value for the identified context for the received request for up to the maximum latency value for the identified context for the received request.
17 . The non-transitory, computer-readable medium of claim 16 , wherein identifying the context for the received request comprises determining whether the received request is from a client device with an interactive user or a client device with a non-interactive user.
18 . The non-transitory, computer-readable medium of claim 16 , wherein identifying the context for the received request comprises determining whether the received request is from a client device with a paid user or a client device with a non-paid user.
19 . The non-transitory, computer-readable medium of claim 16 , wherein identifying the context for the received request comprises determining a request type for the received request.
20 . The non-transitory, computer-readable medium of claim 16 , wherein identifying the context for the received request comprises determining a level of traffic for the distributed cloud-based storage system.