Application chaos injection for improved system resilience
Systems and methods are described for injecting chaos into an application based on a determination that network traffic will be at the application when the chaos is injected. In an example, a gateway can receive requests for applications in a system. The gateway can be configured to periodically insert a header into an application request that causes a proxy agent associated with the requested application to request chaos instructions from a server. When chaos instructions are requested, the server can determine whether and what kind of chaos to inject into the application. The server can send instructions to the proxy agent, which can inject chaos according to the instructions. A monitoring agent can then monitor traffic around the application to determine whether other applications and system components behave appropriately in response to the injected chaos.
1 . A method for injecting chaos into an application, comprising:
receiving, at a gateway device from a user device, an access request for a first application;
inserting, by the gateway device and according to chaos rules associated with the first application, a header into the access request indicating whether a proxy agent for the first application is to request chaos instructions;
forwarding, by the gateway device, the access request including the header to the proxy agent associated with the first application;
at the proxy agent, responsive to determining from the header that the proxy agent is to request the chaos instructions, sending, to a server, a request requesting instructions from the server regarding whether to inject chaos into the first application, wherein the server reads configuration data associating the first application with a chaos type and a chance value representing a probability that chaos is to be injected, and wherein the server determines, based on the chance value, whether to provide the instructions indicating injection of chaos for the access request;
at the proxy agent, receiving instructions from the server to inject chaos of the chaos type into the first application; and
at the proxy agent and based on the instructions, injecting the chaos of the chaos type into the first application.
2 . The method of claim 1 , wherein injecting chaos into the first application includes at least one of terminating the first application, generating latency at the first application, and increasing utilization of a computing resource for the first application.
3 . The method of claim 1 , further comprising:
monitoring a second application to determine whether the second application responds as expected to the chaos injected at the first application; and
in an instance where the second application does not respond as expected, generating a notification.
4 . The method of claim 3 , wherein the second application is monitored based on a determination that the second application will attempt to contact the first application in response to the access request being granted.
5 . The method of claim 3 , wherein monitoring the second application includes determining whether the second application reattempts to communicate with the first application after a predetermined amount of time.
6 . The method of claim 1 , further comprising:
monitoring a Kubernetes pod that hosts the first application to determine whether the Kubernetes pod attempts to restart the first application; and
in an instance where the Kubernetes pod does not attempt to restart the first application, generating a notification.
7 . A non-transitory, computer-readable medium containing instructions that, when executed by a hardware-based processor, causes the processor to perform stages for injecting chaos into an application, the stages comprising:
receiving, at a gateway device from a user device, an access request for a first application;
inserting, by the gateway device and according to chaos rules associated with the first application, a header into the access request indicating whether a proxy agent for the first application is to request chaos instructions;
forwarding, by the gateway device, the access request including the header to the proxy agent associated with the first application;
at the proxy agent, responsive to determining from the header that the proxy agent is to request the chaos instructions, sending, to a server, a request requesting instructions from the server regarding whether to inject chaos into the first application, wherein the server reads configuration data associating the first application with a chaos type and a chance value representing a probability that chaos is to be injected, and wherein the server determines, based on the chance value, whether to provide the instructions indicating injection of chaos for the request;
at the proxy agent, receiving the instructions from the server to inject the chaos of the chaos type into the first application; and
at the proxy agent, based on the instructions, injecting the chaos of the chaos type into the first application.
8 . The non-transitory, computer-readable medium of claim 7 , wherein injecting chaos into the first application includes at least one of terminating the first application, generating latency at the first application, and increasing utilization of a computing resource for the first application.
9 . The non-transitory, computer-readable medium of claim 7 , the stages further comprising:
monitoring a second application to determine whether the second application responds as expected to the chaos injected at the first application; and
in an instance where the second application does not respond as expected, generating a notification.
10 . The non-transitory, computer-readable medium of claim 9 , wherein the second application is monitored based on a determination that the second application will attempt to contact the first application in response to the access request being granted.
11 . The non-transitory, computer-readable medium of claim 9 , wherein monitoring the second application includes determining whether the second application reattempts to communicate with the first application after a predetermined amount of time.
12 . The non-transitory, computer-readable medium of claim 7 , the stages further comprising:
monitoring a Kubernetes pod that hosts the first application to determine whether the Kubernetes pod attempts to restart the first application; and
in an instance where the Kubernetes pod does not attempt to restart the first application, generating a notification.
13 . A system for injecting chaos into an application, comprising:
a memory storage including a non-transitory, computer-readable medium comprising instructions; and
a hardware-based processor that executes the instructions to carry out stages comprising:
receiving, at a gateway device from a user device, an access request for a first application;
inserting, by the gateway device and according to chaos rules associated with the first application, a header into the access request indicating whether a proxy agent for the first application is to request chaos instructions;
forwarding, by the gateway device, the access request including the header to the proxy agent associated with the first application;
at the proxy agent, responsive to determining from the header that the proxy agent is to request the chaos instructions, sending, to a server, a request requesting instructions from the server regarding whether to inject chaos into the first application, wherein the server reads configuration data associating the first application with a chaos type and a chance value representing a probability that chaos is to be injected, and wherein the server determines, based on the chance value, whether to provide the instructions indicating injection of chaos for the request;
at the proxy agent, receiving the instructions from the server to inject the chaos of the chaos type into the first application; and
at the proxy agent, based on the instructions, injecting the chaos of the chaos type into the first application.
14 . The system of claim 13 , wherein injecting chaos into the first application includes at least one of terminating the first application, generating latency at the first application, and increasing utilization of a computing resource for the first application.
15 . The system of claim 13 , the stages further comprising:
monitoring a second application to determine whether the second application responds as expected to the chaos injected at the first application; and
in an instance where the second application does not respond as expected, generating a notification.
16 . The system of claim 15 , wherein the second application is monitored based on a determination that the second application will attempt to contact the first application in response to the access request being granted.
17 . The system of claim 15 , wherein monitoring the second application includes determining whether the second application reattempts to communicate with the first application after a predetermined amount of time.