Data driven chaos engineering based on service mesh and organizational chart
An organizational graph and a microservice graph can be generated. For two neighboring microservices, a combined organizational distance can be determined based on a distance for respective workers of each microservice in the organizational graph. The combined organizational distance can be used to determine a priority for a connection between the two microservices, and chaos testing can be performed on that connection based on its priority.
1. A method, comprising:
determining, by a system comprising a processor, that a first microservice and a second microservice exchange communications;
determining, by the system and from an organizational graph, an organizational distance between a first organizational node that corresponds to the first microservice and a second organizational node that corresponds to the second microservice, wherein a node of the organizational graph represents a person of an organization that produces a group of microservices that comprises the first microservice and the second microservice, and wherein an edge of the organizational graph represents an organizational relationship between two people of the organization, wherein the determining of the organizational distance between the first organizational node that corresponds to the first microservice node and the second organizational node that corresponds to the second microservice node is performed in response to identifying a first identifier associated with a first person who is authorized to manage the first microservice and a second identifier associated with a second person who is authorized to manage the second microservice;
selecting, by the system, a selected connection of a group of connections for the group of microservices based on respective weights of the group of connections that comprise the weight, wherein the selecting of the selected connection of the group of connections of the microservice graph based on the respective weights of the group of connections that comprise the weight is performed in response to determining a weight for manifesting a failure condition in a connection between the first microservice and the second microservice based on the organizational distance between the first organizational node and the second organizational node; and
introducing, by the system, chaos to the selected connection.
2. The method of claim 1 , wherein the weight for the first connection is based on an combined weight of respective organizational distances of people who are authorized to manage the first microservice and people who are authorized to manage the second microservice.
3. The method of claim 2 , wherein the combined weight is an average weight.
4. The method of claim 2 , wherein the combined weight is a summed weight.
5. The method of claim 2 , wherein the combined weight is a median weight.
6. The method of claim 1 , further comprising:
determining, by the system, that a second connection of the group of connections has been assigned a weight of zero; and
increasing the weight of the second connection to a value above zero.
7. The method of claim 1 , wherein the introducing of the chaos to the selected connection comprises:
causing, by the system, a failure of the selected connection.
8. A system, comprising:
a processor; and
a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, comprising:
generating an organizational graph, a node of the organizational graph representing a person of an organization that produces a group of microservices, and an edge of the organizational graph representing an organizational relationship between two people of the organization;
determining a microservice graph based on the group of microservices, a node of the microservice graph representing a microservice of the group of microservices, and an edge of the microservice graph representing a communication between two microservices of the group of microservices;
identifying, from the microservice graph, a first microservice node that corresponds to a first microservice and a second microservice node that corresponds to a second microservice, the first microservice node and the second microservice node being connected by a first edge;
determining, from the organizational graph, an organizational distance between a first organizational node that corresponds to the first microservice node and a second organizational node that corresponds to the second microservice node, wherein the determining of the organizational distance between the first organizational node that corresponds to the first microservice node and the second organizational node that corresponds to the second microservice node is performed in response to identifying a first identifier associated with a first person who is authorized to manage the first microservice and a second identifier associated with a second person who is authorized to manage the second microservice;
selecting a selected edge of a group of edges of the microservice graph based on respective priorities of the group of edges that comprise the priority, wherein the selecting of the selected edge of the group of edges of the microservice graph based on respective priorities of the group of edges that comprise the priority is performed in response to determining a priority for manifesting a failure condition in a connection between the first microservice and the second microservice based on the organizational distance between the first organizational node and the second organizational node; and
causing a connection among the group of microservices that corresponds to the selected edge to manifest a failure condition.
9. The system of claim 8 , wherein the organizational distance between the first organizational node and the second organizational node comprises a shortest undirected path between the first organizational node and the second organizational node in the organizational graph.
10. The system of claim 8 , wherein the priority for the first edge is based on an aggregated priority of respective organizational distances corresponding to pairs of people, a pair comprising one person who is authorized to manage the first microservice and another person who is authorized to manage the second microservice.
11. The system of claim 8 , wherein the priority for the first edge is normalized relative to the respective priorities of the group of edges, and wherein the selecting of the selected edge is based on normalized priorities of the group of edges.
12. The system of claim 8 , wherein the determining of the microservice graph is performed by a component that is separate from the group of microservices, wherein the group of microservices sends information about data transfer between microservices of the group of microservices to the component, and wherein the component identifies the first microservice node of the microservice graph as a source of the data transfer between the microservices, and identifies the second microservice node of the microservice graph as a destination of the data transfer between the microservices.
13. The system of claim 8 , wherein the determining of the microservice graph comprises:
analyzing the first microservice of the group of microservices to determine what other microservices of the group of microservices to which the microservice is configured to send data.
14. The system of claim 8 , wherein the generating of the organizational graph comprises:
receiving, by querying a database, chart data representative of an organizational chart of people identifiers representative of people in the organization that create the group of microservices, the chart data indicating a manager to whom the person reports.
15. The system of claim 8 , wherein the organizational distance between two nodes of the organizational graph comprises a shortest undirected path between the two nodes.
16. A non-transitory computer-readable medium comprising instructions that, in response to execution, cause a system comprising a processor to perform operations, comprising:
determining that a first microservice and a second microservice of microservices are associated with a producer-consumer relationship;
determining, from an organizational graph that represent people of an organization that produces the microservices that comprises the first microservice and the second microservice, an organizational distance between a first node that corresponds to the first microservice and a second organizational node that corresponds to the second microservice, wherein the determining of the organizational distance between the first organizational node that corresponds to the first microservice node and the second organizational node that corresponds to the second microservice node is performed in response to identifying a first identifier associated with a first person who is authorized to manage the first microservice and a second identifier associated with a second person who is authorized to manage the second microservice;
determining a weight for manifesting a failure condition in a first connection between the first microservice and the second microservice based on the organizational distance between the first organizational node and the second organizational node; and
introducing chaos to the first connection at a rate based on the weight.
17. The non-transitory computer-readable medium of claim 16 , wherein the operations further comprise:
selecting the first connection for the chaos based on the first microservice having a largest weight of connections of the microservices.
18. The non-transitory computer-readable medium of claim 16 , wherein the operations further comprise:
initiating a failure of the first connection; and
performing a chaos test on the microservices.
19. The non-transitory computer-readable medium of claim 16 , wherein the operations further comprise:
in response to determining that less than a threshold amount of time has elapsed, selecting a second connection of connections of the microservices for inducing the chaos based on respective weights to the connections.
20. The non-transitory computer-readable medium of claim 16 , wherein the weight of the first connection indicates a probability that the first connection will be selected among connections of the microservices for inducing the chaos.