Creating an intelligent testing queue for improved quality assurance testing of microservices
Described is a system for creating an intelligent testing queue for improved quality assurance (QA) testing of services (or microservices). The system may perform a graphical analysis of interactions between services to derive testing constraints. For example, the system may monitor services to identify interactions (e.g. API accesses) between the services, and store the interactions as a directed graph representation. The system may traverse the directed graph (e.g. via a breadth-first search) to determine service dependencies. Based on the probability of failure for the testing operations and the service dependencies, the system create a specialized testing queue. By performing testing operations according to the specialized queue, the system may improve certain metrics associated with QA processes such as mean time to failure (MTTF) and mean time to diagnose (MTTD).
1. A system comprising:
one or more processors; and
a non-transitory computer readable medium storing a plurality of instructions, which when executed, cause the one or more processors to:
monitor a set of services executing within a production computing environment to identify interactions between the set of services;
store, within a data structure, a directed graph representation of the identified interactions of a subset of the set of services;
create, from the data structure and for each service of the subset, a corresponding dependency list indicating other services of the subset that are dependent upon the service;
identify, for each service of the subset, a testing operation associated with the service;
create a testing queue associated with the subset of services, the testing queue including each of the testing operations identified;
determine, for each of the testing operations, a corresponding failure rate associated with performing the testing operation;
perform a first ordering of the testing queue based on the corresponding failure rates of the testing operations of the testing queue, the first ordering includes ordering the testing operations from a highest corresponding failure rate to a lowest corresponding failure rate; and
perform a second ordering of the testing queue based on the corresponding dependency lists of the services associated with the testing operations of the testing queue, the second ordering includes moving the testing operations associated with a particular service ahead of the testing operations associated with the other services that are dependent upon the particular service.
2. The system of claim 1 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
perform, within a testing environment, a test associated with the subset of services by executing one or more of the testing operations of the testing queue according to the second ordering.
3. The system of claim 1 , wherein determining the corresponding failure rate associated with performing the testing operation includes:
executing the testing operation a predetermined number of times within a testing environment; and
determining the corresponding failure rate based on a number of test results that indicate a failure of performing the testing operation.
4. The system of claim 1 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
filter the identified interactions of the subset of services graph to remove cycles within the graph representation to create a directed acyclic graph (DAG) as the graph representation.
5. The system of claim 1 , wherein monitoring the set of services executing within the production computing environment includes:
receiving, from an interface of a service mesh component associated with the set of services, the identified interactions of the subset of the set of services.
6. The system of claim 1 , wherein the identified interactions of the subset of the set of services includes one service of the subset accessing an application programming interface (API) of another service of the subset.
7. The system of claim 1 , wherein other services of the subset that are dependent upon the service include those services that are connected to the service as part of an API access chain.
8. The system of claim 1 , wherein creating, from the data structure and for each service of the subset, the corresponding dependency list includes traversing the directed graph by performing a breadth-first search.
9. A method of creating a testing queue for testing operations associated with services, comprising:
monitoring a set of services executing within a production computing environment to identify interactions between the set of services;
storing, within a data structure, a directed graph representation of the identified interactions of a subset of the set of services;
creating, from the data structure and for each service of the subset, a corresponding dependency list indicating other services of the subset that are dependent upon the service;
identifying, for each service of the subset, a testing operation associated with the service;
creating the testing queue associated with the subset of services, the testing queue including each of the testing operations identified;
determining, for each of the testing operations, a corresponding failure rate associated with performing the testing operation;
performing a first ordering of the testing queue based on the corresponding failure rates of the testing operations of the testing queue, the first ordering includes ordering the testing operations from a highest corresponding failure rate to a lowest corresponding failure rate; and
performing a second ordering of the testing queue based on the corresponding dependency lists of the services associated with the testing operations of the testing queue, the second ordering includes moving the testing operations associated with a particular service ahead of the testing operations associated with the other services that are dependent upon the particular service.
10. The method of claim 9 , further comprising:
performing, within a testing environment, a test associated with the subset of services by executing one or more of the testing operations of the testing queue according to the second ordering.
11. The method of claim 9 , wherein determining the corresponding failure rate associated with performing the testing operation includes:
executing the testing operation a predetermined number of times within a testing environment; and
determining the corresponding failure rate based on a number of test results that indicate a failure of performing the testing operation.
12. The method of claim 9 , further comprising:
filtering the identified interactions of the subset of services graph to remove cycles within the graph representation to create a directed acyclic graph (DAG) as the graph representation.
13. The method of claim 9 , wherein monitoring the set of services executing within the production computing environment includes:
receiving, from an interface of a service mesh component associated with the set of services, the identified interactions of the subset of the set of services.
14. The method of claim 9 , wherein the identified interactions of the subset of the set of services includes one service of the subset accessing an application programming interface (API) of another service of the subset, and wherein other services of the subset that are dependent upon the service include those services that are connected to the service as part of an API access chain.
15. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, the program code including instructions to:
monitor a set of services executing within a production computing environment to identify interactions between the set of services;
store, within a data structure, a directed graph representation of the identified interactions of a subset of the set of services;
create, from the data structure and for each service of the subset, a corresponding dependency list indicating other services of the subset that are dependent upon the service;
identify, for each service of the subset, a testing operation associated with the service;
create a testing queue associated with the subset of services, the testing queue including each of the testing operations identified;
determine, for each of the testing operations, a corresponding failure rate associated with performing the testing operation;
perform a first ordering of the testing queue based on the corresponding failure rates of the testing operations of the testing queue, the first ordering includes ordering the testing operations from a highest corresponding failure rate to a lowest corresponding failure rate; and
perform a second ordering of the testing queue based on the corresponding dependency lists of the services associated with the testing operations of the testing queue, the second ordering includes moving the testing operations associated with a particular service ahead of the testing operations associated with the other services that are dependent upon the particular service.
16. The computer program product of claim 15 , wherein the program code includes further instructions to:
perform, within a testing environment, a test associated with the subset of services by executing one or more of the testing operations of the testing queue according to the second ordering.
17. The computer program product of claim 15 , wherein determining the corresponding failure rate associated with performing the testing operation includes:
executing the testing operation a predetermined number of times within a testing environment; and
determining the corresponding failure rate based on a number of test results that indicate a failure of performing the testing operation.
18. The computer program product of claim 15 , wherein the program code includes further instructions to:
filter the identified interactions of the subset of services graph to remove cycles within the graph representation to create a directed acyclic graph (DAG) as the graph representation.
19. The computer program product of claim 15 , wherein monitoring the set of services executing within the production computing environment includes:
receiving, from an interface of a service mesh component associated with the set of services, the identified interactions of the subset of the set of services.
20. The computer program product of claim 15 , wherein the identified interactions of the subset of the set of services includes one service of the subset accessing an application programming interface (API) of another service of the subset, and wherein other services of the subset that are dependent upon the service include those services that are connected to the service as part of an API access chain.