Systems and methods for testing distributed systems using injected network partitions

1 . A method for testing a distributed system, the method comprising:

monitoring communication between a plurality of computing devices in a distributed system using a test operation;

identifying, based on the monitoring, each communication link that exists between two respective computing devices in the distributed system;

generating a communications list comprising a plurality of computing device pairs, wherein each pair comprises two of the plurality of computing devices directly connected by a respective communication link;

injecting a network partition in at least one pair of the plurality of computing device pairs, wherein injecting the network partition comprises utilizing software-defined networking (SDN) rules to cut communication between computing devices of the at least one pair;

detecting whether a performance degradation greater than a threshold performance occurs in response to the network partition;

in response to detecting the performance degradation greater than the threshold performance, generating a security report indicative of the performance degradation and the at least one pair of the plurality of computing device pairs causing the performance degradation; and

transmitting the security report to a device of a user associated with the distributed system.

2 . The method of claim 1 , further comprising:

in response to not detecting the performance degradation greater than the threshold performance, reinstating a communication link between the at least one pair of the plurality of computing device pairs; and

injecting the network partition in at least one different pair of the plurality of computing device pairs.

3 . The method of claim 2 , further comprising iterating through all pairs in the communications list by injecting the network partition in each pair and combination of pairs while assessing for performance degradation after each injection.

4 . The method of claim 1 , wherein the test operation comprises one or more of: reading or writing data from a storage device of the distributed system, producing and processing messages in a message queuing system of the distributed system, and reading or writing to a map in a distributed data structure of the distributed system.

5 . The method of claim 1 , wherein detecting whether the performance degradation greater than the threshold performance occurs comprises:

calculating a respective performance value for each computing device of the plurality of computing devices during operation of a job when no network partition is injected;

calculating a system-wide performance value of the distributed system using each calculated respective performance value;

calculating another respective performance value for each computing device of the plurality of computing devices during operation of the job when the network partition is injected;

calculating another system-wide performance value of the distributed system using each calculated respective performance value for when the network partition is injected; and

calculating a difference between the system-wide performance value and the another system-wide performance value.

6 . The method of claim 5 , wherein calculating the respective performance value for each computing device comprises:

retrieving one or more logs that include information about each computing device during the job, wherein the information comprises hardware performance information, network performance information, and job performance information; and

generating the respective performance value based on the information in the one or more logs.

7 . The method of claim 1 , wherein injecting the network partition comprises changing iptables configuration of the at least one pair to cut communication between computing devices of the at least one pair.

8 . The method of claim 1 , further comprising monitoring, subsequent to injecting the network partition, for attributes indicative of critical issues, wherein the attributes comprises one or more of: a job failure, a device failure, data loss, dropped packets, freezes.

9 . A system for testing a distributed system, comprising:

at least one memory; and

at least one hardware processor coupled with the at least one memory and configured, individually or in combination, to:

monitor communication between a plurality of computing devices in a distributed system using a test operation;

identify, based on the monitoring, each communication link that exists between two respective computing devices in the distributed system;

generate a communications list comprising a plurality of computing device pairs, wherein each pair comprises two of the plurality of computing devices directly connected by a respective communication link;

inject a network partition in at least one pair of the plurality of computing device pairs, wherein injecting the network partition comprises utilizing software-defined networking (SDN) rules to cut communication between computing devices of the at least one pair;

detect whether a performance degradation greater than a threshold performance occurs in response to the network partition;

in response to detecting the performance degradation greater than the threshold performance, generate a security report indicative of the performance degradation and the at least one pair of the plurality of computing device pairs causing the performance degradation; and

transmit the security report to a device of a user associated with the distributed system.

10 . The system of claim 9 , wherein the at least one hardware processor is configured to:

in response to not detecting the performance degradation greater than the threshold performance, reinstate a communication link between the at least one pair of the plurality of computing device pairs; and

inject the network partition in at least one different pair of the plurality of computing device pairs.

11 . The system of claim 10 , wherein the at least one hardware processor is configured to iterate through all pairs in the communications list by injecting the network partition in each pair and combination of pairs while assessing for performance degradation after each injection.

12 . The system of claim 9 , wherein the test operation comprises one or more of: reading or writing data from a storage device of the distributed system, producing and processing messages in a message queuing system of the distributed system, and reading or writing to a map in a distributed data structure of the distributed system.

13 . The system of claim 9 , wherein the at least one hardware processor is configured to detect whether the performance degradation greater than the threshold performance by:

calculating a respective performance value for each computing device of the plurality of computing devices during operation of a job when no network partition is injected;

calculating a system-wide performance value of the distributed system using each calculated respective performance value;

calculating another respective performance value for each computing device of the plurality of computing devices during operation of the job when the network partition is injected;

calculating another system-wide performance value of the distributed system using each calculated respective performance value for when the network partition is injected; and

calculating a difference between the system-wide performance value and the another system-wide performance value.

14 . The system of claim 13 , wherein the at least one hardware processor is configured to calculating the respective performance value for each computing device by:

retrieving one or more logs that include information about each computing device during the job, wherein the information comprises hardware performance information, network performance information, and job performance information; and

generating the respective performance value based on the information in the one or more logs.

15 . The system of claim 9 , wherein the at least one hardware processor is configured to inject the network partition by changing iptables configuration of the at least one pair to cut communication between computing devices of the at least one pair.

16 . The system of claim 9 , wherein the at least one hardware processor is configured to monitor, subsequent to injecting the network partition, for attributes indicative of critical issues, wherein the attributes comprises one or more of: a job failure, a device failure, data loss, dropped packets, freezes.

17 . A non-transitory computer readable medium storing thereon computer executable instructions for testing a distributed system, including instructions for:

monitoring communication between a plurality of computing devices in a distributed system using a test operation;

identifying, based on the monitoring, each communication link that exists between two respective computing devices in the distributed system;

generating a communications list comprising a plurality of computing device pairs, wherein each pair comprises two of the plurality of computing devices directly connected by a respective communication link;

injecting a network partition in at least one pair of the plurality of computing device pairs, wherein injecting the network partition comprises utilizing software-defined networking (SDN) rules to cut communication between computing devices of the at least one pair;

detecting whether a performance degradation greater than a threshold performance occurs in response to the network partition;

in response to detecting the performance degradation greater than the threshold performance, generating a security report indicative of the performance degradation and the at least one pair of the plurality of computing device pairs causing the performance degradation; and

transmitting the security report to a device of a user associated with the distributed system.