Automated determination of error-causing network packets utilizing network packet replay
An apparatus comprises at least one processing device configured to obtain a network trace comprising network packets received at a given network adapter of an information technology (IT) asset, to provision a test bed for analyzing the obtained network trace, and to replay the network packets in the obtained network trace from a transmit host of the test bed to a given port of a receive host of the test bed associated with a network adapter having a network adapter configuration corresponding to the given network adapter of the IT asset. The processing device is further configured to parse port statistics of the given port of the receive host, and to determine, based at least in part on the parsed port statistics, whether respective ones of the packets comprise error-causing packets having at least a threshold likelihood of being sources of alarms raised on the IT asset.
1 . An apparatus comprising:
at least one processing device comprising a processor coupled to a memory;
the at least one processing device being configured:
to obtain a network trace comprising a plurality of network packets, the plurality of network packets having been received at a given network adapter of at least one information technology asset;
to provision a test bed different than the at least one information technology asset for analyzing the obtained network trace, the test bed comprising a transmit host and a receive host, the receive host of the test bed comprising two or more ports associated with two or more different network adapter configurations;
to replay at least a subset of the plurality of network packets in the obtained network trace from the transmit host of the test bed to a given one of the two or more ports of the receive host of the test bed, the given port of the receive host of the test bed being associated with a network adapter of the receive host having a given one of the two or more different network adapter configurations selected based at least in part on a network adapter configuration of the given network adapter of the at least one information technology asset;
to parse port statistics of the given port of the receive host, the port statistics comprising error counters for one or more designated error types; and
to determine, based at least in part on the parsed port statistics of the given port of the receive host, whether respective ones of the packets in the subset of the plurality of network packets comprise error-causing packets having at least a threshold likelihood of being sources of one or more alarms raised on the at least one information technology asset.
2 . The apparatus of claim 1 wherein the at least one information technology asset comprises hyper-converged infrastructure utilizing storage virtualization software to implement one or more virtual storage networks, and wherein the given network adapter comprises a physical network interface card used for processing storage traffic in the one or more virtual storage networks.
3 . The apparatus of claim 2 wherein the one or more alarms raised on the at least one information technology asset comprises alarms raised in the storage virtualization software in response to counters of the one or more designated error types for the given network adapter exceeding respective designated error count thresholds.
4 . The apparatus of claim 1 wherein the receive host of the test bed comprises a server with two or more network adapters having the two or more different network adapter configurations, and wherein replaying at least a subset of the plurality of network packets in the obtained network trace from the transmit host of the test bed to the given port of the receive host of the test bed comprises selecting the given port associated with one of the two or more network adapters of the server which matches the given network adapter of the at least one information technology asset.
5 . The apparatus of claim 4 wherein the server comprises a physical server and the two or more network adapters comprise two or more physical network interface cards.
6 . The apparatus of claim 4 wherein the server comprises a virtual server and the two or more network adapters comprise two or more virtual network interface cards.
7 . The apparatus of claim 4 wherein the server runs a bare metal hypervisor with one or more virtual machines configured to run an operating system matching that of the at least one information technology asset.
8 . The apparatus of claim 1 wherein the receive host of the test bed comprises a server with a network adapter configured to emulate the two or more different network adapter configurations, and wherein replaying at least a subset of the plurality of network packets in the obtained network trace from the transmit host of the test bed to the given port of the receive host of the test bed comprises configuring the network adapter of the server to emulate the network adapter configuration of the given network adapter of the at least one information technology asset.
9 . The apparatus of claim 1 wherein parsing the port statistics of the given port of the receive host comprises comparing first port statistics information of the given port prior to replay of each packet of the subset of the plurality of network packets in the obtained network trace with second port statistics information of the given port subsequent to replay of each packet of the subset of the plurality of network packets in the obtained network trace.
10 . The apparatus of claim 1 wherein the one or more designated error types comprise at least one of receive length errors, redundancy check errors, and packet drop errors.
11 . The apparatus of claim 1 wherein the network trace comprises a packet capture file with a plurality of frames corresponding to the plurality of network packets, and wherein determining whether respective ones of the packets in the subset of the plurality of network packets comprise error-causing packets comprises:
identifying frame identifiers for a subset of the plurality of frames corresponding to the error-causing packets; and
generating a filter data structure for filtering the packet capture file to generate another packet capture file including only the subset of the plurality of frames corresponding to the error-causing packets.
12 . The apparatus of claim 1 wherein the at least one processing device is further configured to determine whether a given one of the error-causing packets is benign based at least in part on inspecting contents of the given error-causing packet.
13 . The apparatus of claim 1 wherein the at least one processing device is further configured to determine whether a given one of the error-causing packets is benign based at least in part on which of the error counters the given error-causing packet increments.
14 . The apparatus of claim 1 wherein the at least one processing device is further configured, responsive to determining that a given one of the error-causing packets is not benign, to modify handling of the given error-causing packet by the at least one information technology asset.
15 . A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes the at least one processing device:
to obtain a network trace comprising a plurality of network packets, the plurality of network packets having been received at a given network adapter of at least one information technology asset;
to provision a test bed different than the at least one information technology asset for analyzing the obtained network trace, the test bed comprising a transmit host and a receive host, the receive host of the test bed comprising two or more ports associated with two or more different network adapter configurations;
to replay at least a subset of the plurality of network packets in the obtained network trace from the transmit host of the test bed to a given one of the two or more ports of the receive host of the test bed, the given port of the receive host of the test bed being associated with a network adapter of the receive host having a given one of the two or more different network adapter configurations selected based at least in part on a network adapter configuration of the given network adapter of the at least one information technology asset;
to parse port statistics of the given port of the receive host, the port statistics comprising error counters for one or more designated error types; and
to determine, based at least in part on the parsed port statistics of the given port of the receive host, whether respective ones of the packets in the subset of the plurality of network packets comprise error-causing packets having at least a threshold likelihood of being sources of one or more alarms raised on the at least one information technology asset.
16 . The computer program product of claim 15 wherein the at least one information technology asset comprises hyper-converged infrastructure utilizing storage virtualization software to implement one or more virtual storage networks, and wherein the given network adapter comprises a physical network interface card used for processing storage traffic in the one or more virtual storage networks.
17 . The computer program product of claim 16 wherein the one or more alarms raised on the at least one information technology asset comprises alarms raised in the storage virtualization software in response to counters of the one or more designated error types for the given network adapter exceeding respective designated error count thresholds.
18 . A method comprising:
obtaining a network trace comprising a plurality of network packets, the plurality of network packets having been received at a given network adapter of at least one information technology asset;
provisioning a test bed different than the at least one information technology asset for analyzing the obtained network trace, the test bed comprising a transmit host and a receive host, the receive host of the test bed comprising two or more ports associated with two or more different network adapter configurations;
replaying at least a subset of the plurality of network packets in the obtained network trace from the transmit host of the test bed to a given one of the two or more ports of the receive host of the test bed, the given port of the receive host of the test bed being associated with a network adapter of the receive host having a given one of the two or more different network adapter configurations selected based at least in part on a network adapter configuration of the given network adapter of the at least one information technology asset;
parsing port statistics of the given port of the receive host, the port statistics comprising error counters for one or more designated error types; and
determining, based at least in part on the parsed port statistics of the given port of the receive host, whether respective ones of the packets in the subset of the plurality of network packets comprise error-causing packets having at least a threshold likelihood of being sources of one or more alarms raised on the at least one information technology asset;
wherein the method is performed by at least one processing device comprising a processor coupled to a memory.
19 . The method of claim 18 wherein the at least one information technology asset comprises hyper-converged infrastructure utilizing storage virtualization software to implement one or more virtual storage networks, and wherein the given network adapter comprises a physical network interface card used for processing storage traffic in the one or more virtual storage networks.
20 . The method of claim 19 wherein the one or more alarms raised on the at least one information technology asset comprises alarms raised in the storage virtualization software in response to counters of the one or more designated error types for the given network adapter exceeding respective designated error count thresholds.