Network failover handling in modular switched fabric based data storage systems
Systems, methods, apparatuses, and software for data storage systems are provided herein. In one example, a data storage system is provided that includes a first processor configured to establish a network connection with an external system, and receive first storage operations transferred by the external system over the network connection, the first storage operations related to storage and retrieval of data on at least one storage drive. The first processor is configured to transfer information describing the network connection for delivery to at least a second processor. The second processor is configured to identify when the first processor has failed, responsively establish the network connection with the external system based at least on the information describing the network connection, and receive second storage operations transferred by the external system over the network connection.
1. A data storage system, comprising:
a plurality of storage drives each configured to store data and retrieve the data stored on associated storage media responsive to storage operations;
a first processor configured to establish a network connection with an external system, receive first storage operations transferred by the external system over the network connection, and handle the first storage operations related to storage and retrieval of data on at least one of the plurality of storage drives assigned to a first address space portion of a Peripheral Component Interconnect Express (PCIe) address space shared among the first processor and at least a second processor;
after establishing the network connection with the external system, the first processor configured to transfer information describing the network connection comprising at least a network address and a network port for delivery to at least the second processor; and
the second processor configured to identify when the first processor has failed, responsively inherit the network connection with the external system based at least on the information describing the network connection and packet sequence information received from the external system to resume packet acknowledge/sequence counts established between the first processor and the external system, and handle second storage operations transferred by the external system over the network connection, wherein the second storage operations are related to storage and retrieval of data on at least the one of the plurality of storage drives assigned to the first address space portion of the PCIe address space.
2. The data storage system of claim 1 , comprising:
the second processor configured to establish an inactive network connection for the external system based on the information describing the network connection; and
responsive to identifying when the first processor has failed, the second processor configured to activate the network connection with the external system.
3. The data storage system of claim 1 , comprising:
the second processor configured to determine the packet sequence information by at least transferring a probe packet comprising incorrect counts in acknowledge and sequence portions of a transmission control protocol (TCP) header of the probe packet for delivery to the external system and receiving correct acknowledge/sequence counts included in a response to the probe packet transferred by the external system.
4. The data storage system of claim 3 , comprising:
the second processor configured to splice the packet sequence information into a connection state for the network connection to resume the packet acknowledge/sequence counts established between the first processor and the external system.
5. The data storage system of claim 1 , comprising:
the second processor configured to activate the network connection with the external system by at least associating a network address of the second processor with a media access control (MAC) address used by the first processor.
6. The data storage system of claim 5 , comprising:
the second processor configured to transfer an ARP message on a subnet associated with the second processor to associate a network address of the second processor with the MAC address.
7. The data storage system of claim 1 , wherein the information describing the network connection comprises a network address and a network port associated with the first processor and a network address and a network port associated with the external system.
8. The data storage system of claim 1 , comprising:
the second processor configured to initiate a reboot of the first processor, detect when the first processor functions after the reboot, and initiate transfer of the network connection back to the first processor.
9. The data storage system of claim 1 , wherein the network connection comprises an iSCSI (Internet Small Computer System Interface) network connection for iSCSI storage operations.
10. A method of operating a data storage system, the method comprising:
in a plurality of storage drives, storing data and retrieving the data stored on associated storage media responsive to storage operations;
in a first processor, establishing a network connection with an external system, receiving first storage operations transferred by the external system over the network connection, and handling the first storage operations related to storage and retrieval of data on at least one of the plurality of storage drives assigned to a first address space portion of a Peripheral Component Interconnect Express (PCIe) address space shared among the first processor and at least a second processor;
in the first processor, transferring information describing the network connection comprising at least a network address and a network port for delivery to at least second processor; and
in the second processor, identifying when the first processor has failed, responsively inheriting the network connection with the external system based at least on the information describing the network connection and packet sequence information received from the external system to resume packet acknowledge/sequence counts established between the first processor and the external system, and handling second storage operations transferred by the external system over the network connection, wherein the second storage operations are related to storage and retrieval of data on at least the one of the plurality of storage drives assigned to the first address space portion of the PCIe address space.
11. The method of claim 10 , further comprising:
in the second processor, establishing an inactive network connection for the external system based on the information describing the network connection, and responsive to identifying when the first processor has failed, activating the network connection with the external system.
12. The method of claim 10 , further comprising:
in the second processor, determining the packet sequence information by at least transferring a probe packet comprising incorrect counts in acknowledge and sequence portions of a transmission control protocol (TCP) header of the probe packet for delivery to the external system and receiving correct acknowledge/sequence counts included in a response to the probe packet transferred by the external system.
13. The method of claim 12 , further comprising:
in the second processor splicing the packet sequence information into a connection state for the network connection to resume the packet acknowledge/sequence counts established between the first processor and the external system.
14. The method of claim 10 , further comprising:
in the second processor, activating the network connection with the external system by at least associating a network address of the second processor with a media access control (MAC) address used by the first processor.
15. The method of claim 14 , further comprising:
in the second processor, transferring an ARP message on a subnet associated with the second processor to associate a network address of the second processor with the MAC address.
16. The method of claim 10 , wherein the information describing the network connection comprises a network address and a network port associated with the first processor and a network address and a network port associated with the external system.
17. The method of claim 10 , in the second processor, initiating a reboot of the first processor;
wherein the first processor inherits the network connection from the second processor after functioning from the reboot.
18. The method of claim 10 , wherein the network connection comprises an iSCSI (Internet Small Computer System Interface) network connection for iSCSI storage operations.
19. A modular data storage system, comprising:
a plurality of storage modules each comprising a plurality of solid state drives and configured to communicate over a Peripheral Component Interconnect Express (PCIe) fabric to receive storage operations for storing and retrieving data;
a plurality of processing modules each comprising one or more processors that are communicatively coupled to the PCIe fabric and configured to manage storage operations for a designated subset of the plurality of storage modules over the PCIe fabric;
a first processor configured to establish a network connection with an external system, receive first storage operations transferred by the external system over the network connection, and handle the first storage operations related to storage and retrieval of data on at least one of the plurality of storage drives assigned to a first address space portion of a PCIe address space shared among the PCIe fabric;
the first processor configured to transfer connection state information describing the network connection comprising at least a network address and a network port for delivery to at least a second processor; and
the second processor configured to detect when the first processor has failed, responsively inherit the network connection with the external system in place of the first processor based at least on the connection state information to resume packet acknowledge/sequence counts established between the first processor and the external system, and handle second storage operations transferred by the external system over the network connection, wherein the second storage operations are related to storage and retrieval of data on the at least one of the plurality of storage drives assigned to the first address space portion of the PCIe address space.
20. The modular data storage system of claim 19 , comprising:
the second processor configured to establish an idle network connection for the external system based on the connection state information, wherein the second processor does not transfer any communications over the idle network connection until failure of the first processor;
responsive to detecting when the first processor has failed, the second processor configured to transfer a probe packet comprising incorrect counts in acknowledge and sequence portions of a transmission control protocol (TCP) header of the probe packet for delivery to the external system to identify packet acknowledge/sequence counts based at least on acknowledge/sequence counts received responsive to the probe packet, splice the correct packet acknowledge/sequence counts into a connection state for the network connection to handle the second storage operations in the second processor, and associate a network address of the second processor with a media access control (MAC) address used by the first processor.