System and method for fault recovery in spray based networks
Embodiments of the present disclosure include systems and methods for fault detection and recovery over a network. A value of a set of values is stored in packets transmitted during a data transaction between a source and destination. The value corresponds to ports used by one or more switches in the path between the source and destination. The destination includes the value in an acknowledgement packet. Logic circuits in the source device track packets and corresponding values. When a status indicates a particular packet has not received an acknowledgement, the value for the packet may be removed from the set of values. Particular ports that may be congested or down may be detected and the packets re-routed using the logic circuits in the source device.
1. A circuit comprising:
a memory;
a network interface circuit configured to transmit and receive a plurality of packets and a plurality of acknowledgement packets between a source device and a destination device, wherein said circuit is part of the source device; and
a logic circuit configured to:
store, in the memory, a plurality of values mapping to a plurality of ports of a first switch in which the plurality of packets passes through during transmission to the destination device;
store a value of the plurality of values in each packet of the plurality of packets, cycling through the plurality of values on successive packets, being transmitted to the destination device, wherein the stored value indicates a port of the plurality of ports through which the respective packet is to pass; and
remove a particular value of the plurality of values stored in the memory based on a status of pending acknowledgement packets, each of the pending acknowledgement packets comprising a payload storing the particular value, wherein the status comprises a count of the pending acknowledgement packets being above a threshold.
2. The circuit of claim 1 , wherein the first switch selects an indicated port for a particular packet based on the stored value of the particular packet.
3. The circuit of claim 1 , wherein the status is a timeout.
4. The circuit of claim 1 , wherein the particular value is removed at a sampling interval upon determining that the count of the pending acknowledgement packets is above the threshold at the sampling interval.
5. The circuit of claim 1 , further comprising storing, in the memory, a packet identifier, wherein the packet identifier for each packet being transmitted is associated with the value stored in the packet.
6. The circuit of claim 1 , wherein the value is stored in a User Datagram Protocol (UDP) source field of a header of the respective packet of the plurality of packets being transmitted to the destination.
7. The circuit of claim 1 , wherein the logic circuit is further configured to, for each packet received from a source, store a received value from a received packet in a second acknowledgement packet transmitted back to the source.
8. The circuit of claim 7 , wherein the received value is stored in a payload of the second acknowledgement packet transmitted back to the source.
9. The circuit of claim 1 , wherein a first number of the plurality of values is greater than a second number of the plurality of ports.
10. The circuit of claim 9 , wherein the plurality of ports are specified as hash function outputs of the plurality of values.
11. The circuit of claim 1 , wherein the logic circuit stores different values of the plurality of values in different packets of the plurality of packets being transmitted to the destination so that different packets pass through different ports of the plurality of ports of the first switch during the transmission to the destination.
12. The circuit of claim 1 , wherein the network interface circuit receives the plurality of acknowledgement packets in response to at least a portion of the plurality of packets transmitted to the destination device, the plurality of acknowledgement packets each comprising a value from a corresponding transmitted packet, the logic circuit deleting, from the memory, a first value of the values stored in memory when the value stored in the acknowledgement packet matches the first value.
13. The circuit of claim 1 , wherein the network interface circuit comprises:
a receive processor configured to receive packets; and
a transmit processor configured to transmit packets,
wherein a first portion of the logic circuit is coupled to the transmit processor and a second portion of the logic circuit is coupled to the receive processor,
wherein the first portion of the logic circuit stores the value in each packet being transmitted and further stores, for each packet, a packet identifier and the value in a first portion of the memory, and
wherein the second portion of the logic circuit extracts a first value from an acknowledgement packet and sends a signal coupled to the first portion of the logic circuit that the first value was received, wherein the first portion of the logic circuit deletes, from the first portion of the memory, the first value and the packet identifier in response to the signal.
14. The circuit of claim 1 , wherein the packets are transmitted over a spray network, wherein the plurality of packets being transmitted are sent to the first switch coupled to said circuit, and wherein the plurality of packets are forwarded, by the first switch, to a plurality of second network switches using a plurality of different ports of the first switch, wherein particular ports are selected based on a particular value stored in each particular packet of the plurality of packets.
15. A method of recovering from faults in a network comprising:
storing, in a memory of a source device, a plurality of values mapping to a plurality of ports of a first switch in which a plurality of packets passes through during transmission to a destination device;
storing, using the source device, a value of the plurality of values in each packet of the plurality of packets, cycling through the plurality of values on successive packets, to be transmitted from the source device to the destination device during a data transmission,
wherein the stored value indicates a port of the plurality of ports through which the respective packet is to pass;
transmitting the plurality of packets; and
removing a particular value of the plurality of values stored in the memory based on a status of pending acknowledgement packets, each of the pending acknowledgement packets comprising a payload storing the particular value, wherein the status comprises a count of the pending acknowledgement packets being above a threshold.
16. The method of claim 15 , wherein the status is a timeout.
17. The method of claim 15 , wherein the particular value is removed at a sampling interval upon determining that the count of the pending acknowledgement packets is above the threshold at the sampling interval.
18. The method of claim 15 , wherein the value is stored in a User Datagram Protocol (UDP) source field of a header of the plurality of packets being transmitted to the destination device.
19. The method of claim 15 , wherein the packets are transmitted over a spray network.
20. A storage device for storing a hardware definition language (HDL) program executable by a computer, the program comprising sets of instructions for:
storing, in a memory of a source device, a plurality of values mapping to a plurality of ports of a first switch in which a plurality of packets passes through during transmission to a destination device;
storing, using the source device, a value of the plurality of values in each packet of the plurality of packets, cycling through the plurality of values on successive packets, to be transmitted from the source device to the destination device during a data transmission,
wherein the stored value indicates a port of the plurality of ports through which the respective packet is to pass;
transmitting the plurality of packets; and
removing a particular value of the plurality of values stored in the memory based on a status of pending acknowledgement packets, each of the pending acknowledgement packets comprising a payload storing the particular value, wherein the status comprises a count of the pending acknowledgement packets being above a threshold.