Packet order recovery in a programmable edge switch in a data center network
Systems and methods include receiving incoming packets associated with flows in a data center network where the flows are forwarded on a per-packet basis; maintaining a state of each of the flows and of received incoming packets; and dequeuing the received incoming packets based on one or more packet dequeue conditions and the state. The edge switch can be one of a Top of Rack switch and a Network Interface Card (NIC) communicatively coupled to a corresponding server. The received incoming packets can utilize a transport protocol including any of Transmission Control Protocol (TCP), Xpress Transport Protocol (XTP), and Stream Control Transmission Protocol (SCTP).
1. An edge switch comprising:
one or more programmable Application Specific Integrated Circuits (ASICs) configured to implement programmable functions, wherein the one or more programmable ASICs include
circuitry configured to receive incoming packets associated with flows in a data center network where the flows are forwarded on a per-packet basis, wherein at a particular time at a particular switch port the flows include a number of active concurrent flows;
circuitry configured to maintain a state table that includes a state of the flows and of received incoming packets, wherein the state table includes a corresponding state for a subset of all flows through the edge switch and entries in the state table are only kept for a refresh time window that has a value where the number of active concurrent flows is orders of magnitude higher than a number of flows in the state table, and wherein the state table is stored in registers of the programmable ASICs and managed via the programmable functions; and
circuitry configured to dequeue the received incoming packets based on one or more packet dequeue conditions and the state table, including re-ordering of the received incoming packets in a correct order based on the state table such that the re-ordering is performed at the edge switch as opposed to a server, wherein the re-ordering comprises dequeuing a packet of the received incoming packets within the edge switch before being transmitted to the server when a sequence number of the packet indicates a correct order and, when the sequence number of the packet indicates an incorrect order, maintaining a payload of the packet and recirculating a header of the packet.
2. The edge switch of claim 1 , wherein the circuitry configured to receive incoming packets is configured to queue the received incoming packets, and the circuitry configured to maintain the state table is configured to maintain an entry for each of the received incoming packets based on a flow identifier and a respective sequence number.
3. The edge switch of claim 2 , further comprising
circuitry configured to process respective headers of the received incoming packets in the queue, for the circuitry configured to dequeue the received incoming packets.
4. The edge switch of claim 1 , wherein the state table is managed in memory in the edge switch, and further comprising
circuitry configured to delete entries in the memory that are stale and that are dequeued.
5. The edge switch of claim 1 , wherein the one or more packet dequeue conditions include any of
a packet satisfying a next expected sequence number,
a packet having a much smaller sequence number than expected indicative of a retransmitted packet,
memory storing the state table is overfilled for a particular flow, and
an entry for the state table is stale.
6. The edge switch of claim 1 , wherein the edge switch is one of a Top of Rack switch and a Network Interface Card (NIC) communicatively coupled to a corresponding server.
7. The edge switch of claim 1 , wherein the received incoming packets utilize a transport protocol including any of Transmission Control Protocol (TCP), Xpress Transport Protocol (XTP), and Stream Control Transmission Protocol (SCTP).
8. The edge switch of claim 1 , wherein, for a particular flow, a first packet takes a different path in the data center network than a second packet such that the second packet is received out-of-order at the edge switch.
9. A method comprising:
in an edge switch with one or more programmable Application Specific Integrated Circuits (ASICs) configured to implement programmable functions, receiving incoming packets associated with flows in a data center network where the flows are forwarded on a per-packet basis, wherein at a particular time at a particular switch port the flows include a number of active concurrent flows;
maintaining a state table that includes a state of the flows and of received incoming packets, wherein the state table includes a corresponding state for a subset of all flows through the edge switch and entries in the state table are only kept for a refresh time window that has a value where the number of active concurrent flows is orders of magnitude higher than a number of flows in the state table, and wherein the state table is stored in registers of the programmable ASICs and managed via the programmable functions; and
dequeuing the received incoming packets based on one or more packet dequeue conditions and the state table, including re-ordering of the received incoming packets in a correct order based on the state table such that the re-ordering is performed at the edge switch as opposed to a server, wherein the re-ordering comprises dequeuing a packet of the received incoming packets within the edge switch before being transmitted to the server when a sequence number of the packet indicates a correct order and, when the sequence number of the packet indicates an incorrect order, maintaining a payload of the packet and recirculating a header of the packet.
10. The method of claim 9 , wherein the receiving incoming packets includes queueing the received incoming packets, and the maintaining includes maintaining an entry for each of the received incoming packets based on a flow identifier and a respective sequence number.
11. The method of claim 10 , further comprising
processing respective headers of the received incoming packets in the queue, for the dequeuing.
12. The method of claim 9 , wherein the state table is managed in memory in the edge switch, and further comprising
deleting entries in the memory that are stale and for packets that are dequeued.
13. The method of claim 9 , wherein the one or more packet dequeue conditions include any of
a packet satisfying a next expected sequence number,
a packet having a much smaller sequence number than expected indicative of a retransmitted packet,
memory storing the state table is overfilled for a particular flow, and
an entry for the state table is stale.
14. The method of claim 9 , wherein the method is implemented in an edge switch that is one of a Top of Rack switch and a Network Interface Card (NIC) communicatively coupled to a corresponding server.
15. The method of claim 9 , wherein the received incoming packets utilize a transport protocol including any of Transmission Control Protocol (TCP), Xpress Transport Protocol (XTP), and Stream Control Transmission Protocol (SCTP).
16. The method of claim 9 , wherein, for a particular flow, a first packet takes a different path in the data center network than a second packet such that the second packet is received out-of-order at the edge switch.
17. A data center network comprising:
a plurality of switches communicatively coupled to a plurality of servers,
wherein packets associated with flows in the data center network are forwarded on a per-packet basis, wherein at a particular time at a particular switch port the flows include a number of active concurrent flows;, and
wherein each of the plurality of switches include one or more programmable Application Specific Integrated Circuits (ASICs) configured to implement programmable functions, wherein the one or more programmable ASICs include are configured to
maintain a state table that includes a state of the flows and of received incoming packets, wherein the state table includes a corresponding state for a subset of all flows through a particular edge switch of the plurality of switches and entries in the state table are only kept for a refresh time window that has a value where the number of active concurrent flows is orders of magnitude higher than a number of flows in the state table, and wherein the state table is stored in registers of the programmable ASICs and managed via the programmable functions, and
dequeue the received incoming packets based on one or more packet dequeue conditions and the state table, including re-ordering of the received incoming packets in a correct order based on the state table such that the re-ordering is performed at the particular edge switch as opposed to a server, wherein the re-ordering comprises dequeuing a packet of the received incoming packets within the particular edge switch before being transmitted to the server when a sequence number of the packet indicates a correct order and, when the sequence number of the packet indicates an incorrect order, maintaining a payload of the packet and recirculating a header of the packet.
18. The data center network of claim 17 , wherein the one or more packet dequeue conditions include any of
a packet satisfying a next expected sequence number,
a packet having a much smaller sequence number than expected indicative of a retransmitted packet,
memory storing the state table is overfilled for a particular flow, and
an entry for the state table is stale.
19. The data center network of claim 17 , wherein the plurality of switches are one of a Top of Rack switch and a Network Interface Card (NIC) communicatively coupled to a corresponding server.
20. The data center network of claim 17 , wherein the received incoming packets utilize a transport protocol including any of Transmission Control Protocol (TCP), Xpress Transport Protocol (XTP), and Stream Control Transmission Protocol (SCTP).