IP Library › Granted Patent US 12,218,841
Granted Patent B1
US 12,218,841 · App. 16/712,589 · Granted Feb 4, 2025

Ethernet traffic over scalable reliable datagram protocol

Inventors: Leah Shalev (London, GB); Georgy Zorik Machulsky (San Jose, CA); Peter Nicholas DeSantis (Seattle, WA); Nafea Bshara (San Jose, CA); Omer Ilany (Superior, CO)
Assignee: Amazon Technologies, Inc.
H04L47/10H04L41/0816H04L43/0829H04L69/16H04L69/163
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,218,841
App. No.
16/712,589
Granted
Feb 4, 2025
Kind
B1
Abstract

Methods and apparatuses for improving network packet transmission performance in terms of latency with reduced packet retransmission times and fewer packet drops in congested networks are provided. Packet-switched networks can experience long delays while waiting for out-of-order packets or re-transmissions of lost packets. In addition, network faults such as transmission path failures can result in excessive delay while attempting to find a new route over which to transmit packets. To improve packet transmission performance, application data may be encapsulated into first network packets by a first transport protocol having an interface exposed to the application, in the first network packets may be encapsulated into second network packets according to a second transport protocol. The second transport protocol can enable the second network packets of a same packet flow to be transmitted across multiple paths over the network.

Claims (61)

1. A computing system comprising:

a processor;

a memory coupled to and readable by the processor, wherein the memory is configured to store an application;

a first transport layer protocol module; and

a network interface device having a second transport layer protocol module and operable to communicate with a network,

wherein the first transport layer protocol module is configured to:

receive data from the application executing on the computing system; and

generate, based on the data from the application, first network packets according to a Transmission Control Protocol (TCP) or a User Datagram Protocol (UDP);

wherein the second transport layer protocol module is configured to:

receive the first network packets;

encapsulate the first network packets according to a Scalable Reliable Datagram (SRD) protocol to generate second network packets by adding a set of sequence numbers and path identifiers to the first network packets; and

transmit the second network packets to a destination device via a plurality of paths over a network; and

wherein a retransmission timeout period for the SRD protocol is shorter than a retransmission timeout period for the TCP.

2. The computing system of claim 1 , wherein the second transport layer protocol module is configured to transmit second network packets having a same flow identifier via the plurality of paths.

3. The computing system of claim 1 , wherein the second transport layer protocol module retransmits a packet over a different path than a path over which the packet was originally transmitted.

4. The computing system of claim 1 , wherein the second transport layer protocol module is configured to provide the second network packets to the network via a set of output ports.

5. The computing system of claim 1 , wherein the second transport layer protocol module is configured to provide the second network packets to the network over multiple paths via one output port.

6. An apparatus comprising:

a processor; and

a memory storing instructions, which when executed by the processor, cause the apparatus to:

encapsulate data received from an application according to a first transport layer protocol to form first packets;

encapsulate the first packets according to a second layer transport protocol to form second packets for a first packet flow;

select different paths for transmitting the second packets across a network to a destination device; and

transmit each packet via a path selected for a corresponding packet,

wherein a retransmission timeout period for the second transport layer protocol is shorter than a retransmission timeout period for the first transport layer protocol.

7. The apparatus of claim 6 , further comprising:

a transport layer protocol module configured to encapsulate the first packets according to the second transport layer protocol,

wherein encapsulating the first packets comprises adding a packet flow identifier.

8. The apparatus of claim 7 , wherein the transport layer protocol module is further configured to select different paths for transmitting the second packets having a same flow identifier.

9. The apparatus of claim 7 , wherein encapsulating the first packets comprises adding a packet sequence number, and

wherein the packet sequence number indicates an order of transmission for the second packets.

10. The apparatus of claim 9 , further comprising a transmission timer,

wherein the transport layer protocol module is further configured to retransmit a second packet having a given sequence number when a receipt acknowledgment for the second packet having the given sequence number is not received within a time period set by the transmission timer.

11. The apparatus of claim 6 , wherein the processor is further configured to cause the apparatus to retransmit a second packet over a different network path when a receipt acknowledgment for the second packet is not received.

12. The apparatus of claim 6 , wherein the processor is further configured to:

cause the apparatus to monitor transmission times of the second packets over different paths across the network; and

avoid transmitting the second packets over network paths having longer transmission times than other network paths.

13. The apparatus of claim 6 , wherein the apparatus is further configured to:

receive the second packets via different paths across the network; and

reorder the second packets received out of order based on a sequence number of each second packet.

14. The apparatus of claim 6 , wherein the first transport layer protocol is transport control protocol (TCP) or User Datagram Protocol (UDP), and

wherein the second transport layer protocol is Scalable Reliable Datagram (SRD) transport protocol.

15. A method, comprising:

encapsulating data received from an application according to a first transport layer protocol to form first packets;

encapsulating the first packets according to a second transport layer protocol to form second packets for a first packet flow;

selecting different paths for transmitting the second packets across a network to a destination device; and

transmitting each packet via a path selected for the packet,

wherein a retransmission timeout period for the second transport layer protocol is shorter than a retransmission timeout period for the first transport layer protocol.

16. The method of claim 15 , further comprising:

selecting different paths for transmitting the second packets having a same flow identifier.

17. The method of claim 15 , further comprising:

monitoring transmission times for the different paths; and

selecting a path having a shortest transmission time for transmitting a next second packet.

18. The method of claim 15 , further comprising:

monitoring transmission times for the different paths; and

selecting a path according to network congestion based on flowlet status information and network conditions.

19. The method of claim 15 , further comprising:

retransmitting a second packet over a different path when a receipt acknowledgment is not received within the retransmission time period.

20. The method of claim 15 , further comprising:

receiving the second packets via different paths across the network; and

reordering the second packets received out of order based on a sequence number of each second packet.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 31, 2019
From: SHALEV, LEAH; MACHULSKY, GEORGY ZORIK; DESANTIS, PETER NICHOLAS; BSHARA, NAFEA; ILANY, OMER
To: AMAZON TECHNOLOGIES, INC.
Reel/Frame 051394/0771 →
References Cited (114)
US 5699500A · Dasgupta · 1997 [cited by applicant]
US 6788686B1 · Khotimsky et al. · 2004 [cited by applicant]
US 6990528B1 · Neal et al. · 2006 [cited by applicant]
US 7088718B1 · Srivastava · 2006 [cited by applicant]
US 7171484B1 · Krause et al. · 2007 [cited by applicant]
US 7293107B1 · Hanson · 2007 [cited by examiner]
US 7912064B2 · Elzur · 2011 [cited by applicant]
US 7913077B2 · Ko et al. · 2011 [cited by applicant]
US 8031729B2 · Elzur · 2011 [cited by applicant]
US 8190960B1 · Bahadur et al. · 2012 [cited by applicant]
US 9654385B2 · Chu et al. · 2017 [cited by applicant]
US 9667723B2 · Pandya · 2017 [cited by applicant]
US 9747249B2 · Cherian et al. · 2017 [cited by applicant]
US 9985903B2 · Shalev et al. · 2018 [cited by applicant]
US 9985904B2 · Shalev et al. · 2018 [cited by applicant]
US 10148570B2 · Shalev et al. · 2018 [cited by applicant]
US 10375167B2 · Shamis et al. · 2019 [cited by applicant]
US 10430373B2 · Cherian et al. · 2019 [cited by applicant]
US 10498654B2 · Shalev et al. · 2019 [cited by applicant]
US 10645019B2 · Shalev et al. · 2020 [cited by applicant]
US 10673772B2 · Shalev et al. · 2020 [cited by applicant]
US 10917344B2 · Shalev et al. · 2021 [cited by applicant]
US 20020198927A1 · Craddock et al. · 2002 [cited by applicant]
US 20030031183A1 · Kashyap et al. · 2003 [cited by applicant]
US 20030035433A1 · Craddock et al. · 2003 [cited by applicant]
US 20030053457A1 · Fox et al. · 2003 [cited by applicant]
US 20030065856A1 · Kagan et al. · 2003 [cited by applicant]
US 20030200315A1 · Goldenberg et al. · 2003 [cited by applicant]
US 20040003126A1 · Boucher et al. · 2004 [cited by applicant]
US 20040042458A1 · Elzu · 2004 [cited by applicant]
US 20040042483A1 · Elzur et al. · 2004 [cited by applicant]
US 20040044796A1 · Vangal et al. · 2004 [cited by applicant]
US 20040049612A1 · Boyd et al. · 2004 [cited by applicant]
US 20040049774A1 · Boyd et al. · 2004 [cited by applicant]
US 20040190516A1 · Williams · 2004 [cited by applicant]
US 20050060442A1 · Beverly et al. · 2005 [cited by applicant]
US 20050144310A1 · Biran et al. · 2005 [cited by applicant]
US 20050180327A1 · Banerjee et al. · 2005 [cited by applicant]
US 20060007935A1 · Bennett et al. · 2006 [cited by applicant]
US 20060075067A1 · Blackmore et al. · 2006 [cited by applicant]
US 20060101090A1 · Aloni et al. · 2006 [cited by applicant]
US 20060101225A1 · Aloni et al. · 2006 [cited by applicant]
US 20060168274A1 · Aloni et al. · 2006 [cited by applicant]
US 20070208820A1 · Makhervaks et al. · 2007 [cited by applicant]
US 20080043750A1 · Keels et al. · 2008 [cited by applicant]
US 20090106771A1 · Benner et al. · 2009 [cited by applicant]
US 20090219939A1 · Isosaari · 2009 [cited by applicant]
US 20090222564A1 · Freimuth et al. · 2009 [cited by applicant]
US 20110149977A1 · Thomas et al. · 2011 [cited by applicant]
US 20110280247A1 · Roskind · 2011 [cited by examiner]
US 20120023304A1 · Chan et al. · 2012 [cited by applicant]
US 20120089694A1 · Pandya · 2012 [cited by applicant]
US 20120155458A1 · Larson et al. · 2012 [cited by applicant]
US 20120155468A1 · Greenberg et al. · 2012 [cited by applicant]
US 20120265837A1 · Grant et al. · 2012 [cited by applicant]
US 20120281714A1 · Chang et al. · 2012 [cited by applicant]
US 20120287944A1 · Pandit et al. · 2012 [cited by applicant]
US 20130051222A1 · Gavrilov et al. · 2013 [cited by applicant]
US 20130080561A1 · Fox et al. · 2013 [cited by applicant]
US 20130114606A1 · Schrum, Jr. et al. · 2013 [cited by applicant]
US 20130145035A1 · Pope et al. · 2013 [cited by applicant]
US 20130268980A1 · Russell · 2013 [cited by applicant]
US 20130304796A1 · Jackowski et al. · 2013 [cited by applicant]
US 20130315237A1 · Kagan et al. · 2013 [cited by applicant]
US 20140052808A1 · Krishnan et al. · 2014 [cited by applicant]
US 20140153572A1 · Hampel et al. · 2014 [cited by applicant]
US 20140310369A1 · Makhervaks et al. · 2014 [cited by applicant]
US 20140358972A1 · Guarrieri et al. · 2014 [cited by applicant]
US 20150010090A1 · Sugimoto · 2015 [cited by applicant]
US 20150052280A1 · Lawson · 2015 [cited by applicant]
US 20150124652A1 · Dharmapurikar et al. · 2015 [cited by applicant]
US 20150127797A1 · Attar et al. · 2015 [cited by applicant]
US 20150163144A1 · Koponen et al. · 2015 [cited by applicant]
US 20150172226A1 · Borshteen et al. · 2015 [cited by applicant]
US 20150179227A1 · Russell · 2015 [cited by applicant]
US 20150263932A1 · Chunduri et al. · 2015 [cited by applicant]
US 20150373165A1 · Anand et al. · 2015 [cited by applicant]
US 20160026604A1 · Pandit et al. · 2016 [cited by applicant]
US 20160094356A1 · Xiang et al. · 2016 [cited by applicant]
US 20160212214A1 · Rahman et al. · 2016 [cited by applicant]
US 20160226755A1 · Hammam et al. · 2016 [cited by applicant]
US 20160234127A1 · Agarwal et al. · 2016 [cited by applicant]
US 20160344633A1 · Jiao et al. · 2016 [cited by applicant]
US 20170006142A1 · Kwon · 2017 [cited by applicant]
US 20170134535A1 · Osamura et al. · 2017 [cited by applicant]
US 20170171075A1 · Sajeepa et al. · 2017 [cited by applicant]
US 20170187496A1 · Shalev · 2017 [cited by applicant]
US 20170187621A1 · Shalev · 2017 [cited by examiner]
US 20170187629A1 · Shalev · 2017 [cited by examiner]
US 20170187846A1 · Shalev · 2017 [cited by applicant]
US 20170195240A1 · Chen et al. · 2017 [cited by applicant]
US 20170223154A1 · Hammam et al. · 2017 [cited by applicant]
US 20170346742A1 · Shahar et al. · 2017 [cited by applicant]
US 20180004705A1 · Menachem et al. · 2018 [cited by applicant]
US 20180219980A1 · McCulley · 2018 [cited by examiner]
US 20190363989A1 · Shalev et al. · 2019 [cited by applicant]
US 20200259759A1 · Shalev et al. · 2020 [cited by applicant]
US 20210119930A1 · Debbage · 2021 [cited by examiner]
CN 1625179 · 2005 [cited by applicant]
CN 101848203 · 2010 [cited by applicant]
CN 103929492 · 2014 [cited by applicant]
CN 103986647 · 2014 [cited by applicant]
CN 104541257 · 2015 [cited by applicant]
CN 104620664 · 2015 [cited by applicant]
JP 2002305535 · 2002 [cited by applicant]
JP 2004531175 · 2004 [cited by applicant]
JP 2005524264 · 2005 [cited by applicant]
JP 2008507201 · 2008 [cited by applicant]
JP 2015050746 · 2015 [cited by applicant]
WO 2013169073 · 2013 [cited by applicant]
“LibFabrics: Open Fabrics”, OpenFabrics Interfaces Working Group, Accessed from Internet on Jan. 9, 2017, 6 pages <URL: htth://ofiwg.github.io/libfabric/>. [cited by applicant]
“Open Fabrics Software”, OFED Overview. Open Fabrics Alliance, Sep. 7, 2015, 2 pages <URL: https://www.openfabrics.org/index.php/openfabrics-software.html>. [cited by applicant]
Bryant et al., “Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for Use over an MPLS PSN”, XP015044817, Feb. 1, 2006, 12 pages. [cited by applicant]
Russell, “Introduction to RDMA Programming”, University of New Hampshire InterOperability Laboratom, 2012, 76 pages <URL: http://www.cs.unh.edu/˜rdr/rdma-intro-module.ppt>. [cited by applicant]
Cited By (1)
US 12,598,142