IP Library › Granted Patent US 12,261,926
Granted Patent B2
US 12,261,926 · App. 17/454,731 · Granted Mar 25, 2025

Fabric control protocol for data center networks with packet spraying over multiple alternate data paths

Inventors: Deepak Goel (San Jose, CA); Narendra Jayawant Gathoo (San Jose, CA); Philip A. Thomas (San Jose, CA); Srihari Raju Vegesna (San Jose, CA); Pradeep Sindhu (Los Altos Hills, CA); Wael Noureddine (Santa Clara, CA); Robert William Bowdidge (San Jose, CA); Ayaskant Pani (Fremont, CA); Gopesh Goyal (Cupertino, CA)
Assignee: Microsoft Technology Licensing, LLC
H04L69/26H04L12/4633H04L45/16H04L45/42H04L45/64H04L47/15H04L47/18H04L47/52H04L49/25H04L69/324
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,261,926
App. No.
17/454,731
Granted
Mar 25, 2025
Kind
B2
Abstract

A fabric control protocol is described for use within a data center in which a switch fabric provides full mesh interconnectivity such that any of the servers may communicate packet data for a given packet flow to any other of the servers using any of a number of parallel data paths within the data center switch fabric. The fabric control protocol enables spraying of individual packets for a given packet flow across some or all of the multiple parallel data paths in the data center switch fabric and, optionally, reordering of the packets for delivery to the destination. The fabric control protocol may provide end-to-end bandwidth scaling and flow fairness within a single tunnel based on endpoint-controlled requests and grants for flows. In some examples, the fabric control protocol packet structure is carried over an underlying protocol, such as the User Datagram Protocol (UDP).

Claims (41)

1. A device comprising at least one access node, wherein the at least one access node comprises a source access node configured to:

establish a logical tunnel over a plurality of parallel data paths between the source access node and a destination access node within a computer network, wherein the source access node is coupled to a source server and the destination access node is coupled to a destination server, wherein the source access node and the destination access node are connected by an intermediate network comprising a switch fabric having a plurality of core switches;

send a request message for a packet flow of packets from the source server to the destination server; and

in response to receipt of a grant message for the packet flow, send the packets of the packet flow over the logical tunnel, wherein to send the packets over the logical tunnel, the source access node is configured to spray the packets of the packet flow across the plurality of parallel data paths to the destination access node by directing each of the packets to one of the parallel data paths selected based on bandwidth characteristics of the one of the parallel data paths, wherein the bandwidth characteristics are determined based on a tracking of a number of bytes transmitted on each of the parallel data paths.

2. The device of claim 1 , wherein the source access node has full mesh connectivity to a set of access nodes included in a logical rack as a first-level network fanout, and wherein the source access node is configured to spray the packets of the packet flow across the first-level network fanout to the set of the access nodes included in the logical rack.

3. The device of claim 2 , wherein each access node in the set of access nodes, including the source access node, has full mesh connectivity to a subset of the plurality of core switches as a second-level network fanout, and wherein each access node in the set of access nodes is configured to spray the packets of the packet flow across the second-level network fanout to the subset of the plurality of core switches.

4. The device of claim 1 , wherein the grant message for the packet flow indicates an amount of reserved bandwidth for the packet flow, and wherein to spray the packets of the packet flow, the source access node is configured to spray the packets of the packet flow until an amount of data that is less than or equal to the reserved bandwidth for the packet flow is sent, stopping at a packet boundary.

5. The device of claim 1 , wherein, to spray the packets of the packet flow across the plurality of parallel data paths, the source access node is configured to direct each of the packets of the packet flow to a least loaded one of the parallel data paths selected based on a byte count per path.

6. The device of claim 1 , wherein to spray the packets of the packet flow across the plurality of parallel data paths, the source access node is configured to direct each of the packets of the packet flow to a randomly, pseudo-randomly, or round-robin selected one of the parallel data paths.

7. The device of claim 1 , wherein to spray the packets of the packet flow across the plurality of parallel data paths, the source access node is configured to direct each of the packets of the packet flow to a weighted randomly selected one of the parallel data paths in proportion to available bandwidth in the switch fabric.

8. The device of claim 1 , wherein to spray the packets of the packet flow across the plurality of parallel data paths, the source access node is configured to:

randomly set a different user datagram protocol (UDP) source port in a UDP portion of a header for each of the packets of the packet flow, wherein the plurality of core switches compute a hash of N-fields from the UDP portion of the header for each of the packets; and

direct each of the packets of the packet flow to one of the parallel data paths selected based on the randomly set UDP source port for the respective one of the packets.

9. The device of claim 1 , wherein the source access node executes a fabric control protocol (FCP), wherein the source access node is configured to, in response to the receipt of the grant message, encapsulate the packets of the packet flow within payloads of FCP packets, and

wherein to send the packet of the packet flow, the source access node is configured to spray the FCP packets of the packet flow across the plurality of parallel data paths to the destination access node by directing each of the FCP packets to one of the parallel data paths selected based on bandwidth characteristics of the one of the parallel data paths.

10. A method comprising:

establishing a logical tunnel over a plurality of parallel data paths between a source access node and a destination access node within a computer network, wherein the source access node is coupled to a source server and the destination access node is coupled to a destination server, wherein the source and destination access nodes are connected by an intermediate network comprising a switch fabric having a plurality of core switches;

sending, by the source access node, a request message for a packet flow of packets from the source server to the destination server; and

in response to receipt of a grant message for the packet flow, sending the packets of the packet flow over the logical tunnel, wherein sending the packets over the logical tunnel comprises spraying, by the source access node, the packets across the plurality of parallel data paths to the destination access node by directing each of the packets to one of the parallel data paths selected based on bandwidth characteristics of the one of the parallel data paths, wherein the bandwidth characteristics are determined based on a tracking of a number of bytes transmitted on each of the parallel data paths.

11. A device comprising at least one access node, wherein the at least one access node comprises a destination access node configured to:

establish a logical tunnel over a plurality of parallel data paths between a source access node and the destination access node within a computer network, wherein the source access node is coupled to a source server and the destination access node is coupled to a destination server, wherein the source access node and the destination access node are connected by an intermediate network comprising a switch fabric having a plurality of core switches;

in response to receipt of a request message for a packet flow of packets from the source server to the destination server, perform grant scheduling;

send a grant message for the packet flow of packets;

receive the packets of the packet flow over the logical tunnel in which the packets are sprayed across the plurality of parallel data paths to the destination access node, wherein to receive the packets over the logical tunnel, the destination access node is configured to receive each of the packets on one of the parallel data paths to which the respective packet was directed based on bandwidth characteristics of the one of the parallel data paths, wherein the bandwidth characteristics are determined based on a tracking of a number of bytes transmitted on each of the parallel data paths; and

deliver the packets of the packet flow to the destination server.

12. The device of claim 11 , wherein the destination access node has full mesh connectivity to a set of access nodes included in a logical rack, and wherein the source access node is configured to receive the packets of the packet flow from across the set of the access nodes included in the logical rack.

13. The device of claim 12 , wherein each access node in the set of access nodes, including the destination access node, has full mesh connectivity to a subset of the plurality of core switches, and wherein each access node in the set of access nodes is configured to receive the packets of the packet flow from across the subset of the plurality of core switches.

14. The device of claim 11 , wherein to perform grant scheduling, the destination access node is configured to:

determine an amount of reserved bandwidth to distribute to the packet flow, wherein the grant message for the packet flow indicates the amount of reserved bandwidth for the packet flow; and

send the amount of reserved bandwidth for the packet flow to the source access node in the grant message for the packet flow, wherein the packets of the packet flow are sprayed across the plurality of parallel data paths in accordance with the reserved bandwidth.

15. The device of claim 14 , wherein to receive the packets of the packet flow, the destination access node is configured to receive the packets of the packet flow comprising an amount of data that is less than or equal to the reserved bandwidth for the packet flow up to a packet boundary.

16. The device of claim 11 , wherein the destination access node computes a scale down factor for request messages from the source access node based on a global view of packet flows in the switch fabric, and sends the scale down factor in the grant message for the packet flow, wherein subsequent request messages are sent by the source access node in accordance with the scale down factor.

17. The device of claim 11 , wherein, in response to receiving the packets of the packet flow, the destination access node is configured to reorder the packets into an original sequence of the packet flow prior to delivering the packets of the packet flow to the destination server.

18. The device of claim 17 , wherein each of the packets of the packet flow are assigned a packet sequence number in accordance with the original sequence of the packet flow, and wherein to reorder the packets, the destination access node is configured to reorder the packets into the original sequence of the packet flow based on the packet sequence number assigned to each of the packets.

19. The device of claim 11 , wherein the destination access node executes a fabric control protocol (FCP), wherein the packets of the packet flow are encapsulated within payloads of FCP packets, and wherein in response to receiving the FCP packets of the packet flow, the destination access node is configured to extract the packets of the packet flow that are encapsulated within the payloads of the FCP packets prior to delivering the packets of the packet flow to the destination server.

20. A method comprising:

establishing a logical tunnel over a plurality of parallel data paths between a source access node and a destination access node within a computer network, wherein the source access node is coupled to a source server and the destination access node is coupled to a destination server, wherein the source and destination access nodes are connected by an intermediate network comprising a switch fabric having a plurality of core switches;

in response to receipt of a request message for a packet flow of packets from the source server to the destination server, performing, by the destination access node, grant scheduling;

sending, by the destination access node, a grant message for the packet flow;

receiving, by the destination access node, the packets of the packet flow over the logical tunnel in which the packets of the packet flow were sprayed across the plurality of parallel data paths from the source access node to the destination access node, wherein receiving the packets over the logical tunnel comprises receiving each packet on one of the parallel data paths to which the respective packet was directed based on bandwidth characteristics of the one of the parallel data paths, wherein the bandwidth characteristics are determined based on a tracking of a number of bytes transmitted on each of the parallel data paths; and

delivering, by the destination access node, the packets of the packet flow to the destination server.

Assignments (4)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 31, 2023
From: FUNGIBLE, INC.
To: MICROSOFT TECHNOLOGY LICENSING, LLC
Reel/Frame 064434/0430 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 27, 2023
From: GOEL, DEEPAK; GATHOO, NARENDRA JAYAWANT; THOMAS, PHILIP A.; VEGESNA, SRIHARI RAJU; SINDHU, PRADEEP; NOUREDDINE, WAEL; BOWDIDGE, ROBERT WILLIAM; PANI, AYASKANT; GOYAL, GOPESH
To: FUNGIBLE, INC.
Reel/Frame 064087/0880 →
RELEASE OF SECURITY INTEREST Recorded Jan 9, 2023
From: HERCULES CAPITAL, INC., AS AGENT
To: FUNGIBLE, INC.
Reel/Frame 062335/0803 →
SECURITY INTEREST Recorded Dec 16, 2021
From: FUNGIBLE, INC.
To: HERCULES CAPITAL, INC., AS AGENT
Reel/Frame 058533/0429 →
Continuity (4)
Continuation 16147070 · Sep 28, 2018
Provisional Application 62638788 · Mar 5, 2018
Provisional Application 62566060 · Sep 29, 2017
Related Publication 20220103661A1 · Mar 31, 2022
References Cited (400)
US 4872157A · Hemmady et al. · 1989 [cited by applicant]
US 4872159A · Hemmady et al. · 1989 [cited by applicant]
US 5301324A · Dewey et al. · 1994 [cited by applicant]
US 5812549A · Sethu · 1998 [cited by applicant]
US 5828860A · Miyaoku et al. · 1998 [cited by applicant]
US 6021473A · Davis et al. · 2000 [cited by applicant]
US 6055579A · Goyal et al. · 2000 [cited by applicant]
US 6314491B1 · Freerksen et al. · 2001 [cited by applicant]
US 6526022B1 · Chiu · 2003 [cited by applicant]
US 6553030B2 · Ku et al. · 2003 [cited by applicant]
US 6842906B1 · Bowman-Amuah · 2005 [cited by applicant]
US 6901451B1 · Miyoshi et al. · 2005 [cited by applicant]
US 6901500B1 · Hussain · 2005 [cited by applicant]
US 6990063B1 · Lenoski et al. · 2006 [cited by applicant]
US 6993630B1 · Williams et al. · 2006 [cited by applicant]
US 7035914B1 · Payne et al. · 2006 [cited by applicant]
US 7058009B1 · Skirmont et al. · 2006 [cited by applicant]
US 7102999B1 · Sindhu et al. · 2006 [cited by applicant]
US 7275103B1 · Thrasher et al. · 2007 [cited by applicant]
US 7289436B2 · Schaller et al. · 2007 [cited by applicant]
US 7289513B1 · Medved et al. · 2007 [cited by applicant]
US 7289964B1 · Bowman-Amuah et al. · 2007 [cited by applicant]
US 7342887B1 · Sindhu · 2008 [cited by applicant]
US 7480304B2 · Yeh et al. · 2009 [cited by applicant]
US 7483370B1 · Dayal et al. · 2009 [cited by applicant]
US 7486678B1 · Devanagondi et al. · 2009 [cited by applicant]
US 7623524B2 · Muthukrishnan et al. · 2009 [cited by applicant]
US 7633861B2 · Willhite et al. · 2009 [cited by applicant]
US 7664110B1 · Lovett et al. · 2010 [cited by applicant]
US 7733781B2 · Petersen · 2010 [cited by applicant]
US 7738452B1 · O'Rourke et al. · 2010 [cited by applicant]
US 7802001B1 · Petry et al. · 2010 [cited by applicant]
US 7822731B1 · Yu et al. · 2010 [cited by applicant]
US 7843907B1 · Abou-Emara et al. · 2010 [cited by applicant]
US 7965624B2 · Ripa et al. · 2011 [cited by applicant]
US 8345675B1 · Raghunath · 2013 [cited by applicant]
US 8386540B1 · McAlister et al. · 2013 [cited by applicant]
US 8560757B2 · Pangborn et al. · 2013 [cited by applicant]
US 8565218B2 · Sharma · 2013 [cited by applicant]
US 8582440B2 · Ofelt et al. · 2013 [cited by applicant]
US 8599863B2 · Davis · 2013 [cited by applicant]
US 8625427B1 · Terry et al. · 2014 [cited by applicant]
US 8689049B2 · Ziegler et al. · 2014 [cited by applicant]
US 8737410B2 · Davis et al. · 2014 [cited by applicant]
US 8798077B2 · Mehra et al. · 2014 [cited by applicant]
US 8811183B1 · Anand · 2014 [cited by applicant]
US 8848728B1 · Revah et al. · 2014 [cited by applicant]
US 8850101B2 · Pangborn et al. · 2014 [cited by applicant]
US 8850125B2 · Pangborn et al. · 2014 [cited by applicant]
US 8854953B2 · Enyedi · 2014 [cited by applicant]
US 8918631B1 · Kumar et al. · 2014 [cited by applicant]
US 8937865B1 · Kumar et al. · 2015 [cited by applicant]
US 8953441B2 · Nakil · 2015 [cited by applicant]
US 8966152B2 · Bouchard et al. · 2015 [cited by applicant]
US 9065860B2 · Pangborn et al. · 2015 [cited by applicant]
US 9118984B2 · DeCusatis et al. · 2015 [cited by applicant]
US 9154376B2 · Aziz · 2015 [cited by applicant]
US 9225628B2 · Zahavi · 2015 [cited by applicant]
US 9262225B2 · Davis et al. · 2016 [cited by applicant]
US 9282384B1 · Graves · 2016 [cited by applicant]
US 9294304B2 · Sindhu · 2016 [cited by applicant]
US 9294398B2 · DeCusatis et al. · 2016 [cited by applicant]
US 9369408B1 · Raghavan et al. · 2016 [cited by applicant]
US 9405550B2 · Biran et al. · 2016 [cited by applicant]
US 9565114B1 · Kabbani et al. · 2017 [cited by applicant]
US 9569366B2 · Pangborn et al. · 2017 [cited by applicant]
US 9632936B1 · Zuckerman et al. · 2017 [cited by applicant]
US 9660897B1 · Gredler · 2017 [cited by applicant]
US 9800495B2 · Lu · 2017 [cited by applicant]
US 9853901B2 · Kampmann et al. · 2017 [cited by applicant]
US 9866427B2 · Yadav et al. · 2018 [cited by applicant]
US 9876735B2 · Davis et al. · 2018 [cited by applicant]
US 9946671B1 · Tawri et al. · 2018 [cited by applicant]
US 10003552B2 · Kumar et al. · 2018 [cited by applicant]
US 10135731B2 · Davis et al. · 2018 [cited by applicant]
US 10140245B2 · Davis et al. · 2018 [cited by applicant]
US 10218629B1 · An et al. · 2019 [cited by applicant]
US 10304154B2 · Appu et al. · 2019 [cited by applicant]
US 10387179B1 · Hildebrant et al. · 2019 [cited by applicant]
US 10425707B2 · Sindhu et al. · 2019 [cited by applicant]
US 10540288B2 · Noureddine et al. · 2020 [cited by applicant]
US 10565112B2 · Noureddine et al. · 2020 [cited by applicant]
US 10637685B2 · Goel et al. · 2020 [cited by applicant]
US 10645187B2 · Goyal et al. · 2020 [cited by applicant]
US 10659254B2 · Sindhu et al. · 2020 [cited by applicant]
US 10686729B2 · Sindhu et al. · 2020 [cited by applicant]
US 10725825B2 · Sindhu et al. · 2020 [cited by applicant]
US 10841245B2 · Gray et al. · 2020 [cited by applicant]
US 10904367B2 · Goel et al. · 2021 [cited by applicant]
US 10929175B2 · Goyal et al. · 2021 [cited by applicant]
US 10965586B2 · Goel et al. · 2021 [cited by applicant]
US 11048634B2 · Noureddine et al. · 2021 [cited by applicant]
US 11070474B1 · Jain · 2021 [cited by applicant]
US 11165691B1 · Kompella · 2021 [cited by applicant]
US 11178262B2 · Goel et al. · 2021 [cited by applicant]
US 11934964B2 · Goyal · 2024 [cited by applicant]
US 20020015387A1 · Houh · 2002 [cited by applicant]
US 20020049859A1 · Bruckert et al. · 2002 [cited by applicant]
US 20020075862A1 · Mayes · 2002 [cited by applicant]
US 20020094151A1 · Li et al. · 2002 [cited by applicant]
US 20020118415A1 · Dasylva et al. · 2002 [cited by applicant]
US 20020126634A1 · Mansharamani et al. · 2002 [cited by applicant]
US 20020126671A1 · Ellis et al. · 2002 [cited by applicant]
US 20030043798A1 · Pugel · 2003 [cited by applicant]
US 20030091271A1 · Dragone · 2003 [cited by applicant]
US 20030229839A1 · Wang et al. · 2003 [cited by applicant]
US 20040236912A1 · Glasco · 2004 [cited by applicant]
US 20050013311A1 · Samudrala et al. · 2005 [cited by applicant]
US 20050073963A1 · Goodfellow et al. · 2005 [cited by applicant]
US 20050100035A1 · Chiou et al. · 2005 [cited by applicant]
US 20050108444A1 · Flauaus et al. · 2005 [cited by applicant]
US 20050166086A1 · Watanabe · 2005 [cited by applicant]
US 20050259632A1 · Malpani et al. · 2005 [cited by applicant]
US 20060029323A1 · Nikonov et al. · 2006 [cited by applicant]
US 20060056406A1 · Bouchard et al. · 2006 [cited by applicant]
US 20060067273A1 · Suman · 2006 [cited by applicant]
US 20060112226A1 · Hady et al. · 2006 [cited by applicant]
US 20060277421A1 · Balestriere · 2006 [cited by applicant]
US 20070030807A1 · Gummalla · 2007 [cited by applicant]
US 20070036072A1 · Raj et al. · 2007 [cited by applicant]
US 20070073966A1 · Corbin · 2007 [cited by applicant]
US 20070172235A1 · Snider et al. · 2007 [cited by applicant]
US 20070192545A1 · Gara et al. · 2007 [cited by applicant]
US 20070198656A1 · Mazzaferri et al. · 2007 [cited by applicant]
US 20070255906A1 · Handgen et al. · 2007 [cited by applicant]
US 20080002702A1 · Bajic · 2008 [cited by applicant]
US 20080138067A1 · Beshai · 2008 [cited by applicant]
US 20080244231A1 · Kunze et al. · 2008 [cited by applicant]
US 20080253294A1 · Ripa et al. · 2008 [cited by applicant]
US 20090024836A1 · Shen et al. · 2009 [cited by applicant]
US 20090046576A1 · Shand et al. · 2009 [cited by applicant]
US 20090046587A1 · Kothari · 2009 [cited by applicant]
US 20090083263A1 · Felch et al. · 2009 [cited by applicant]
US 20090135739A1 · Hoover et al. · 2009 [cited by applicant]
US 20090135832A1 · Fan et al. · 2009 [cited by applicant]
US 20090154391A1 · Wittenschlaeger · 2009 [cited by applicant]
US 20090228690A1 · Muff · 2009 [cited by applicant]
US 20090228890A1 · Vaitovirta et al. · 2009 [cited by applicant]
US 20090234987A1 · Lee et al. · 2009 [cited by applicant]
US 20090285228A1 · Bagepalli et al. · 2009 [cited by applicant]
US 20090303880A1 · Maltz et al. · 2009 [cited by applicant]
US 20090310610A1 · Sandstrom · 2009 [cited by applicant]
US 20100061257A1 · Kitawaki · 2010 [cited by applicant]
US 20100061391A1 · Sindhu et al. · 2010 [cited by applicant]
US 20100125903A1 · Devarajan et al. · 2010 [cited by applicant]
US 20100318725A1 · Kwon · 2010 [cited by applicant]
US 20110289179A1 · Pekcan et al. · 2011 [cited by applicant]
US 20110055827A1 · Lin et al. · 2011 [cited by applicant]
US 20110113184A1 · Chu · 2011 [cited by applicant]
US 20110170553A1 · Beecroft et al. · 2011 [cited by applicant]
US 20110173392A1 · Gara et al. · 2011 [cited by applicant]
US 20110202658A1 · Okuno et al. · 2011 [cited by applicant]
US 20110225594A1 · Iyengar et al. · 2011 [cited by applicant]
US 20110228783A1 · Flynn et al. · 2011 [cited by applicant]
US 20110238816A1 · Vohra et al. · 2011 [cited by applicant]
US 20110238923A1 · Hooker et al. · 2011 [cited by applicant]
US 20110289180A1 · Sonnier et al. · 2011 [cited by applicant]
US 20110289279A1 · Sonnier et al. · 2011 [cited by applicant]
US 20120030431A1 · Anderson et al. · 2012 [cited by applicant]
US 20120036178A1 · Gavini et al. · 2012 [cited by applicant]
US 20120076153A1 · Manzella et al. · 2012 [cited by applicant]
US 20120096211A1 · Davis et al. · 2012 [cited by applicant]
US 20120163375A1 · Sindhu et al. · 2012 [cited by applicant]
US 20120176890A1 · Balus et al. · 2012 [cited by applicant]
US 20120177047A1 · Roitshtein · 2012 [cited by applicant]
US 20120207165A1 · Davis · 2012 [cited by applicant]
US 20120254587A1 · Biran et al. · 2012 [cited by applicant]
US 20120314710A1 · Shikano · 2012 [cited by applicant]
US 20130003725A1 · Hendel et al. · 2013 [cited by applicant]
US 20130010636A1 · Regula · 2013 [cited by applicant]
US 20130024875A1 · Wang et al. · 2013 [cited by applicant]
US 20130028083A1 · Yoshida et al. · 2013 [cited by applicant]
US 20130060940A1 · Koponen et al. · 2013 [cited by applicant]
US 20130088971A1 · Anantharam et al. · 2013 [cited by applicant]
US 20130145375A1 · Kang · 2013 [cited by applicant]
US 20130191443A1 · Gan et al. · 2013 [cited by applicant]
US 20130258912A1 · Zimmerman et al. · 2013 [cited by applicant]
US 20130308597A1 · Murphy et al. · 2013 [cited by applicant]
US 20130330076A1 · Liboiron-Ladouceur et al. · 2013 [cited by applicant]
US 20130346789A1 · Brunel et al. · 2013 [cited by applicant]
US 20140023080A1 · Zhang et al. · 2014 [cited by applicant]
US 20140040909A1 · Winser et al. · 2014 [cited by applicant]
US 20140044128A1 · Suresh et al. · 2014 [cited by applicant]
US 20140059537A1 · Kamble et al. · 2014 [cited by applicant]
US 20140075085A1 · Schroder et al. · 2014 [cited by applicant]
US 20140161450A1 · Graves et al. · 2014 [cited by applicant]
US 20140187317A1 · Kohler et al. · 2014 [cited by applicant]
US 20140237156A1 · Regula · 2014 [cited by applicant]
US 20140258479A1 · Tenginakai · 2014 [cited by examiner]
US 20140269261A1 · D'Souza et al. · 2014 [cited by applicant]
US 20140269351A1 · Graves et al. · 2014 [cited by applicant]
US 20140310467A1 · Shalf et al. · 2014 [cited by applicant]
US 20140359044A1 · Davis et al. · 2014 [cited by applicant]
US 20150019702A1 · Kancherla · 2015 [cited by applicant]
US 20150037032A1 · Xu et al. · 2015 [cited by applicant]
US 20150043330A1 · Hu et al. · 2015 [cited by applicant]
US 20150117860A1 · Braun · 2015 [cited by applicant]
US 20150124826A1 · Edsall et al. · 2015 [cited by applicant]
US 20150143045A1 · Han et al. · 2015 [cited by applicant]
US 20150143073A1 · Winser et al. · 2015 [cited by applicant]
US 20150146526A1 · Kulkarni · 2015 [cited by applicant]
US 20150163171A1 · Sindhu et al. · 2015 [cited by applicant]
US 20150180603A1 · Darling et al. · 2015 [cited by applicant]
US 20150186313A1 · Sodhi et al. · 2015 [cited by applicant]
US 20150222533A1 · Birrittella et al. · 2015 [cited by applicant]
US 20150244617A1 · Nakil et al. · 2015 [cited by applicant]
US 20150256405A1 · Janardhanan et al. · 2015 [cited by applicant]
US 20150278148A1 · Sindhu · 2015 [cited by applicant]
US 20150278984A1 · Koker et al. · 2015 [cited by applicant]
US 20150280939A1 · Sindhu · 2015 [cited by applicant]
US 20150281128A1 · Sindhu · 2015 [cited by applicant]
US 20150324205A1 · Eisen et al. · 2015 [cited by applicant]
US 20150325272A1 · Murphy · 2015 [cited by applicant]
US 20150334034A1 · Smedley · 2015 [cited by examiner]
US 20150334202A1 · Frydman et al. · 2015 [cited by applicant]
US 20150378776A1 · Lippett · 2015 [cited by applicant]
US 20150381528A9 · Davis et al. · 2015 [cited by applicant]
US 20160056911A1 · Ye et al. · 2016 [cited by applicant]
US 20160062800A1 · Stanfill et al. · 2016 [cited by applicant]
US 20160087885A1 · Tripathi · 2016 [cited by applicant]
US 20160092362A1 · Barron et al. · 2016 [cited by applicant]
US 20160154756A1 · Dodson · 2016 [cited by applicant]
US 20160164625A1 · Gronvall et al. · 2016 [cited by applicant]
US 20160210159A1 · Wilson et al. · 2016 [cited by applicant]
US 20160212038A1 · Musiol · 2016 [cited by applicant]
US 20160239415A1 · Davis et al. · 2016 [cited by applicant]
US 20160241430A1 · Yadav et al. · 2016 [cited by applicant]
US 20160269278A1 · Gopalarathnam · 2016 [cited by applicant]
US 20160337723A1 · Graves · 2016 [cited by applicant]
US 20160364333A1 · Brown et al. · 2016 [cited by applicant]
US 20160364334A1 · Asaro et al. · 2016 [cited by applicant]
US 20160380885A1 · Jani et al. · 2016 [cited by applicant]
US 20170005921A1 · Liu et al. · 2017 [cited by applicant]
US 20170031719A1 · Clark et al. · 2017 [cited by applicant]
US 20170032011A1 · Song et al. · 2017 [cited by applicant]
US 20170060615A1 · Thakkar et al. · 2017 [cited by applicant]
US 20170061566A1 · Min et al. · 2017 [cited by applicant]
US 20170068639A1 · Davis et al. · 2017 [cited by applicant]
US 20170163569A1 · Koganti · 2017 [cited by applicant]
US 20170187632A1 · Ko et al. · 2017 [cited by applicant]
US 20170235581A1 · Nickolls et al. · 2017 [cited by applicant]
US 20170265220A1 · Andreoli-Fang et al. · 2017 [cited by applicant]
US 20170279708A1 · Liu · 2017 [cited by applicant]
US 20170286143A1 · Wagner et al. · 2017 [cited by applicant]
US 20170286157A1 · Hasting et al. · 2017 [cited by applicant]
US 20170346766A1 · Dutta · 2017 [cited by applicant]
US 20180011739A1 · Pothula et al. · 2018 [cited by applicant]
US 20180024771A1 · Miller et al. · 2018 [cited by applicant]
US 20180026901A1 · Sugunadass · 2018 [cited by applicant]
US 20180115494A1 · Bhatia et al. · 2018 [cited by applicant]
US 20180152317A1 · Chang et al. · 2018 [cited by applicant]
US 20180183673A1 · Iyengar · 2018 [cited by applicant]
US 20180234300A1 · Mayya et al. · 2018 [cited by applicant]
US 20180239702A1 · Farmahini Farahani et al. · 2018 [cited by applicant]
US 20180241688A1 · Williams, Jr. · 2018 [cited by applicant]
US 20180262392A1 · White et al. · 2018 [cited by applicant]
US 20180287818A1 · Goel et al. · 2018 [cited by applicant]
US 20180287858A1 · Flajslik · 2018 [cited by applicant]
US 20180287965A1 · Sindhu et al. · 2018 [cited by applicant]
US 20180288505A1 · Sindhu et al. · 2018 [cited by applicant]
US 20180293168A1 · Noureddine et al. · 2018 [cited by applicant]
US 20180300928A1 · Koker et al. · 2018 [cited by applicant]
US 20180307494A1 · Ould-Ahmed-Vall et al. · 2018 [cited by applicant]
US 20180307535A1 · Suzuki et al. · 2018 [cited by applicant]
US 20180322386A1 · Sridharan et al. · 2018 [cited by applicant]
US 20180357169A1 · Lai · 2018 [cited by applicant]
US 20190005176A1 · Tllikkal et al. · 2019 [cited by applicant]
US 20190012278A1 · Sindhu et al. · 2019 [cited by applicant]
US 20190012350A1 · Sindhu et al. · 2019 [cited by applicant]
US 20190013965A1 · Sindhu et al. · 2019 [cited by applicant]
US 20190018806A1 · Koufaty et al. · 2019 [cited by applicant]
US 20190042292A1 · Palermo et al. · 2019 [cited by applicant]
US 20190042518A1 · Marolia et al. · 2019 [cited by applicant]
US 20190095333A1 · Heirman et al. · 2019 [cited by applicant]
US 20190102311A1 · Gupta et al. · 2019 [cited by applicant]
US 20190104057A1 · Goel et al. · 2019 [cited by applicant]
US 20190104206A1 · Goel et al. · 2019 [cited by applicant]
US 20190104207A1 · Goel et al. · 2019 [cited by applicant]
US 20190158428A1 · Gray et al. · 2019 [cited by applicant]
US 20190188079A1 · Kohli · 2019 [cited by applicant]
US 20190243765A1 · Sindhu et al. · 2019 [cited by applicant]
US 20190319871A1 · Indiresan et al. · 2019 [cited by applicant]
US 20190363989A1 · Shalev et al. · 2019 [cited by applicant]
US 20200021664A1 · Goyal et al. · 2020 [cited by applicant]
US 20200021898A1 · Sindhu et al. · 2020 [cited by applicant]
US 20200067839A1 · Iny et al. · 2020 [cited by applicant]
US 20200084155A1 · Song · 2020 [cited by applicant]
US 20200119903A1 · Thomas et al. · 2020 [cited by applicant]
US 20200133771A1 · Goyal et al. · 2020 [cited by applicant]
US 20200145680A1 · Dikshit et al. · 2020 [cited by applicant]
US 20200151101A1 · Noureddine et al. · 2020 [cited by applicant]
US 20200159568A1 · Goyal et al. · 2020 [cited by applicant]
US 20200159859A1 · Beckman et al. · 2020 [cited by applicant]
US 20200162282A1 · Ashtaputre et al. · 2020 [cited by applicant]
US 20200169513A1 · Goel et al. · 2020 [cited by applicant]
US 20200183841A1 · Noureddine et al. · 2020 [cited by applicant]
US 20200236052A1 · Srinivasan · 2020 [cited by applicant]
US 20200259682A1 · Goel et al. · 2020 [cited by applicant]
US 20200280462A1 · Sindhu et al. · 2020 [cited by applicant]
US 20200314026A1 · Sindhu et al. · 2020 [cited by applicant]
US 20200356414A1 · Sindhu et al. · 2020 [cited by applicant]
US 20210097108A1 · Goyal et al. · 2021 [cited by applicant]
US 20210176347A1 · Goel et al. · 2021 [cited by applicant]
US 20210218665A1 · Goel et al. · 2021 [cited by applicant]
US 20210297343A1 · Vegesna et al. · 2021 [cited by applicant]
US 20210297350A1 · Vegesna et al. · 2021 [cited by applicant]
US 20210297351A1 · Vegesna et al. · 2021 [cited by applicant]
US 20210320820A1 · Ruan et al. · 2021 [cited by applicant]
US 20210399941A1 · Fang · 2021 [cited by applicant]
US 20230208748A1 · Goel · 2023 [cited by applicant]
US 20230269184A1 · Zemach · 2023 [cited by applicant]
US 20230388222A1 · Sindhu · 2023 [cited by applicant]
CN 102123052A · 2011 [cited by applicant]
CN 103067291A · 2013 [cited by applicant]
CN 104052618A · 2014 [cited by applicant]
CN 104937892A · 2015 [cited by applicant]
CN 104954251A · 2015 [cited by applicant]
CN 105024844A · 2015 [cited by applicant]
CN 105814848A · 2016 [cited by applicant]
CN 107196854A · 2017 [cited by applicant]
EP 1079571A2 · 2001 [cited by applicant]
EP 1489796A2 · 2004 [cited by applicant]
EP 1501246A2 · 2005 [cited by applicant]
EP 2289206A2 · 2011 [cited by applicant]
EP 2928134A2 · 2015 [cited by applicant]
WO 2009114554A2 · 2009 [cited by applicant]
WO 2013184846A1 · 2013 [cited by applicant]
WO 2014178854A1 · 2014 [cited by applicant]
WO 2015087474A1 · 2015 [cited by applicant]
WO 2016037262A1 · 2016 [cited by applicant]
WO 2019014268A1 · 2019 [cited by applicant]
“Non Final Office Action Issued in U.S. Appl. No. 16/774,941”, dated May 24, 2023, 18 Pages. [cited by applicant]
“QFX10000 Switches System Architecture,” White Paper, Juniper Networks, Apr. 2015, 15 pp. [cited by applicant]
“Transmission Control Protocol,” DARPA Internet Program Protocol Specification, Information Sciences Institute, RFC 793, Sep. 1981, 91 pp. [cited by applicant]
Adya et al., “Cooperative Task Management without Manual Stack Management,” Proceedings of the 2002 Usenix Annual Technical Conference, Jun. 2002, 14 pp. [cited by applicant]
Al-Fares et al., “Hedera: Dynamic Flow Scheduling for Data Center Networks,” NSDI'10 Proceedings of the 7th USENIX Conference on Networked Systems Design and Implementation, Apr. 28-30, 2010, 15 pp. [cited by applicant]
Alizadeh et al., “Analysis of DCTCP: Stability, Convergence, and Fairness,” Department of Electrical Engineering, Stanford University, SIGMETRICS'11, Jun. 7-11, 2011, 14 pp. [cited by applicant]
Alizadeh et al., “CONGA: Distributed Congestion-Aware Load Balancing for Datacenters,” SIGCOMM '14 Proceedings of the 2014 ACM Conference on SIGCOMM, Aug. 17-22, 2014, pp. 503-514. [cited by applicant]
Allman et al., “TCP Congestion Control”, IETF Network Working Group, RFC 5681, Sep. 2009, 18 pp. [cited by applicant]
Bakkum et al., “Accelerating SQL Database Operations on a GPU with CUDA,” Proceedings of the 3rd Workshop on Genral-Purpose Computation on Graphics Processing Units, Mar. 14, 2010, 10 pp. [cited by applicant]
Banga et al., “Better operating system features for faster network servers,” ACM Sigmetrics Performance Evaluation Review, vol. 26, Issue 3, Dec. 1998, 11 pp. [cited by applicant]
Barroso et al., “Attack of the killer Microseconds,” Communications of the ACM, vol. 60, No. 4, Apr. 2017, 7 pp. [cited by applicant]
Bensley et al., “Data Center TCP (DCTCP): TCP Congestion Control for Data Centers”, IETF, RFC 8257, Oct. 2017, 17 pp. [cited by applicant]
Benson et al., “MicroTE: Fine Grained Traffic Engineering for Data Centers,” CoNEXT '11 Proceedings of the Seventh Conference on emerging Networking EXperiments and Technologies Article No. 8, Dec. 6-9, 2011, 12 pp. [cited by applicant]
Benson et al., “Network Traffic Characteristics of Data Centers in the Wild,” IMC '10 Proceedings of the 10th ACM SIGCOMM Conference on Internet Measurement, Nov. 1-30, 2010, pp. 267-280. [cited by applicant]
Cardwell et al., “BBR Congestion Control”, IETF Draft, Jul. 3, 2017, 34 pp. [cited by applicant]
Cardwell et al., “BBR Congestion Control”, IETF, Nov. 2016, 37 pp. [cited by applicant]
Chen et al., “ Data Center Congestion Management requirements; draft-yueven-tsvwg-dccm-requirements-01,” TSVWG, Internet-Draft, Retrieved Jul. 7, 2021 from: https://tools.ietf.org/id/draft-yueven-tsvwg-dccm-requirements… [cited by applicant]
First Office Action and Search Report, and translation thereof, from counterpart Chinese Application No. 201880062872.7 dated Oct. 8, 2021, 11 pp. [cited by applicant]
Ford et al., “TCP Extensions for Multipath Operation with Multiple Addresses,” Internet Engineering Task Force (IETF), RFC 6824, Jan. 2013, 64 pp. [cited by applicant]
Friedman et al., “Programming with Continuations,” Technical Report 151, Nov. 1983, 14 pp. [cited by applicant]
Gay et al., “The nesC Language: A Holistic Approach to Networked Embedded Systems,” accessed from http://nescc.sourceforge.net, last updated Dec. 14, 2004, 11 pp. [cited by applicant]
Halbwachs et al., “The Synchronous Data Flow Programming Language LUSTRE,” Proceedings of the IEEE, vol. 79, No. 9, Sep. 1991, 16 pp. [cited by applicant]
Haynes et al., “Continuations and Coroutines,” Technical Report No. 158, Jun. 1984, 19 pp. [cited by applicant]
Hewitt, “Viewing Control Structures as Patterns of Passing Messages,” Massachusetts Institute of Technology, Artificial Intelligence Laboratory, Dec. 1976, 61 pp. [cited by applicant]
Hseush et al., Data Path Debugging: Data-Oriented Debugging for a Concurrent Programming Language, PADD '88 Proceedings of the 1988 ACM SIGPLAN and SIGOPS workshop on Parallel and distributed debugging, May 5-6, 1988, 1… [cited by applicant]
Huang et al., “Erasure Coding in Windows Azure Storage,” 2012 USENIX Annual Technical Conference, Jun. 13-15, 2012, 12 pp. [cited by applicant]
International Preliminary Report on Patentability from International Application No. PCT/US2018/053591, dated Dec. 18, 2019, 38 pp. [cited by applicant]
International Search Report and Written Opinion of International Application No. PCT/US2018/053591, dated Jan. 17, 2019, 20 pp. [cited by applicant]
Isen et al., “ESKIMO—Energy Savings using Semantic Knowledge of Inconsequential Memory Occupancy for DRAM subsystem,” 42nd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), Dec. 12-16, 2009, 10 pp. [cited by applicant]
Jacobson et al., “TCP Extensions for High Performance”, IETF Network Working Group, RFC 1323, May 1992, 37 pp. [cited by applicant]
Kahn et al., “Actors as a Special Case of Concurrent Constraint Programming,” ECOOP/OOPSLA '90 Proceedings, Oct. 21-25, 1990, 10 pp. [cited by applicant]
Kaminow, “Optical Integrated Circuits: A Personal Perspective,” Journal of Lightwave Technology, vol. 26, No. 9, May 1, 2008, pp. 994-1004. [cited by applicant]
Kandula et al., “Dynamic Load Balancing Without Packet Reordering,” SIGCOMM Computer Communication Review, vol. 37, No. 2, Apr. 2007, pp. 53-62. [cited by applicant]
Kandula et al., “The Nature of Datacenter Traffic: Measurements & Analysis,” IMC '09 Proceedings of the 9th ACM SIGCOMM conference on Internet measurement, Nov. 4-6, 2009, pp. 202-208. [cited by applicant]
Kelly et al., A Block Diagram Compiler, The Bell System Technical Journal, Dec. 7, 1960, 10 pp. [cited by applicant]
Kounavis et al., “Programming the data path in network processor-based routers,” Software—Practice and Experience, Oct. 21, 2003, 38 pp. [cited by applicant]
Larus et al., “Using Cohort Scheduling to Enhance Server Performance,” Usenix Annual Technical Conference, Jun. 2002, 12 pp. [cited by applicant]
Levis et al., “Tiny OS: An Operating System for Sensor Networks,” Ambient Intelligence, Jan. 2005, 34 pp. [cited by applicant]
Lin et al., A Parameterized Dataflow Language Extension for Embedded Streaming Systems, 2008 International Conference on Embedded Computer Systems: Architectures, Modeling, and Simulation, Jul. 21-24, 2008, 8 pp. [cited by applicant]
Mathis et al., “TCP Selective Acknowledgment Options”, IETF Network Working Group, RFC 2018, Oct. 1996, 12 pp. [cited by applicant]
Mishra et al., “Thread-based vs Event-based Implementation of a Group Communication Service,” Proceedings of the First Merged International Parallel Processing Symposium and Symposium on Parallel and Distributed Process… [cited by applicant]
Mittal et al., “Revisiting Network Support for RDMA”, Department of Electrical Engineering, Stanford University, SIGMETRICS, Jun. 21, 2018, 18 pp. [cited by applicant]
Office Action from U.S. Appl. No. 16/901,991, dated Nov. 3, 2021, 12 pp. [cited by applicant]
Office Action from U.S. Appl. No. 17/248,354, dated Dec. 9, 2021, 11 pp. [cited by applicant]
Oh et al., “Feedback-Based Path Failure Detection and Buffer Blocking Protection for MPTCP,” IEEE/ACM Transactions on Networking, vol. 24, No. 6, Dec. 2016, 12 pp. [cited by applicant]
Paxson et al . . . , “Known TCP Implementation Problems”, IETF Network Working Group, RFC 2525, Mar. 1999, 61 pp. [cited by applicant]
Prosecution History from U.S. Appl. No. 15/939,227, dated Jun. 26, 2019 through Apr. 23, 2020, 51 pp. [cited by applicant]
Prosecution History from U.S. Appl. No. 16/147,070, now issued U.S. Pat. No. 11,178,262, dated Dec. 31, 2019 through Sep. 23, 2021, 284 pp. [cited by applicant]
Prosecution History from U.S. Appl. No. 16/147,099, dated Jan. 8, 2020 through Dec. 30, 2020, 41 pp. [cited by applicant]
Prosecution History from U.S. Appl. No. 16/147,134 dated Jan. 31, 2020 through Mar. 4, 2021, 80 pp. [cited by applicant]
Raiciu et al., “Improving Datacenter Performance and Robustness with Multipath TCP,” ACM SIGCOMM Computer Communication Review—SIGCOMM '11, vol. 41, No. 4, Aug. 2011, pp. 266-277. [cited by applicant]
Response to Office Action dated Nov. 3, 2021, from U.S. Appl. No. 16/901,991, filed Feb. 2, 2022, 8 pp. [cited by applicant]
Response to Second Written Opinion dated Sep. 10, 2019, from International Application No. PCT/ US2018/053591, filed Nov. 8, 2019, 9 pp. [cited by applicant]
Response to Written Opinion dated Jan. 17, 2019, from International Application No. PCT/US2018/053591, filed Jul. 29, 2019, 5 pp. [cited by applicant]
Schroeder et al., “Flash Reliability in Production: The Expected and the Unexpected,” 14th USENIX Conference on File and Storage Technologies (FAST '16), Feb. 22-25, 2016, 15 pp. [cited by applicant]
Second Written Opinion of International Application No. PCT/US2018/053591, dated Sep. 10, 2019, 10 pp. [cited by applicant]
Stevens, “TCP Slow Start, Congestion Avoidance, Fast Retransmit and Fast Recovery Algorithms”, IETF Network Working Group, RFC 2001, Jan. 1997, 6 pp. [cited by applicant]
Varela et al., “The Salsa Programming Language 2.0.0alpha Release Tutorial,” Tensselaer Polytechnic Institute, Nov. 2009, 52 pp. [cited by applicant]
Von Behren et al., “Why Events Are A Bad Idea (for high-concurrency servers),” Proceedings of HotOS IX, May 2003, 6 pp. [cited by applicant]
Welsh et al., “SEDA: An Architecture for Well-Conditioned, Scalable Internet Services,” Eighteenth Symposium on Operating Systems Principles, Oct. 21-24, 2001, 14 pp. [cited by applicant]
Zhang et al., “A LDP Fast Protection Switching Scheme for Concurrent Multiple Failures in MPLS Network,” IEEE, 2009 International Conference on Multimedia Information Networking and Security, Nov. 18-20, 2009, pp. 259-2… [cited by applicant]
Zhu et al., “Congestion Control for Large-Scale RDMA Deployments,” SIGCOMM '15 Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication, Aug. 17-21, 2015, pp. 523-536. [cited by applicant]
Response to Office Action dated Apr. 26, 2022 from U.S. Appl. No. 17/301,185, filed Jul. 26, 2022, 10 pp. [cited by applicant]
Advisory Action from U.S. Appl. No. 16/901,991 dated Jun. 27, 2022, 5 pp. [cited by applicant]
Final Office Action from U.S. Appl. No. 16/901,991, dated Mar. 28, 2022, 10 pp. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 17/248,354, dated Apr. 12, 2022, 9 pp. [cited by applicant]
Response to Office Action dated Dec. 9, 2021, from U.S. Appl. No. 17/248,354, filed Mar. 4, 2022, 3 pp. [cited by applicant]
Office Action from U.S. Appl. No. 17/301,185, dated Apr. 26, 2022, 20 pp. [cited by applicant]
Response to Final Office Action dated Mar. 28, 2022 from U.S. Appl. No. 16/901,991, filed filed Jun. 28, 2022, 10 pp. [cited by applicant]