IP Library › Granted Patent US 12,231,353
Granted Patent B2
US 12,231,353 · App. 16/774,941 · Granted Feb 18, 2025

Fabric control protocol for data center networks with packet spraying over multiple alternate data paths

Inventors: Deepak Goel (San Jose, CA); Narendra Jayawant Gathoo (San Jose, CA); Philip A Thomas (San Jose, CA); Srihari Raju Vegesna (San Jose, CA); Pradeep Sindhu (Los Altos Hills, CA); Wael Noureddine (Santa Clara, CA); Robert William Bowdidge (San Jose, CA); Ayaskant Pani (Fremont, CA); Gopesh Goyal (Cupertino, CA)
Assignee: Microsoft Technology Licensing, LLC
H04L49/25H04L12/4633H04L45/22H04L47/34H04L47/41H04L67/10H04L69/16
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,231,353
App. No.
16/774,941
Granted
Feb 18, 2025
Kind
B2
Abstract

A fabric control protocol is described for use within a data center in which a switch fabric provides full mesh interconnectivity such that any of the servers may communicate packet data for a given packet flow to any other of the servers using any of a number of parallel data paths within the data center switch fabric. The fabric control protocol enables spraying of individual packets for a given packet flow across some or all of the multiple parallel data paths in the data center switch fabric and, optionally, reordering of the packets for delivery to the destination. The fabric control protocol may provide end-to-end bandwidth scaling and flow fairness within a single tunnel based on endpoint-controlled requests and grants for flows. In some examples, the fabric control protocol packet structure is carried over an underlying protocol, such as the User Datagram Protocol (UDP).

Claims (51)

1. A system comprising:

a plurality of servers, including a source server and a destination server;

a switch fabric comprising a plurality of core switches;

a destination access node coupled to at least the destination server, and

a source access node coupled to at least the source server, wherein the source access node communicates with the destination access node over a plurality of data paths through the switch fabric, wherein the source access node is configured to spray, over the plurality of data paths, packets of a packet flow between the source access node and the destination access node, and wherein to spray the packets of the packet flow, the source access node:

tracks a number of bytes of the packets of the packet flow previously transmitted on each of the plurality of data paths by the source access node,

determines, for each of the plurality of data paths, a count of bytes previously transmitted on each respective path, and

directs each of the packets of the packet flow to a least loaded data path selected based on the count of bytes previously transmitted on each respective path, wherein a first packet and a second packet of the packets of the packet flow travel over different data paths of the plurality of data paths.

2. The system of claim 1 , wherein the source access node is further configured to:

insert a sequence number into each of the packets of the packet flow.

3. The system of claim 2 , wherein the destination access node is further configured to:

use the sequence number within each of the packets of the packet flow received over the switch fabric to reorder the packets.

4. The system of claim 1 , wherein the source access node is further configured to:

identify a failed data path from among the plurality of data paths.

5. The system of claim 4 , wherein to spray the packets of the packet flow between the source access node and the destination access node, the source access node is further configured to:

direct each of the packets of the packet flow to a data path selected further based on information about the failed data path.

6. The system of claim 5 , wherein to spray the packets of the packet flow between the source access node and the destination access node, the source access node is further configured to:

refrain from transmitting packets over the failed data path.

7. The system of claim 1 , wherein to spray the packets of the packet flow between the source access node and the destination access node, the source access node is further configured to:

direct each of the packets of the packet flow to a data path selected further based on a bandwidth value associated with the data path.

8. The system of claim 1 , wherein the source access node is further configured to:

determine, for each of the plurality of data paths, a weighted usage value, wherein the weighted usage value is a proportion of the number of bytes transmitted over one of the plurality of data paths to the number of bytes transmitted over all of the plurality of data paths.

9. The system of claim 8 , wherein to spray the packets of the packet flow between the source access node and the destination access node, the source access node is further configured to:

direct each of the packets of the packet flow to a data path based on the weighted usage value for each of the plurality of data paths.

10. The system of claim 9 , wherein to direct each of the packets of the packet flow to a data path based on the weighted usage value for each of the plurality of data paths, the source access node is further configured to:

choose the data path having the smallest weighted usage value.

11. A method comprising:

establishing a logical tunnel over a plurality of data paths between a source access node and a destination access node within a computer network, wherein the source access node is coupled to a source server, and the destination access node is coupled to a destination server, and wherein the source access node and the destination access node are connected by a switch fabric having a plurality of core switches; and

spraying, by the source access node and over the plurality of data paths through the switch fabric, packets of a packet flow between the source access node and the destination access node, wherein spraying the packets of the packet flow includes:

tracking a number of bytes of the packets of the packet flow previously transmitted on each of the plurality of data paths by the source access node,

determining, for each of the plurality of data paths, a count of bytes previously transmitted on each respective path, and

directing each of the packets of the packet flow to a data path selected based on the count of bytes previously transmitted on each respective path, wherein a first packet and a second packet of the packets of the packet flow travel over different data paths of the plurality of data paths.

12. The method of claim 11 , further comprising:

inserting, by the source access node, a sequence number into each of the packets of the packet flow; and

using, by the destination access node, the sequence number within each of the packets of the packet flow received over the switch fabric to reorder the packets.

13. The method of claim 11 , further comprising:

identifying, by the source access node, a failed data path from among the plurality of data paths.

14. The method of claim 13 , wherein spraying the packets of the packet flow between the source access node and the destination access node includes:

directing each of the packets of the packet flow to the data path selected further based on information about the failed data path.

15. The method of claim 14 , wherein directing each of the packets of the packet flow to the data path selected further based on information about the failed data path includes:

refraining from transmitting packets over the failed data path.

16. A system comprising:

a plurality of servers, including a source server and a destination server;

a switch fabric comprising a plurality of core switches; and

a source access node coupled to at least the source server, wherein the source access node communicates with the destination server over a plurality of data paths through the switch fabric, and wherein the source access node is configured to spray packets of a packet flow between the source access node and a destination access node coupled to the destination server over the plurality of data paths by:

tracking a number of bytes of the packets of the packet flow previously transmitted on each of the plurality of data paths by the source access node,

determining, for each of the plurality of data paths, a count of bytes previously transmitted on each respective path, and

directing each of the packets of the packet flow to a data path selected based on the count of bytes previously transmitted on each respective path, wherein a first packet and a second packet of the packets of the packet flow travel over different data paths of the plurality of data paths.

17. The system of claim 16 , wherein the source access node is further configured to:

insert a sequence number into each of the packets of the packet flow; and

enable the destination server to use the sequence number within each of the packets of the packet flow received over the switch fabric to reorder the packets.

Assignments (6)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 31, 2023
From: FUNGIBLE, INC.
To: MICROSOFT TECHNOLOGY LICENSING, LLC
Reel/Frame 064434/0430 →
RELEASE OF SECURITY INTEREST Recorded Jan 9, 2023
From: HERCULES CAPITAL, INC., AS AGENT
To: FUNGIBLE, INC.
Reel/Frame 062335/0803 →
RELEASE OF SECURITY INTEREST Recorded Jan 5, 2023
From: SILICON VALLEY BANK
To: FUNGIBLE, INC.
Reel/Frame 062308/0191 →
SECURITY INTEREST Recorded Dec 16, 2021
From: FUNGIBLE, INC.
To: SILICON VALLEY BANK
Reel/Frame 058523/0960 →
SECURITY INTEREST Recorded Dec 16, 2021
From: FUNGIBLE, INC.
To: HERCULES CAPITAL, INC., AS AGENT
Reel/Frame 058533/0429 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 15, 2020
From: GOEL, DEEPAK; GATHOO, NARENDRA JAYAWANT; THOMAS, PHILIP A; VEGESNA, SRIHARI RAJU; SINDHU, PRADEEP; NOUREDDINE, WAEL; BOWDIDGE, ROBERT WILLIAM; PANI, AYASKANT; GOYAL, GOPESH
To: FUNGIBLE, INC.
Reel/Frame 052671/0094 →
Continuity (4)
Continuation In Part 16147070 · Sep 28, 2018
Provisional Application 62638788 · Mar 5, 2018
Provisional Application 62566060 · Sep 29, 2017
Related Publication 20200169513A1 · May 28, 2020
References Cited (400)
US 4872157A · Hemmady et al. · 1989 [cited by applicant]
US 4872159A · Hemmady et al. · 1989 [cited by applicant]
US 5301324A · Dewey et al. · 1994 [cited by applicant]
US 5812549A · Sethu · 1998 [cited by applicant]
US 5828860A · Miyaoku et al. · 1998 [cited by applicant]
US 6021473A · Davis et al. · 2000 [cited by applicant]
US 6055579A · Goyal et al. · 2000 [cited by applicant]
US 6314491B1 · Freerksen et al. · 2001 [cited by applicant]
US 6526022B1 · Chiu · 2003 [cited by applicant]
US 6553030B2 · Ku et al. · 2003 [cited by applicant]
US 6842906B1 · Bowman-Amuah · 2005 [cited by applicant]
US 6901451B1 · Miyoshi et al. · 2005 [cited by applicant]
US 6901500B1 · Hussain · 2005 [cited by applicant]
US 6990063B1 · Lenoski et al. · 2006 [cited by applicant]
US 6993630B1 · Williams et al. · 2006 [cited by applicant]
US 7035914B1 · Payne et al. · 2006 [cited by applicant]
US 7058009B1 · Skirmont et al. · 2006 [cited by applicant]
US 7102999B1 · Sindhu et al. · 2006 [cited by applicant]
US 7275103B1 · Thrasher · 2007 [cited by examiner]
US 7289436B2 · Schaller et al. · 2007 [cited by applicant]
US 7289513B1 · Medved et al. · 2007 [cited by applicant]
US 7289964B1 · Bowman-Amuah et al. · 2007 [cited by applicant]
US 7342887B1 · Sindhu · 2008 [cited by applicant]
US 7480304B2 · Yeh et al. · 2009 [cited by applicant]
US 7483370B1 · Dayal et al. · 2009 [cited by applicant]
US 7486678B1 · Devanagondi et al. · 2009 [cited by applicant]
US 7623524B2 · Muthukrishnan et al. · 2009 [cited by applicant]
US 7633861B2 · Willhite et al. · 2009 [cited by applicant]
US 7664110B1 · Lovett et al. · 2010 [cited by applicant]
US 7733781B2 · Petersen · 2010 [cited by applicant]
US 7738452B1 · O'Rourke et al. · 2010 [cited by applicant]
US 7802001B1 · Petry et al. · 2010 [cited by applicant]
US 7822731B1 · Yu et al. · 2010 [cited by applicant]
US 7843907B1 · Abou-Emara et al. · 2010 [cited by applicant]
US 7965624B2 · Ripa et al. · 2011 [cited by applicant]
US 8345675B1 · Raghunath · 2013 [cited by applicant]
US 8386540B1 · McAlister et al. · 2013 [cited by applicant]
US 8560757B2 · Pangborn et al. · 2013 [cited by applicant]
US 8565218B2 · Sharma · 2013 [cited by applicant]
US 8582440B2 · Ofelt et al. · 2013 [cited by applicant]
US 8599863B2 · Davis · 2013 [cited by applicant]
US 8625427B1 · Terry et al. · 2014 [cited by applicant]
US 8689049B2 · Ziegler et al. · 2014 [cited by applicant]
US 8737410B2 · Davis et al. · 2014 [cited by applicant]
US 8798077B2 · Mehra et al. · 2014 [cited by applicant]
US 8811183B1 · Anand · 2014 [cited by applicant]
US 8848728B1 · Revah · 2014 [cited by examiner]
US 8850101B2 · Pangborn et al. · 2014 [cited by applicant]
US 8850125B2 · Pangborn et al. · 2014 [cited by applicant]
US 8854953B2 · Enyedi · 2014 [cited by applicant]
US 8918631B1 · Kumar et al. · 2014 [cited by applicant]
US 8937865B1 · Kumar · 2015 [cited by examiner]
US 8953441B2 · Nakil · 2015 [cited by applicant]
US 8966152B2 · Bouchard et al. · 2015 [cited by applicant]
US 9065860B2 · Pangborn et al. · 2015 [cited by applicant]
US 9118984B2 · DeCusatis et al. · 2015 [cited by applicant]
US 9154376B2 · Aziz et al. · 2015 [cited by applicant]
US 9225628B2 · Zahavi · 2015 [cited by applicant]
US 9262225B2 · Davis et al. · 2016 [cited by applicant]
US 9282384B1 · Graves · 2016 [cited by applicant]
US 9294304B2 · Sindhu · 2016 [cited by applicant]
US 9294398B2 · DeCusatis et al. · 2016 [cited by applicant]
US 9369408B1 · Raghavan et al. · 2016 [cited by applicant]
US 9405550B2 · Biran et al. · 2016 [cited by applicant]
US 9565114B1 · Kabbani et al. · 2017 [cited by applicant]
US 9569366B2 · Pangborn et al. · 2017 [cited by applicant]
US 9632936B1 · Zuckerman et al. · 2017 [cited by applicant]
US 9660897B1 · Gredler · 2017 [cited by applicant]
US 9800495B2 · Lu et al. · 2017 [cited by applicant]
US 9853901B2 · Kampmann et al. · 2017 [cited by applicant]
US 9866427B2 · Yadav et al. · 2018 [cited by applicant]
US 9876735B2 · Davis et al. · 2018 [cited by applicant]
US 9946671B1 · Tawri et al. · 2018 [cited by applicant]
US 10003552B2 · Kumar et al. · 2018 [cited by applicant]
US 10135731B2 · Davis et al. · 2018 [cited by applicant]
US 10140245B2 · Davis et al. · 2018 [cited by applicant]
US 10218629B1 · An et al. · 2019 [cited by applicant]
US 10304154B2 · Appu et al. · 2019 [cited by applicant]
US 10387179B1 · Hildebrant et al. · 2019 [cited by applicant]
US 10425707B2 · Sindhu et al. · 2019 [cited by applicant]
US 10540288B2 · Noureddine et al. · 2020 [cited by applicant]
US 10565112B2 · Noureddine et al. · 2020 [cited by applicant]
US 10637685B2 · Goel et al. · 2020 [cited by applicant]
US 10645187B2 · Goyal et al. · 2020 [cited by applicant]
US 10659254B2 · Sindhu et al. · 2020 [cited by applicant]
US 10686729B2 · Sindhu et al. · 2020 [cited by applicant]
US 10725825B2 · Sindhu et al. · 2020 [cited by applicant]
US 10841245B2 · Gray et al. · 2020 [cited by applicant]
US 10904367B2 · Goel et al. · 2021 [cited by applicant]
US 10929175B2 · Goyal et al. · 2021 [cited by applicant]
US 10965586B2 · Goel et al. · 2021 [cited by applicant]
US 11048634B2 · Noureddine et al. · 2021 [cited by applicant]
US 11070474B1 · Jain · 2021 [cited by applicant]
US 11165691B1 · Kompella et al. · 2021 [cited by applicant]
US 11178262B2 · Goel et al. · 2021 [cited by applicant]
US 11412076B2 · Goel et al. · 2022 [cited by applicant]
US 11601359B2 · Goel et al. · 2023 [cited by applicant]
US 11934964B2 · Goyal · 2024 [cited by applicant]
US 20020015387A1 · Houh · 2002 [cited by applicant]
US 20020049859A1 · Bruckert et al. · 2002 [cited by applicant]
US 20020075862A1 · Mayes · 2002 [cited by applicant]
US 20020094151A1 · Li · 2002 [cited by applicant]
US 20020118415A1 · Dasylva et al. · 2002 [cited by applicant]
US 20020126634A1 · Mansharamani et al. · 2002 [cited by applicant]
US 20020126671A1 · Ellis et al. · 2002 [cited by applicant]
US 20030043798A1 · Pugel · 2003 [cited by applicant]
US 20030091271A1 · Dragone · 2003 [cited by applicant]
US 20030229839A1 · Wang et al. · 2003 [cited by applicant]
US 20040236912A1 · Glasco · 2004 [cited by applicant]
US 20050013311A1 · Samudrala et al. · 2005 [cited by applicant]
US 20050073963A1 · Goodfellow et al. · 2005 [cited by applicant]
US 20050100035A1 · Chiou et al. · 2005 [cited by applicant]
US 20050108444A1 · Flauaus et al. · 2005 [cited by applicant]
US 20050166086A1 · Watanabe · 2005 [cited by applicant]
US 20050259632A1 · Malpani et al. · 2005 [cited by applicant]
US 20060029323A1 · Nikonov et al. · 2006 [cited by applicant]
US 20060056406A1 · Bouchard et al. · 2006 [cited by applicant]
US 20060067273A1 · Suman · 2006 [cited by applicant]
US 20060112226A1 · Hady et al. · 2006 [cited by applicant]
US 20060277421A1 · Balestriere · 2006 [cited by applicant]
US 20070030807A1 · Gummalla · 2007 [cited by applicant]
US 20070036072A1 · Raj et al. · 2007 [cited by applicant]
US 20070073966A1 · Corbin · 2007 [cited by applicant]
US 20070172235A1 · Snider et al. · 2007 [cited by applicant]
US 20070192545A1 · Gara et al. · 2007 [cited by applicant]
US 20070198656A1 · Mazzaferri et al. · 2007 [cited by applicant]
US 20070255906A1 · Handgen et al. · 2007 [cited by applicant]
US 20080002702A1 · Bajic · 2008 [cited by applicant]
US 20080138067A1 · Beshai · 2008 [cited by applicant]
US 20080244231A1 · Kunze et al. · 2008 [cited by applicant]
US 20080253294A1 · Ripa et al. · 2008 [cited by applicant]
US 20090024836A1 · Shen et al. · 2009 [cited by applicant]
US 20090046576A1 · Shand et al. · 2009 [cited by applicant]
US 20090046587A1 · Kothari · 2009 [cited by applicant]
US 20090083263A1 · Felch et al. · 2009 [cited by applicant]
US 20090135739A1 · Hoover et al. · 2009 [cited by applicant]
US 20090135832A1 · Fan et al. · 2009 [cited by applicant]
US 20090154391A1 · Wittenschlaeger · 2009 [cited by applicant]
US 20090228690A1 · Muff · 2009 [cited by applicant]
US 20090228890A1 · Vaitovirta et al. · 2009 [cited by applicant]
US 20090234987A1 · Lee et al. · 2009 [cited by applicant]
US 20090285228A1 · Bagepalli et al. · 2009 [cited by applicant]
US 20090303880A1 · Maltz et al. · 2009 [cited by applicant]
US 20090310610A1 · Sandstrom · 2009 [cited by examiner]
US 20100061257A1 · Kitawaki · 2010 [cited by applicant]
US 20100061391A1 · Sindhu et al. · 2010 [cited by applicant]
US 20100125903A1 · Devarajan et al. · 2010 [cited by applicant]
US 20100318725A1 · Kwon · 2010 [cited by applicant]
US 20110289179A1 · Pekcan et al. · 2011 [cited by applicant]
US 20110055827A1 · Lin et al. · 2011 [cited by applicant]
US 20110113184A1 · Chu · 2011 [cited by applicant]
US 20110170553A1 · Beecroft et al. · 2011 [cited by applicant]
US 20110173392A1 · Gara et al. · 2011 [cited by applicant]
US 20110202658A1 · Okuno et al. · 2011 [cited by applicant]
US 20110223783A1 · Pearson · 2011 [cited by applicant]
US 20110225594A1 · Iyengar et al. · 2011 [cited by applicant]
US 20110228783A1 · Flynn et al. · 2011 [cited by applicant]
US 20110233923A1 · Kouketsu · 2011 [cited by applicant]
US 20110238816A1 · Vohra et al. · 2011 [cited by applicant]
US 20110238923A1 · Hooker et al. · 2011 [cited by applicant]
US 20110289180A1 · Sonnier et al. · 2011 [cited by applicant]
US 20110289279A1 · Sonnier et al. · 2011 [cited by applicant]
US 20120030431A1 · Anderson et al. · 2012 [cited by applicant]
US 20120036178A1 · Gavini et al. · 2012 [cited by applicant]
US 20120076153A1 · Manzella et al. · 2012 [cited by applicant]
US 20120096211A1 · Davis et al. · 2012 [cited by applicant]
US 20120163375A1 · Sindhu et al. · 2012 [cited by applicant]
US 20120176890A1 · Balus et al. · 2012 [cited by applicant]
US 20120177047A1 · Roitshtein · 2012 [cited by applicant]
US 20120207165A1 · Davis · 2012 [cited by applicant]
US 20120254587A1 · Biran et al. · 2012 [cited by applicant]
US 20120314710A1 · Shikano · 2012 [cited by applicant]
US 20130003725A1 · Hendel et al. · 2013 [cited by applicant]
US 20130010636A1 · Regula · 2013 [cited by applicant]
US 20130024875A1 · Wang et al. · 2013 [cited by applicant]
US 20130028083A1 · Yoshida et al. · 2013 [cited by applicant]
US 20130060940A1 · Koponen et al. · 2013 [cited by applicant]
US 20130083971A1 · Du · 2013 [cited by applicant]
US 20130088971A1 · Anantharam et al. · 2013 [cited by applicant]
US 20130145375A1 · Kang · 2013 [cited by applicant]
US 20130191443A1 · Gan et al. · 2013 [cited by applicant]
US 20130258912A1 · Zimmerman et al. · 2013 [cited by applicant]
US 20130308597A1 · Murphy et al. · 2013 [cited by applicant]
US 20130330076A1 · Liboiron-Ladouceur et al. · 2013 [cited by applicant]
US 20130346789A1 · Brunel et al. · 2013 [cited by applicant]
US 20140023080A1 · Zhang et al. · 2014 [cited by applicant]
US 20140040909A1 · Winser et al. · 2014 [cited by applicant]
US 20140044128A1 · Suresh et al. · 2014 [cited by applicant]
US 20140059537A1 · Kamble et al. · 2014 [cited by applicant]
US 20140075085A1 · Schroder et al. · 2014 [cited by applicant]
US 20140161450A1 · Graves et al. · 2014 [cited by applicant]
US 20140187317A1 · Kohler et al. · 2014 [cited by applicant]
US 20140237156A1 · Regula et al. · 2014 [cited by applicant]
US 20140258479A1 · Tenginakai et al. · 2014 [cited by applicant]
US 20140269261A1 · D'souza et al. · 2014 [cited by applicant]
US 20140269351A1 · Graves et al. · 2014 [cited by applicant]
US 20140310467A1 · Shalf et al. · 2014 [cited by applicant]
US 20140359044A1 · Davis et al. · 2014 [cited by applicant]
US 20150019702A1 · Kancherla · 2015 [cited by applicant]
US 20150037032A1 · Xu et al. · 2015 [cited by applicant]
US 20150043330A1 · Hu · 2015 [cited by examiner]
US 20150117860A1 · Braun · 2015 [cited by applicant]
US 20150124826A1 · Edsall et al. · 2015 [cited by applicant]
US 20150143045A1 · Han et al. · 2015 [cited by applicant]
US 20150143073A1 · Winser et al. · 2015 [cited by applicant]
US 20150146526A1 · Kulkarni · 2015 [cited by applicant]
US 20150163171A1 · Sindhu et al. · 2015 [cited by applicant]
US 20150180603A1 · Darling et al. · 2015 [cited by applicant]
US 20150186313A1 · Sodhi et al. · 2015 [cited by applicant]
US 20150222533A1 · Birrittella et al. · 2015 [cited by applicant]
US 20150244617A1 · Nakil et al. · 2015 [cited by applicant]
US 20150256405A1 · Janardhanan et al. · 2015 [cited by applicant]
US 20150278148A1 · Sindhu · 2015 [cited by applicant]
US 20150278984A1 · Koker et al. · 2015 [cited by applicant]
US 20150280939A1 · Sindhu · 2015 [cited by examiner]
US 20150281128A1 · Sindhu · 2015 [cited by applicant]
US 20150324205A1 · Eisen et al. · 2015 [cited by applicant]
US 20150325272A1 · Murphy · 2015 [cited by applicant]
US 20150334034A1 · Smedley et al. · 2015 [cited by applicant]
US 20150334202A1 · Frydman et al. · 2015 [cited by applicant]
US 20150378776A1 · Lippett · 2015 [cited by applicant]
US 20150381528A9 · Davis et al. · 2015 [cited by applicant]
US 20160056911A1 · Ye et al. · 2016 [cited by applicant]
US 20160062800A1 · Stanfill et al. · 2016 [cited by applicant]
US 20160087885A1 · Tripathi · 2016 [cited by applicant]
US 20160092362A1 · Barron et al. · 2016 [cited by applicant]
US 20160154756A1 · Dodson et al. · 2016 [cited by applicant]
US 20160164625A1 · Gronvall et al. · 2016 [cited by applicant]
US 20160210159A1 · Wilson et al. · 2016 [cited by applicant]
US 20160212038A1 · Musiol · 2016 [cited by applicant]
US 20160239415A1 · Davis et al. · 2016 [cited by applicant]
US 20160241430A1 · Yadav et al. · 2016 [cited by applicant]
US 20160269278A1 · Gopalarathnam et al. · 2016 [cited by applicant]
US 20160337723A1 · Graves · 2016 [cited by applicant]
US 20160364333A1 · Brown et al. · 2016 [cited by applicant]
US 20160364334A1 · Asaro et al. · 2016 [cited by applicant]
US 20160380885A1 · Jani et al. · 2016 [cited by applicant]
US 20170005921A1 · Liu et al. · 2017 [cited by applicant]
US 20170031719A1 · Clark et al. · 2017 [cited by applicant]
US 20170032011A1 · Song et al. · 2017 [cited by applicant]
US 20170060615A1 · Thakkar et al. · 2017 [cited by applicant]
US 20170061566A1 · Min et al. · 2017 [cited by applicant]
US 20170068639A1 · Davis et al. · 2017 [cited by applicant]
US 20170163569A1 · Koganti · 2017 [cited by applicant]
US 20170187632A1 · Ko et al. · 2017 [cited by applicant]
US 20170235581A1 · Nickolls et al. · 2017 [cited by applicant]
US 20170265220A1 · Andreoli-Fang et al. · 2017 [cited by applicant]
US 20170279708A1 · Liu · 2017 [cited by applicant]
US 20170286143A1 · Wagner et al. · 2017 [cited by applicant]
US 20170286157A1 · Hasting et al. · 2017 [cited by applicant]
US 20170346766A1 · Dutta · 2017 [cited by applicant]
US 20180011739A1 · Pothula et al. · 2018 [cited by applicant]
US 20180024771A1 · Miller et al. · 2018 [cited by applicant]
US 20180026901A1 · Sugunadass · 2018 [cited by applicant]
US 20180115494A1 · Bhatia et al. · 2018 [cited by applicant]
US 20180152317A1 · Chang et al. · 2018 [cited by applicant]
US 20180183673A1 · Iyengar · 2018 [cited by applicant]
US 20180234300A1 · Mayya et al. · 2018 [cited by applicant]
US 20180239702A1 · Farmahini Farahani et al. · 2018 [cited by applicant]
US 20180241688A1 · Williams, Jr. · 2018 [cited by applicant]
US 20180262392A1 · White et al. · 2018 [cited by applicant]
US 20180287818A1 · Goel et al. · 2018 [cited by applicant]
US 20180287858A1 · Flajslik et al. · 2018 [cited by applicant]
US 20180287965A1 · Sindhu et al. · 2018 [cited by applicant]
US 20180288505A1 · Sindhu et al. · 2018 [cited by applicant]
US 20180293168A1 · Noureddine et al. · 2018 [cited by applicant]
US 20180300928A1 · Koker et al. · 2018 [cited by applicant]
US 20180307494A1 · Ould-Ahmed-Vall et al. · 2018 [cited by applicant]
US 20180307535A1 · Suzuki et al. · 2018 [cited by applicant]
US 20180322386A1 · Sridharan et al. · 2018 [cited by applicant]
US 20180357169A1 · Lai · 2018 [cited by applicant]
US 20190005176A1 · Illikkal et al. · 2019 [cited by applicant]
US 20190012278A1 · Sindhu et al. · 2019 [cited by applicant]
US 20190012350A1 · Sindhu et al. · 2019 [cited by applicant]
US 20190013965A1 · Sindhu et al. · 2019 [cited by applicant]
US 20190018806A1 · Koufaty et al. · 2019 [cited by applicant]
US 20190042292A1 · Palermo et al. · 2019 [cited by applicant]
US 20190042518A1 · Marolia et al. · 2019 [cited by applicant]
US 20190095333A1 · Heirman et al. · 2019 [cited by applicant]
US 20190102311A1 · Gupta et al. · 2019 [cited by applicant]
US 20190104057A1 · Goel et al. · 2019 [cited by applicant]
US 20190104206A1 · Goel et al. · 2019 [cited by applicant]
US 20190104207A1 · Goel et al. · 2019 [cited by applicant]
US 20190158428A1 · Gray et al. · 2019 [cited by applicant]
US 20190188079A1 · Kohli · 2019 [cited by applicant]
US 20190243765A1 · Sindhu et al. · 2019 [cited by applicant]
US 20190319871A1 · Indiresan et al. · 2019 [cited by applicant]
US 20190363989A1 · Shalev et al. · 2019 [cited by applicant]
US 20200021664A1 · Goyal et al. · 2020 [cited by applicant]
US 20200021898A1 · Sindhu et al. · 2020 [cited by applicant]
US 20200067839A1 · Iny et al. · 2020 [cited by applicant]
US 20200084155A1 · Song et al. · 2020 [cited by applicant]
US 20200119903A1 · Thomas et al. · 2020 [cited by applicant]
US 20200133771A1 · Goyal et al. · 2020 [cited by applicant]
US 20200145680A1 · Dikshit et al. · 2020 [cited by applicant]
US 20200151101A1 · Noureddine et al. · 2020 [cited by applicant]
US 20200159568A1 · Goyal et al. · 2020 [cited by applicant]
US 20200159859A1 · Beckman et al. · 2020 [cited by applicant]
US 20200162282A1 · Ashtaputre et al. · 2020 [cited by applicant]
US 20200169513A1 · Goel et al. · 2020 [cited by applicant]
US 20200183841A1 · Noureddine et al. · 2020 [cited by applicant]
US 20200236052A1 · Srinivasan · 2020 [cited by applicant]
US 20200259682A1 · Goel et al. · 2020 [cited by applicant]
US 20200280462A1 · Sindhu et al. · 2020 [cited by applicant]
US 20200314026A1 · Sindhu et al. · 2020 [cited by applicant]
US 20200356414A1 · Sindhu et al. · 2020 [cited by applicant]
US 20210097108A1 · Goyal et al. · 2021 [cited by applicant]
US 20210176347A1 · Goel et al. · 2021 [cited by applicant]
US 20210218665A1 · Goel et al. · 2021 [cited by applicant]
US 20210297343A1 · Vegesna et al. · 2021 [cited by applicant]
US 20210297350A1 · Vegesna et al. · 2021 [cited by applicant]
US 20210297351A1 · Vegesna et al. · 2021 [cited by applicant]
US 20210320820A1 · Ruan et al. · 2021 [cited by applicant]
US 20210399941A1 · Fang · 2021 [cited by applicant]
US 20220103661A1 · Goel et al. · 2022 [cited by applicant]
US 20230208748A1 · Goel · 2023 [cited by applicant]
US 20230269184A1 · Zemach · 2023 [cited by applicant]
US 20230388222A1 · Sindhu · 2023 [cited by applicant]
CN 102123052A · 2011 [cited by applicant]
CN 103067291A · 2013 [cited by applicant]
CN 104052618A · 2014 [cited by applicant]
CN 104521196A · 2015 [cited by applicant]
CN 104937892A · 2015 [cited by applicant]
CN 104954251A · 2015 [cited by applicant]
CN 105024844A · 2015 [cited by applicant]
CN 105814848A · 2016 [cited by applicant]
CN 107196854A · 2017 [cited by applicant]
EP 1079571A2 · 2001 [cited by applicant]
EP 1489796A2 · 2004 [cited by applicant]
EP 1501246A2 · 2005 [cited by applicant]
EP 2289206A2 · 2011 [cited by applicant]
EP 2928134A2 · 2015 [cited by applicant]
WO 2009114554A2 · 2009 [cited by applicant]
WO 2013184846A1 · 2013 [cited by applicant]
WO 2014178854 · 2014 [cited by applicant]
WO 2015087474A1 · 2015 [cited by applicant]
WO 2016037262A1 · 2016 [cited by applicant]
WO 2019014268A1 · 2019 [cited by applicant]
Notice of Allowance from U.S. Appl. No. 16/901,991 dated Feb. 10, 2023, 7 pp. [cited by applicant]
Response to Office Action dated Oct. 24, 2022 from U.S. Appl. No. 16/901,991, filed Jan. 24, 2023, 9 pp. [cited by applicant]
Zhang et al., “A LDP Fast Protection Switching Scheme for Concurrent Multiple Failures in MPLS Network,” IEEE, 2009 International Conference on Multimedia Information Networking and Security, Nov. 18-20, 2009, pp. 259-2… [cited by applicant]
Advisory Action from U.S. Appl. No. 16/901,991 dated Jun. 27, 2022, 5 pp. [cited by applicant]
Response to Final Office Action dated Mar. 28, 2022 from U.S. Appl. No. 16/901,991, filed Jun. 28, 2022, 10 pp. [cited by applicant]
Response to Final Office Action dated Mar. 28, 2022, from U.S. Appl. No. 16/901,991, filed May 31, 2022, 9 pp. [cited by applicant]
“Transmission Control Protocol,” Darpa Internet Program Protocol Specification, Information Sciences Institute, RFC 793, Sep. 1981, 91 pp. [cited by applicant]
Alizadeh et al., “Analysis of DCTCP: Stability, Convergence, and Fairness,” Department of Electrical Engineering, Standford University, SIGMETRICS'11, Jun. 7-11, 2011, 14 pp. [cited by applicant]
Allman et al., “TCP Congestion Control”, IETF Network Working Group, RFC 5681, Sep. 2009, 18 pp. [cited by applicant]
Bensley et al., “Data Center TCP (DCTCP): TCP Congestion Control for Data Centers”, IETF, RFC 8257, Oct. 2017, 17 pp. [cited by applicant]
Cardwell et al., “BBR Congestion Control”, IETF Draft, Jul. 3, 2017, 34 pp. [cited by applicant]
Cardwell et al., “BBR Congestion Control”, IETF, Nov. 2016, 37 pp. [cited by applicant]
Chen et al., “Data Center Congestion Management requirements; draft-yueven-tsvwg-dccm-requirements-01,” TSVWG, Internet-Draft, Retrieved Jul. 7, 2021 from: https://tools.ietf.org/id/draft-yueven-tsvwg-dccm-requirements-… [cited by applicant]
Jacobson et al., “TCP Extensions for High Performance”, IETF Network Working Group, RFC 1323, May 1992, 37 pp. [cited by applicant]
Mathis et al., “TCP Selective Acknowledgment Options”, IETF Network Working Group, RFC 2018, Oct. 1996, 12 pp. [cited by applicant]
Mittal et al., “Revisiting Network Support for RDMA”, Department of Electrical Engineering, Standford University, SIGMETRICS, Jun. 21, 2018, 18 pp. [cited by applicant]
Paxson et al.., “Known TCP Implementation Problems”, IETF Network Working Group, RFC 2525, Mar. 1999, 61 pp. [cited by applicant]
Stevens, “TCP Slow Start, Congestion Avoidance, Fast Retransmit and Fast Recovery Algorithms”, IETF Network Working Group, RFC 2001, Jan. 1997, 6 pp. [cited by applicant]
Final Office Action from U.S. Appl. No. 16/901,991, dated Mar. 28, 2022, 11 pp. [cited by applicant]
Response to Office Action dated Nov. 3, 2021, from U.S. Appl. No. 16/901,991, filed Feb. 2, 2022, 8 pp. [cited by applicant]
Response to Office Action dated Mar. 28, 2022 from U.S. Appl. No. 16/901,991, filed May 31, 2022, 9 pp. [cited by applicant]
U.S. Appl. No. 17/303,887, filed Jun. 9, 2021, naming inventors Vegesna et al. [cited by applicant]
U.S. Appl. No. 17/304,654, filed Jun. 24, 2021, naming inventors Ruan et al. [cited by applicant]
U.S. Appl. No. 17/303,900, filed Jun. 10, 2021, naming inventors Vegesna et al. [cited by applicant]
U.S. Appl. No. 17/303,901, filed Jun. 10, 2021, naming inventors Vegesna et al. [cited by applicant]
Office Action from U.S. Appl. No. 16/901,991 dated Oct. 24, 2022, 10 pp. [cited by applicant]
“Non-Final Office Action Issued in U.S. Appl. No. 17/454,731”,dated Jul. 20, 2023, 39 Pages. [cited by applicant]
“Non-Final Office Action Issued in U.S. Appl. No. 18/175,376”, dated Aug. 3, 2023, 22 Pages. [cited by applicant]
“Notice of Allowance Issued in U.S. Appl. No. 17/301,185”, dated Oct. 11, 2022, 9 Pages. [cited by applicant]
“QFX10000 Switches System Architecture,” White Paper, Juniper Networks, Apr. 2015, 15 pp. [cited by applicant]
Adya et al., “Cooperative Task Management without Manual Stack Management,” Proceedings of the 2002 Usenix Annual Technical Conference, Jun. 2002, 14 pp. [cited by applicant]
Al-Fares et al., “Hedera: Dynamic Flow Scheduling for Data Center Networks,” NSDI'10 Proceedings of the 7th USENIX Conference on Networked Systems Design and Implementation, Apr. 28-30, 2010, 15 pp. [cited by applicant]
Alizadeh et al., “Conga: Distributed Congestion-Aware Load Balancing for Datacenters,” SIGCOMM '14 Proceedings of the 2014 ACM Conference on SIGCOMM, Aug. 17-22, 2014, pp. 503-514. [cited by applicant]
Bakkum et al., “Accelerating SQL Database Operations on a GPU with CUDA,” Proceedings of the 3rd Workshop on Genral-Purpose Computation on Graphics Processing Units, Mar. 14, 2010, 10 pp. [cited by applicant]
Banga et al., “Better operating system features for faster network servers,” ACM Sigmetrics Performance Evaluation Review, vol. 26, Issue 3, Dec. 1998, 11 pp. [cited by applicant]
Barroso et al., “Attack of the killer Microseconds,” Communications of the ACM, vol. 60, No. 4, Apr. 2017, 7 pp. [cited by applicant]
Benson et al., “MicroTE: Fine Grained Traffic Engineering for Data Centers,” CoNEXT '11 Proceedings of the Seventh Conference on emerging Networking Experiments and Technologies Article No. 8, Dec. 6-9, 2011, 12 pp. [cited by applicant]
Benson et al., “Network Traffic Characteristics of Data Centers in the Wild,” IMC '10 Proceedings of the 10th ACM SIGCOMM Conference on Internet Measurement, Nov. 1-30, 2010, pp. 267-280. [cited by applicant]
Ford et al., “TCP Extensions for Multipath Operation with Multiple Addresses,” Internet Engineering Task Force (IETF), RFC 6824, Jan. 2013, 64 pp. [cited by applicant]
Friedman et al., “Programming with Continuations,” Technical Report 151, Nov. 1983, 14 pp. [cited by applicant]
Gay et al., “The nesC Language: A Holistic Approach to Networked Embedded Systems,” accessed from http://nescc.sourceforge.net, last updated Dec. 14, 2004, 11 pp. [cited by applicant]
Halbwachs et al., “The Synchronous Data Flow Programming Language LUSTRE,” Proceedings of the IEEE, vol. 79, No. 9, Sep. 1991, 16 pp. [cited by applicant]
Haynes et al., “Continuations and Coroutines,” Technical Report No. 158, Jun. 1984, 19 pp. [cited by applicant]
Hewitt, “Viewing Control Structures as Patterns of Passing Messages,” Massachusetts Institute of Technology, Artificial Intelligence Laboratory, Dec. 1976, 61 pp. [cited by applicant]
Hseush et al., Data Path Debugging: Data-Oriented Debugging for a Concurrent Programming Language, PADD 88 Proceedings of the 1988 ACM SIGPLAN and SIGOPS workshop on Parallel and distributed debugging, May 5-6, 1988, 12… [cited by applicant]
Huang et al., “Erasure Coding in Windows Azure Storage,” 2012 USENIX Annual Technical Conference, Jun. 13-15, 2012, 12 pp. [cited by applicant]
Isen et al., “ESKIMO—Energy Savings using Semantic Knowledge of Inconsequential Memory Occupancy for DRAM subsystem,” 42nd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), Dec. 12-16, 2009, 10 pp. [cited by applicant]
Kahn et al., “Actors as a Special Case of Concurrent Constraint Programming,” ECOOP/OOPSLA '90 Proceedings, Oct. 21-25, 1990, 10 pp. [cited by applicant]
Kaminow, “Optical Integrated Circuits: A Personal Perspective,” Journal of Lightwave Technology, vol. 26, No. 9, May 1, 2008, pp. 994-1004. [cited by applicant]
Kandula et al., “Dynamic Load Balancing Without Packet Reordering,” SIGCOMM Computer Communication Review, vol. 37, No. 2, Apr. 2007, pp. 53-62. [cited by applicant]
Kandula et al., “The Nature of Datacenter Traffic: Measurements & Analysis,” IMC '09 Proceedings of the 9th ACM SIGCOMM conference on Internet measurement, Nov. 4-6, 2009, pp. 202-208. [cited by applicant]
Kelly et al., A Block Diagram Compiler, the Bell System Technical Journal, Dec. 7, 1960, 10 pp. [cited by applicant]
Kounavis et al., “Programming the data path in network processor-based routers,” Software—Practice and Experience, Oct. 21, 2003, 38 pp. [cited by applicant]
Larus et al., “Using Cohort Scheduling to Enhance Server Performance,” Usenix Annual Technical Conference, Jun. 2002, 12 pp. [cited by applicant]
Levis et al., “Tiny OS: An Operating System for Sensor Networks,” Ambient Intelligence, Jan. 2005, 34 pp. [cited by applicant]
Lin et al., A Parameterized Dataflow Language Extension for Embedded Streaming Systems, 2008 International Conference on Embedded Computer Systems: Architectures, Modeling, and Simulation, Jul. 21-24, 2008, 8 pp. [cited by applicant]
Mishra et al., “Thread-based vs Event-based Implementation of a Group Communication Service,” Proceedings of the First Merged International Parallel Processing Symposium and Symposium on Parallel and Distributed Process… [cited by applicant]
Raiciu et al., “Improving Datacenter Performance and Robustness with Multipath TCP,” ACM SIGCOMM Computer Communication Review—SIGCOMM '11, vol. 41, No. 4, Aug. 2011, pp. 266-277. [cited by applicant]
Schroeder et al., “Flash Reliability in Production: The Expected and the Unexpected,” 14th USENIX Conference on File and Storage Technologies (FAST '16), Feb. 22-25, 2016, 15 pp. [cited by applicant]
Varela et al., “The Salsa Programming Language 2.0.0alpha Release Tutorial,” Tensselaer Polytechnic Institute, Nov. 2009, 52 pp. [cited by applicant]
Von Behren et al., “Why Events Are a Bad Idea (for high-concurrency servers),” Proceedings of HotOS IX, May 2003, 6 pp. [cited by applicant]
Welsh et al., “SEDA: An Architecture for Well-Conditioned, Scalable Internet Services,” Eighteenth Symposium on Operating Systems Principles, Oct. 21-24, 2001, 14 pp. [cited by applicant]
Cited By (4)
US 12,407,621 US 12,463,917 US 12,556,474 US 12,556,489