IP Library › Granted Patent US 12,574,339
Granted Patent B2
US 12,574,339 · App. 18/595,195 · Granted Mar 10, 2026

Fabric control protocol for data center networks with packet spraying over multiple alternate data paths

Inventors: Deepak Goel (San Jose, CA); Narendra Jayawant Gathoo (San Jose, CA); Philip A. Thomas (San Jose, CA); Srihari Raju Vegesna (San Jose, CA); Pradeep Sindhu (Los Altos Hills, CA); Wael Noureddine (Santa Clara, CA); Robert William Bowdidge (San Jose, CA); Ayaskant Pani (Fremont, CA); Gopesh Goyal (Cupertino, CA)
Assignee: Microsoft Technology Licensing, LLC
H04L49/25H04L12/4633H04L45/243H04L47/34H04L47/41H04L67/10H04L67/1097H04L69/16
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,574,339
App. No.
18/595,195
Granted
Mar 10, 2026
Kind
B2
Abstract

A fabric control protocol is described for use within a data center in which a switch fabric provides full mesh interconnectivity such that any of the servers may communicate packet data for a given packet flow to any other of the servers using any of a number of parallel data paths within the data center switch fabric. The fabric control protocol enables spraying of individual packets for a given packet flow across some or all of the multiple parallel data paths in the data center switch fabric and, optionally, reordering of the packets for delivery to the destination. The fabric control protocol may provide end-to-end bandwidth scaling and flow fairness within a single tunnel based on endpoint-controlled requests and grants for flows. In some examples, the fabric control protocol packet structure is carried over an underlying protocol, such as the User Datagram Protocol (UDP).

Claims (47)

1 . A system comprising:

a source access node coupled to a source server, wherein the source access node communicates with a destination access node coupled to a destination server over a plurality of data paths through a switch fabric, wherein the source access node is configured to spray packets of a data flow of packets between the source server and the destination server over the plurality of data paths, and wherein to spray the packets of the data flow, the source access node:

tracks a number of bytes previously transmitted on each of the plurality of data paths by the source access node;

determines, for each of the plurality of data paths, a count of bytes previously transmitted on each respective path; and

directs each of the packets within the data flow to a data path of the plurality of data paths selected based on the count of bytes previously transmitted on each respective path, wherein a first packet and a second packet of the data flow travel over different data paths of the plurality of data paths.

2 . The system of claim 1 , wherein the source access node is further configured to:

insert a sequence number into each of the packets of the data flow.

3 . The system of claim 2 , wherein sequence numbers within the packets of the data flow can be used by the destination access node to reorder the packets.

4 . The system of claim 1 , wherein the source access node is further configured to:

identify a failed data path from among the plurality of data paths.

5 . The system of claim 4 , wherein to spray the data flow of packets between the source server and the destination server over the plurality of data paths, the source access node is further configured to:

direct each of the packets within the data flow to a data path selected further based on information about the failed data path.

6 . The system of claim 5 , wherein to spray the data flow of packets between the source server and the destination server over the plurality of data paths, the source access node is further configured to:

refrain from transmitting packets over the failed data path.

7 . The system of claim 1 , wherein to spray the data flow of packets between the source server and the destination server over the plurality of data paths, the source access node is further configured to:

direct each of the packets within the data flow to a data path selected further based on a bandwidth value associated with the data path.

8 . The system of claim 1 , wherein the source access node is further configured to:

determine, for each of the plurality of data paths, a weighted usage value, wherein the weighted usage value is a proportion of an amount of data transmitted over one of the plurality of data paths to an amount of data transmitted over all of the plurality of data paths.

9 . The system of claim 8 , wherein to spray the data flow of packets between the source server and the destination server over the plurality of data paths, the source access node is further configured to:

direct each of the packets within the data flow to a data path based on the weighted usage value for each of the plurality of data paths.

10 . The system of claim 9 , wherein to direct each of the packets within the data flow to a data path based on the weighted usage value for each of the plurality of data paths, the source access node is further configured to:

choose the data path having a smallest weighted usage value.

11 . Enacted on a source access node, a method comprising:

spraying, over a plurality of data paths through a switch fabric, packets of a data flow between a source server and a destination server by:

tracking a number of bytes of the packets of the data flow previously transmitted on each of the plurality of data paths by the source access node;

determining, for each of the plurality of data paths, a count of bytes previously transmitted on each respective path; and

directing each of the packets of the data flow to a data path selected based on the count of bytes previously transmitted on each respective path, wherein a first packet and a second packet of the packets of the data flow travel over different data paths of the plurality of data paths.

12 . The method of claim 11 , further comprising:

inserting a sequence number into each of the packets of the data flow.

13 . The method of claim 12 , wherein sequence numbers within the packets of the data flow can be used by a destination access node of the destination server to reorder the packets.

14 . The method of claim 11 , further comprising:

identifying a failed data path from among the plurality of data paths.

15 . The method of claim 14 , wherein spraying the data flow of packets between the source server and the destination server further includes:

directing each of the packets within the data flow to a data path selected further based on information about the failed data path.

16 . The method of claim 15 , wherein directing each of the packets within the data flow to a data path selected further based on information about the failed data path includes:

refraining from transmitting packets over the failed data path.

17 . The method of claim 11 , wherein spraying the data flow of packets between the source server and the destination server further includes:

directing each of the packets within the data flow to a data path selected further based on a bandwidth value associated with the data path.

18 . The method of claim 11 , further comprising:

determining, for each of the plurality of data paths, a weighted usage value, wherein the weighted usage value is a proportion of an amount of data transmitted over one of the plurality of data paths to an amount of data transmitted over all of the plurality of data paths.

19 . The method of claim 18 , wherein spraying the data flow of packets between the source server and the destination server further includes:

directing each of the packets within the data flow to a data path based on the weighted usage value for each of the plurality of data paths.

20 . A source server comprising:

a plurality of source access nodes, wherein each source access node communicates with a destination server over a plurality of data paths through a switch fabric, and wherein each source access node is configured to spray packets of a data flow of packets between the source server and the destination server over the plurality of data paths by:

tracking a number of bytes of the packets of the data flow previously transmitted on each of the plurality of data paths;

determining, for each of the plurality of data paths, a count of bytes previously transmitted on each respective path; and

directing each of the packets of the data flow to data path selected based on the count of bytes previously transmitted on each respective path, wherein a first packet and a second packet of the packets of the data flow travel over different data paths of the plurality of data paths.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 5, 2024
From: GOEL, DEEPAK; GATHOO, NARENDRA JAYAWANT; THOMAS, PHILIP A.; VEGESNA, SRIHARI RAJU; SINDHU, PRADEEP; NOUREDDINE, WAEL; BOWDIDGE, ROBERT WILLIAM; PANI, AYASKANT; GOYAL, GOPESH
To: FUNGIBLE, INC.
Reel/Frame 066658/0557 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 5, 2024
From: FUNGIBLE, INC.
To: MICROSOFT TECHNOLOGY LICENSING, LLC
Reel/Frame 066658/0567 →
Continuity (5)
Continuation 16774941 · Jan 28, 2020
Continuation In Part 16147070 · Sep 28, 2018
Provisional Application 62638788 · Mar 5, 2018
Provisional Application 62566060 · Sep 29, 2017
Related Publication 20240250919A1 · Jul 25, 2024
References Cited (86)
US 6526022B1 · Chiu · 2003 [cited by applicant]
US 7802001B1 · Petry et al. · 2010 [cited by applicant]
US 8565218B2 · Sharma · 2013 [cited by applicant]
US 8811183B1 · Anand · 2014 [cited by applicant]
US 8848728B1 · Revah et al. · 2014 [cited by applicant]
US 8854953B2 · Enyedi · 2014 [cited by applicant]
US 8937865B1 · Kumar · 2015 [cited by applicant]
US 8953441B2 · Nakil · 2015 [cited by applicant]
US 9294398B2 · DeCusatis et al. · 2016 [cited by applicant]
US 9866427B2 · Yadav et al. · 2018 [cited by applicant]
US 10929175B2 · Goyal et al. · 2021 [cited by applicant]
US 11048634B2 · Noureddine et al. · 2021 [cited by applicant]
US 11070474B1 · Jain · 2021 [cited by applicant]
US 11934964B2 · Goyal · 2024 [cited by applicant]
US 20050013311A1 · Samudrala et al. · 2005 [cited by applicant]
US 20050100035A1 · Chiou et al. · 2005 [cited by applicant]
US 20060067273A1 · Suman · 2006 [cited by applicant]
US 20070030807A1 · Gummalla · 2007 [cited by applicant]
US 20090046587A1 · Kothari · 2009 [cited by applicant]
US 20090228690A1 · Muff · 2009 [cited by applicant]
US 20090310610A1 · Sandstrom · 2009 [cited by examiner]
US 20100061257A1 · Kitawaki · 2010 [cited by applicant]
US 20100061391A1 · Sindhu et al. · 2010 [cited by applicant]
US 20130258912A1 · Zimmerman et al. · 2013 [cited by applicant]
US 20130308597A1 · Murphy et al. · 2013 [cited by applicant]
US 20140237156A1 · Regula et al. · 2014 [cited by applicant]
US 20140258479A1 · Tenginakai · 2014 [cited by applicant]
US 20150043330A1 · Hu et al. · 2015 [cited by applicant]
US 20150146526A1 · Kulkarni · 2015 [cited by applicant]
US 20150244617A1 · Nakil et al. · 2015 [cited by applicant]
US 20150280939A1 · Sindhu · 2015 [cited by applicant]
US 20160087885A1 · Tripathi · 2016 [cited by applicant]
US 20160154756A1 · Dodson et al. · 2016 [cited by applicant]
US 20160212038A1 · Musiol · 2016 [cited by applicant]
US 20160241430A1 · Yadav et al. · 2016 [cited by applicant]
US 20170032011A1 · Song et al. · 2017 [cited by applicant]
US 20170279708A1 · Liu · 2017 [cited by applicant]
US 20180183673A1 · Iyengar · 2018 [cited by applicant]
US 20180241688A1 · Williams, Jr. · 2018 [cited by applicant]
US 20180262392A1 · White et al. · 2018 [cited by applicant]
US 20180278550A1 · Rosen · 2018 [cited by examiner]
US 20190116160A1 · Bhat · 2019 [cited by examiner]
US 20190199636A1 · Narayanan · 2019 [cited by examiner]
US 20200084155A1 · Song · 2020 [cited by applicant]
US 20200162282A1 · Ashtaputre et al. · 2020 [cited by applicant]
US 20200236052A1 · Srinivasan · 2020 [cited by applicant]
US 20210297351A1 · Vegesna et al. · 2021 [cited by applicant]
US 20210320820A1 · Ruan et al. · 2021 [cited by applicant]
US 20210399941A1 · Fang · 2021 [cited by applicant]
US 20230269184A1 · Zemach · 2023 [cited by applicant]
US 20230388222A1 · Sindhu · 2023 [cited by applicant]
CN 104521196A · 2015 [cited by applicant]
EP 1501246A2 · 2005 [cited by applicant]
Non Final office mailed on Oct. 10, 2024 in U.S. Appl. No. 17/303,900 19 Pages. [cited by applicant]
Notice of Allowance mailed on Dec. 11, 2024, in U.S. Appl. No. 17/303,901, 09 Pages. [cited by applicant]
Notice of Allowance mailed on Oct. 1, 2024, in U.S. Appl. No. 16/774,941, 7 Pages. [cited by applicant]
Notice of Allowance mailed on Oct. 28, 2024, in U.S. Appl. No. 17/454,731, 9 pages. [cited by applicant]
China National Intellectual Property Administration, First Office Action and Search Report mailed on Aug. 18, 2021, in Chinese Application No. 201880063473.2, 8 pages. [cited by applicant]
Non-Final Office Action mailed on Apr. 9, 2020, in U.S. Appl. No. 16/147,134, 20 pages. [cited by applicant]
Non-Final Office Action mailed on Apr. 26, 2022, in U.S. Appl. No. 17/301,185, 20 pages. [cited by applicant]
Non-Final Office Action mailed on Aug. 17, 2020, in U.S. Appl. No. 16/147,134, 18 pages. [cited by applicant]
Non-Final Office Action mailed on Dec. 9, 2021, in U.S. Appl. No. 17/248,354, 11pages. [cited by applicant]
Non-Final Office Action mailed on May 28, 2020, in U.S. Appl. No. 16/147,099, 10 pages. [cited by applicant]
Notice of Allowance mailed on Apr. 12, 2022, in U.S. Appl. No. 17/248,354, 9 pages. [cited by applicant]
Notice of Allowance mailed on Dec. 4, 2020, in U.S. Appl. No. 16/147,134, 9 pages. [cited by applicant]
Notice of Allowance mailed on Dec. 18, 2023, in U.S. Appl. No. 17/454,731, 9 Pages. [cited by applicant]
Notice of Allowance mailed on Feb. 26, 2025, in U.S. Appl. No. 17/303,900, 6 pages. [cited by applicant]
Notice of Allowance mailed on Jun. 17, 2021, in U.S. Appl. No. 16/147,070, 6 pages. [cited by applicant]
Notice of Allowance mailed on Sep. 23, 2021, U.S. Appl. No. 16/147,070, 11 pages. [cited by applicant]
Notice of Allowance mailed on Sep. 25, 2020, in U.S. Appl. No. 16/147,099, 9 pages. [cited by applicant]
Non-Final Office Action mailed on Feb. 13, 2024, in U.S. Appl. No. 17/303,887, 21 pages. [cited by applicant]
Notice of Allowance mailed on Mar. 18, 2024, in U.S. Appl. No. 18/175,376, 9 pages. [cited by applicant]
Alizadeh et al., “CONGA: Distributed Congestion-Aware Load Balancing for Datacenters,” SIGCOMM '14 Proceedings of the 2014 ACM Conference on SIGCOMM, Aug. 17-22, 2014, pp. 503-514. [cited by applicant]
First Office Action and Search Report, and translation thereof, from counterpart Chinese Application No. 201880062872.7 dated Oct. 8, 2021, 11 pp. [cited by applicant]
International Search Report and Written Opinion of International Application No. PCT/US2018/053586, mailed Jan. 17, 2019, 16 pages. [cited by applicant]
International Search Report and Written Opinion of International Application No. PCT/US2018/053591, mailed Jan. 17, 2019, 15 pp. [cited by applicant]
International Search Report and Written Opinion of International Application No. PCT/US2018/053597, mailed Jan. 17, 2019, 18 pages. [cited by applicant]
Notice of Allowance mailed on Aug. 2, 2024, in U.S. Appl. No. 17/303,901, 10 pages. [cited by applicant]
Notice of Allowance mailed on Jul. 15, 2024, in U.S. Appl. No. 18/175,376, 9 pages. [cited by applicant]
Notice of Allowance mailed on Jun. 12, 2024, in U.S. Appl. No. 17/303,887, 9 pages. [cited by applicant]
Notice of Allowance mailed on Jun. 17, 2024, in U.S. Appl. No. 16/774,941, 09 pages. [cited by applicant]
Notice of Allowance mailed on Jun. 26, 2024, in U.S. Appl. No. 17/454,731, 9 pages. [cited by applicant]
Oh et al., “Feedback-Based Path Failure Detection and Buffer Blocking Protection for MPTCP,” IEEE/ACM Transactions on Networking, vol. 24, No. 6, Dec. 2016, 12 pp. [cited by applicant]
Raiciu et al., “Improving Datacenter Performance and Robustness with Multipath TCP,” ACM SIGCOMM Computer Communication Review—SIGCOMM '11, vol. 41, No. 4, Aug. 2011, pp. 266-277. [cited by applicant]
Non-Final Office Action mailed on Apr. 5, 2024, in U.S. Appl. No. 17/304,654, 19 pages. [cited by applicant]
Non-Final Office action mailed on Mar. 21, 2024, in U.S. Appl. No. 17/303,901, 26 pages. [cited by applicant]