IP Library › Granted Patent US 12,375,408
Granted Patent B2
US 12,375,408 · App. 18/621,516 · Granted Jul 29, 2025

Dynamic load balancing for multi-core computing environments

Inventors: Stephen Palermo (Chandler, AZ); Bradley Chaddick (Portland, OR); Gage Eads (Austin, TX); Mrittika Ganguli (Tempe, AZ); Abhishek Khade (Chandler, AZ); Abhirupa Layek (Chandler, AZ); Sarita Maini (Tempe, AZ); Niall McDonnell (Limerick, IE); Rahul Shah (Chandler, AZ); Shrikant Shah (Chandler, AZ); William Burroughs (Macungie, PA); David Sonnier (Austin, TX)
Assignee: Intel Corporation
H04L47/125H04L47/62H04L47/624H04L47/6255H04L47/6275
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,375,408
App. No.
18/621,516
Filed
Mar 29, 2024
Granted
Jul 29, 2025
Kind
B2
Art Unit
2471
USPC
370/235
Abstract

Methods, apparatus, systems, and articles of manufacture are disclosed for dynamic load balancing for multi-core computing environments. An example apparatus includes a first and a plurality of second cores of a processor, and circuitry in a die of the processor separate from the first and the second cores, the circuitry to enqueue identifiers in one or more queues in the circuitry associated with respective ones of data packets of a packet flow, allocate one or more of the second cores to dequeue first ones of the identifiers in response to a throughput parameter of the first core not satisfying a throughput threshold to cause the one or more of the second cores to execute one or more operations on first ones of the data packets, and provide the first ones to one or more data consumers to distribute the first data packets.

Claims (31)

1. At least one non-transitory computer readable medium comprising instructions to cause at least one programmable circuit in a semiconductor die to at least:

identify first packets that belong to an elephant packet flow;

queue entries associated with the first packets among different queues of a plurality of queues to distribute the first packets to different cores of a plurality of processor cores in the semiconductor die, the at least one programmable circuit separate from the plurality of processor cores, the plurality of queues respectively associated with the plurality of processor cores; and

queue entries associated with second packets different from the first packets in the queues such that packets of the second packets belonging to a same flow are queued to a same queue of the queues.

2. The at least one non-transitory computer readable medium of claim 1 , wherein the different cores are to process the first packets to produce processed first packets, and the instructions are to cause one or more of the at least one programmable circuit to re-order the processed first packets to produce a processed elephant packet flow.

3. The at least one non-transitory computer readable medium of claim 2 , wherein the first packets are from a first network interface, and the instructions are to cause one or more of the at least one programmable circuit to provide the processed elephant packet flow to a second network interface.

4. The at least one non-transitory computer readable medium of claim 1 , wherein the instructions are to cause one or more of the at least one programmable circuit to identify an input packet flow as the elephant packet flow based on a duration of the input packet flow.

5. The at least one non-transitory computer readable medium of claim 1 , wherein the instructions are to cause one or more of the at least one programmable circuit to identify an input packet flow as the elephant packet flow based on a bandwidth associated with the input packet flow.

6. The at least one non-transitory computer readable medium of claim 1 , wherein the entries associated with the first packets include pointers to the first packets.

7. A semiconductor die comprising:

a plurality of processor cores; and

circuitry in the semiconductor die, the circuitry separate from the plurality of processor cores, the circuitry to:

identify first packets that belong to an elephant packet flow;

queue entries associated with the first packets among different queues of a plurality of queues to distribute the first packets to different cores of the plurality of processor cores, the plurality of queues respectively associated with the plurality of processor cores; and

queue entries associated with second packets different from the first packets in the queues such that packets of the second packets belonging to a same flow are queued to a same queue.

8. The semiconductor die of claim 7 , wherein the different cores are to process the first packets to produce processed first packets, and the circuitry is to cause the processed first packets to be re-ordered to produce a processed elephant packet flow.

9. The semiconductor die of claim 8 , wherein the first packets are from a first network interface, and the circuitry is to cause the processed elephant packet flow to be provided to a second network interface.

10. The semiconductor die of claim 7 , wherein the circuitry is to identify an input packet flow as the elephant packet flow based on a duration of the input packet flow.

11. The semiconductor die of claim 7 , wherein the circuitry is to identify an input packet flow as the elephant packet flow based on a bandwidth associated with the input packet flow.

12. The semiconductor die of claim 7 , wherein the entries associated with the first packets include pointers to the first packets.

13. The semiconductor die of claim 7 , wherein the different cores are to perform cryptographic operations on the first packets.

14. The semiconductor die of claim 7 , wherein the different cores are to perform firewall operations on the first packets.

15. A method comprising:

identifying first packets that belong to an elephant packet flow;

queuing, with circuitry in a semiconductor die, entries associated with the first packets among different queues of a plurality of queues to distribute the first packets to different cores of a plurality of processor cores in the semiconductor die, the circuitry separate from the plurality of processor cores, the plurality of queues respectively associated with the plurality of processor cores; and

queuing, with the circuitry, entries associated with second packets different from the first packets in the queues such that packets of the second packets belonging to a same flow are queued to a same queue of the queues.

16. The method of claim 15 , wherein the different cores are to process the first packets to produce processed first packets, and including re-ordering the processed first packets to produce a processed elephant packet flow.

17. The method of claim 16 , wherein the first packets are from a first network interface, and including providing the processed elephant packet flow to a second network interface.

18. The method of claim 15 , further including identifying an input packet flow as the elephant packet flow based on a duration of the input packet flow.

19. The method of claim 15 , including identifying an input packet flow as the elephant packet flow based on a bandwidth associated with the input packet flow.

20. The method of claim 15 , wherein the entries associated with the first packets include pointers to the first packets.

Continuity (5)
Continuation 18154619 · Jan 13, 2023
Continuation 17018809 · Sep 11, 2020
Provisional Application 62979963 · Feb 21, 2020
Provisional Application 62899061 · Sep 11, 2019
Related Publication 20240267334A1 · Aug 8, 2024
References Cited (162)
US 7289442B1 · Srinivasan et al. · 2007 [cited by applicant]
US 8327187B1 · Metcalf · 2012 [cited by examiner]
US 9178815B2 · Gasparakis · 2015 [cited by examiner]
US 9313047B2 · Michels et al. · 2016 [cited by applicant]
US 9432298B1 · Smith · 2016 [cited by applicant]
US 9485326B1 · Ulman et al. · 2016 [cited by applicant]
US 10025619B1 · Felter · 2018 [cited by examiner]
US 10469404B1 · Mao et al. · 2019 [cited by applicant]
US 10505849B1 · Iny · 2019 [cited by applicant]
US 10686729B2 · Sindhu et al. · 2020 [cited by applicant]
US 10686763B2 · Kantecki et al. · 2020 [cited by applicant]
US 10826841B2 · Malloy et al. · 2020 [cited by applicant]
US 11575607B2 · Palermo et al. · 2023 [cited by applicant]
US 20100061245A1 · Larsen · 2010 [cited by applicant]
US 20110044171A1 · Csaszar · 2011 [cited by examiner]
US 20110142064A1 · Dubal · 2011 [cited by examiner]
US 20120033673A1 · Goel · 2012 [cited by examiner]
US 20120182870A1 · Francini · 2012 [cited by examiner]
US 20120314710A1 · Shikano · 2012 [cited by examiner]
US 20130159669A1 · Comparan et al. · 2013 [cited by applicant]
US 20140201305A1 · Dalal et al. · 2014 [cited by applicant]
US 20140281349A1 · Peters · 2014 [cited by applicant]
US 20150016266A1 · Dumitrescu · 2015 [cited by examiner]
US 20150019702A1 · Kancherla · 2015 [cited by examiner]
US 20150055457A1 · Agarwal · 2015 [cited by examiner]
US 20150071076A1 · Izhak-Ratzin · 2015 [cited by examiner]
US 20150082319A1 · Liu · 2015 [cited by examiner]
US 20150163146A1 · Zhang et al. · 2015 [cited by applicant]
US 20150200854A1 · Buchnik · 2015 [cited by examiner]
US 20150263968A1 · Jain · 2015 [cited by examiner]
US 20150263974A1 · Jain · 2015 [cited by examiner]
US 20160036693A1 · Galdy · 2016 [cited by applicant]
US 20160173398A1 · Reinig · 2016 [cited by applicant]
US 20160241482A1 · Tsuruoka · 2016 [cited by examiner]
US 20160285971A1 · Bilas et al. · 2016 [cited by applicant]
US 20160301632A1 · Anand · 2016 [cited by examiner]
US 20160344629A1 · Gray · 2016 [cited by applicant]
US 20170048144A1 · Liu · 2017 [cited by applicant]
US 20170054659A1 · Ergin et al. · 2017 [cited by applicant]
US 20170070356A1 · Dumitrescu et al. · 2017 [cited by applicant]
US 20170118120A1 · Testicioglu et al. · 2017 [cited by applicant]
US 20170192921A1 · Wang et al. · 2017 [cited by applicant]
US 20170195238A1 · Luo · 2017 [cited by examiner]
US 20170223107A1 · Ben Dayan et al. · 2017 [cited by applicant]
US 20170230298A1 · Perry · 2017 [cited by examiner]
US 20170286337A1 · Wang et al. · 2017 [cited by applicant]
US 20170289129A1 · Ghetie et al. · 2017 [cited by applicant]
US 20170318082A1 · Thakur et al. · 2017 [cited by applicant]
US 20170324713A1 · Kantecki et al. · 2017 [cited by applicant]
US 20170366449A1 · Loveless · 2017 [cited by examiner]
US 20180048524A1 · Mendoza et al. · 2018 [cited by applicant]
US 20180054485A1 · Warfield et al. · 2018 [cited by applicant]
US 20180069924A1 · Tumuluru et al. · 2018 [cited by applicant]
US 20180083878A1 · Francini et al. · 2018 [cited by applicant]
US 20180103060A1 · Li · 2018 [cited by examiner]
US 20180107766A1 · Dumitrescu et al. · 2018 [cited by applicant]
US 20180114012A1 · Sood · 2018 [cited by examiner]
US 20180123950A1 · Garg et al. · 2018 [cited by applicant]
US 20180157515A1 · Malloy et al. · 2018 [cited by applicant]
US 20180183860A1 · Majumdar · 2018 [cited by applicant]
US 20180191523A1 · Shah et al. · 2018 [cited by applicant]
US 20180191630A1 · Kenny et al. · 2018 [cited by applicant]
US 20180205656A1 · Atli · 2018 [cited by examiner]
US 20180212889A1 · Xu et al. · 2018 [cited by applicant]
US 20180220008A1 · Reith · 2018 [cited by applicant]
US 20180253084A1 · Diaz et al. · 2018 [cited by applicant]
US 20180278530A1 · Connor · 2018 [cited by examiner]
US 20180278532A1 · Ashner et al. · 2018 [cited by applicant]
US 20180285151A1 · Wang et al. · 2018 [cited by applicant]
US 20180293168A1 · Noureddine et al. · 2018 [cited by applicant]
US 20180300766A1 · Ashner et al. · 2018 [cited by applicant]
US 20180307521A1 · Pinto · 2018 [cited by applicant]
US 20180336067A1 · Lee et al. · 2018 [cited by applicant]
US 20180351692A1 · Rozenboim · 2018 [cited by applicant]
US 20180365176A1 · Finkelstein · 2018 [cited by examiner]
US 20180367460A1 · Gao et al. · 2018 [cited by applicant]
US 20190007332A1 · Fleming · 2019 [cited by examiner]
US 20190007347A1 · Chayat · 2019 [cited by examiner]
US 20190042304A1 · Wang · 2019 [cited by examiner]
US 20190042305A1 · McDonnell et al. · 2019 [cited by applicant]
US 20190042319A1 · Sood et al. · 2019 [cited by applicant]
US 20190042331A1 · McDonnell et al. · 2019 [cited by applicant]
US 20190043357A1 · Stinnett · 2019 [cited by applicant]
US 20190044892A1 · Mangan et al. · 2019 [cited by applicant]
US 20190095236A1 · Felter · 2019 [cited by examiner]
US 20190132253A1 · Thubert et al. · 2019 [cited by applicant]
US 20190158371A1 · Dillon et al. · 2019 [cited by applicant]
US 20190179668A1 · Wang et al. · 2019 [cited by applicant]
US 20190182180A1 · Frankel · 2019 [cited by examiner]
US 20190190838A1 · Shpiner · 2019 [cited by examiner]
US 20190253362A1 · Ruthstein · 2019 [cited by examiner]
US 20190260685A1 · Tsirkin · 2019 [cited by applicant]
US 20190303222A1 · Masputra et al. · 2019 [cited by applicant]
US 20190327126A1 · Rivaud · 2019 [cited by examiner]
US 20190327188A1 · Rivaud · 2019 [cited by examiner]
US 20190334828A1 · Fairhurst et al. · 2019 [cited by applicant]
US 20190334837A1 · Fairhurst et al. · 2019 [cited by applicant]
US 20200004584A1 · Burroughs et al. · 2020 [cited by applicant]
US 20200042479A1 · Wang et al. · 2020 [cited by applicant]
US 20200053019A1 · Singh · 2020 [cited by examiner]
US 20200084155A1 · Song · 2020 [cited by examiner]
US 20200104195A1 · Sanghi et al. · 2020 [cited by applicant]
US 20200151120A1 · Thyamagondlu et al. · 2020 [cited by applicant]
US 20200153756A1 · Thyamagondlu et al. · 2020 [cited by applicant]
US 20200196034A1 · Szymanski · 2020 [cited by examiner]
US 20200260317A1 · Francini et al. · 2020 [cited by applicant]
US 20200264914A1 · Dasgupta et al. · 2020 [cited by applicant]
US 20200267053A1 · Zheng et al. · 2020 [cited by applicant]
US 20200274796A1 · Garg et al. · 2020 [cited by applicant]
US 20200374742A1 · Chong et al. · 2020 [cited by applicant]
US 20200389410A1 · Guim Bernat et al. · 2020 [cited by applicant]
US 20200412659A1 · Arditti Ilitzky et al. · 2020 [cited by applicant]
US 20200412660A1 · Ji et al. · 2020 [cited by applicant]
US 20210011785A1 · Sanghi et al. · 2021 [cited by applicant]
US 20210051117A1 · Yoshida · 2021 [cited by examiner]
US 20210075730A1 · Palermo et al. · 2021 [cited by applicant]
US 20210297343A1 · Vegesna · 2021 [cited by examiner]
US 20210297350A1 · Vegesna · 2021 [cited by examiner]
US 20210297351A1 · Vegesna et al. · 2021 [cited by applicant]
US 20210336895A1 · He · 2021 [cited by applicant]
US 20210344648A1 · Patel · 2021 [cited by examiner]
US 20220091977A1 · Grunwald et al. · 2022 [cited by applicant]
US 20220166721A1 · Lam et al. · 2022 [cited by applicant]
US 20220286399A1 · McDonnell et al. · 2022 [cited by applicant]
US 20230198912A1 · McDonnell et al. · 2023 [cited by applicant]
US 20230231809A1 · Palermo et al. · 2023 [cited by applicant]
US 20230231811A1 · Dalal · 2023 [cited by examiner]
DE 112016006154T5 · 2018 [cited by applicant]
DE 112017001808T5 · 2019 [cited by applicant]
EP 3629189A2 · 2020 [cited by applicant]
EP 3637704B1 · 2023 [cited by applicant]
EP 3625755B1 · 2024 [cited by examiner]
WO 2014106201A1 · 2014 [cited by applicant]
WO 2017119980A1 · 2017 [cited by applicant]
WO 2017172217A1 · 2017 [cited by applicant]
WO 2018119153A2 · 2018 [cited by applicant]
WO 2018191355A1 · 2018 [cited by applicant]
WO 2021026740A1 · 2021 [cited by applicant]
WO 2021050951A1 · 2021 [cited by applicant]
Papaefstathiou et al., “Queue Management in Network Processors,” IEEE Computer Society, Proceedings of the Design Automation and Test in Europe Conference and Exhibition, 2005, 6 pages. [cited by applicant]
Mohammdadem, “Optimizing Mobile Backhaul Using Machine Learning,” Master Thesis, Aalto University School of Electrical Engineering, May 30, 2019, 63 pages. [cited by applicant]
DPDK, “NXP DPAA2 Even dev Driver,” retrieved from https://doc.dpdk.org/guides/eventdevs/dpaa2.html on Sep. 9, 2020, 3 pages. [cited by applicant]
DPDK, “Octeon TX2 SSO Eventdev Driver,” retrieved from https://doc-dpdk.org/guides/eventdevs/octeontx2.html on Sep. 9, 2020, 4 pages. [cited by applicant]
International Searching Authority, “Written Opinion”, mailed in connection with International Application No. PCT/US2020/050506 on Dec. 23, 2020, 9 pages. [cited by applicant]
International Searching Authority, “International Search Report,” issued in connection with International Patent Application No. PCT/US2020/050506, mailed on Dec. 23, 2020, 3 pages. [cited by applicant]
DPDK, “Octeon TX SSOVF Eventdev Driver,” retrieved from https://doc.dpdk.org/guides/eventdevs/octeontx.html on Jan. 11, 2021, 3 pages. [cited by applicant]
DPDK, “Octeon TX2 SSO Eventdev Driver,” retrieved from https://doc.dpdk.org/guides/eventdevs/octeontx2.html on Jan. 11, 2021, 4 pages. [cited by applicant]
International Searching Authority, “International Preliminary Report on Patentability,” issued in connection with International Patent Application No. PCT/US2020/050506, issued on Mar. 15, 2022, 11 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Action” issued in U.S. Appl. No. 17/018,809 on Jun. 7, 2022, 22 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” in connection with U.S. Appl. No. 17/018,809, mailed on Sep. 30, 2022, 13 pages. [cited by applicant]
European Patent Office, “Extended European Search Report,” issued in connection with European Patent Application No. 20863993.0, dated Feb. 3, 2023, 10 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 18/154,619, mailed on Jun. 1, 2023, 43 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 18/154,619, mailed on Oct. 20, 2023, 13 pages. [cited by applicant]
European Patent Office, “Communication Pursuant to Article 94(3) EPC,” issued in connection with European Patent Application No. 20863993.0-1213, dated Nov. 29, 2023, 8 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance,” issued in connection with U.S. Appl. No. 18/154,619, dated Jan. 17, 2024, 13 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 17/637,416, dated Apr. 19, 2024, 14 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 18/154,619, dated Jun. 7, 2024, 15 pages. [cited by applicant]
European Patent Office, “Communication under Rule 71(3) EPC—Intention to Grant,” issued in connection with European Patent Application No. 20 863 993.0-1206, dated Jul. 18, 2024, 140 pages. [cited by applicant]
United States Patent and Trademark Office, “Final Office Action,” issued in connection with U.S. Appl. No. 17/637,416, dated Sep. 12, 2024, 11 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 18/154,619, dated Aug. 7, 2024, 13 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 17/637,416, dated Nov. 26, 2024, 8 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 18/154,619, dated Dec. 24, 2024, 13 pages. [cited by applicant]