IP Library Granted Patent US 12,289,239
Granted Patent B2
US 12,289,239 · App. 18/154,619 · Granted Apr 29, 2025

Dynamic load balancing for multi-core computing environments

Inventors: Stephen Palermo (Chandler, AZ); Bradley Chaddick (Portland, OR); Gage Eads (Austin, TX); Mrittika Ganguli (Tempe, AZ); Abhishek Khade (Chandler, AZ); Abhirupa Layek (Chandler, AZ); Sarita Maini (Tempe, AZ); Niall McDonnell (Limerick, IE); Rahul Shah (Chandler, AZ); Shrikant Shah (Chandler, AZ); William Burroughs (Macungie, PA); David Sonnier (Austin, TX)
Assignee: Intel Corporation
H04L47/125H04L47/62H04L47/624H04L47/6255H04L47/6275
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,289,239
App. No.
18/154,619
Granted
Apr 29, 2025
Kind
B2
Abstract

Methods, apparatus, systems, and articles of manufacture are disclosed for dynamic load balancing for multi-core computing environments. An example apparatus includes a first and a plurality of second cores of a processor, and circuitry in a die of the processor separate from the first and the second cores, the circuitry to enqueue identifiers in one or more queues in the circuitry associated with respective ones of data packets of a packet flow, allocate one or more of the second cores to dequeue first ones of the identifiers in response to a throughput parameter of the first core not satisfying a throughput threshold to cause the one or more of the second cores to execute one or more operations on first ones of the data packets, and provide the first ones to one or more data consumers to distribute the first data packets.

Claims (37)

1. A semiconductor die comprising:

a first core and a second core; and

circuitry in the semiconductor die, the circuitry separate from the first core and the second core, the circuitry to:

assign the first core to process first data packets of a data flow, the first core to execute one or more operations on the first data packets to produce processed first data packets;

assign the second core to process second data packets of the data flow based on a size of the data flow, the second core to execute the one or more operations on the second data packets to produce processed second data packets; and

re-order the processed first data packets and the processed second data packets to produce a processed data flow.

2. The semiconductor die of claim 1 , wherein the first data packets and the second data packets are to be received from a first network interface card, and the circuitry is to aggregate the processed first data packets and the processed second data packets after being re-ordered in preparation for distribution to a second network interface card.

3. The semiconductor die of claim 1 , wherein the semiconductor die includes a third core, and the circuitry is to assign the third core to process third data packets of a second data flow.

4. The semiconductor die of claim 1 , wherein the circuitry is to assign the second core to process the second data packets of the data flow based on a comparison of the size of the data flow to a threshold.

5. The semiconductor die of claim 1 , wherein the data flow corresponds to a single session associated with a network connection, and the circuitry is to assign the second core to process the second data packets based on a duration of the network connection.

6. The semiconductor die of claim 5 , wherein the circuitry is to assign the second core to process the second data packets based on a comparison of the duration of the network connection to a threshold.

7. The semiconductor die of claim 1 , wherein the semiconductor die includes a third core, and the circuitry is to assign the third core to process the second data packets of the data flow based on a bandwidth associated with the second core.

8. The semiconductor die of claim 7 , wherein the circuitry is to assign the third core to process the second data packets of the data flow based on a comparison of the bandwidth associated with the second core to a threshold.

9. The semiconductor die of claim 1 , wherein the first core and the second core are included in a plurality of cores and the circuitry is to select ones of the plurality of cores based on at least one of an atomic scheduling or an ordered scheduling to distribute processing of the data flow.

10. The semiconductor die of claim 1 , wherein the data flow includes ciphertext, and the one or more operations include one or more cryptographic operations.

11. A method comprising:

assigning, with circuitry of a semiconductor die, first data packets of a data flow to a first core of the semiconductor die, the first core to execute one or more operations on the first data packets to produce processed first data packets, the circuitry of the semiconductor die separate from the first core and a second core of the semiconductor die;

assigning, with the circuitry, second data packets of the data flow to the second core based on a size of the data flow, the second core to execute the one or more operations on the second data packets to produce processed second data packets; and

re-ordering the processed first data packets and the processed second data packets to produce a processed data flow.

12. The method of claim 11 , further including:

receiving the first data packets and the second data packets from a first network interface card; and

aggregating the processed first data packets and the processed second data packets after being re-ordered in preparation for distribution to a second network interface card.

13. The method of claim 11 , wherein the semiconductor die includes a third core, and further including assigning the third core to process third data packets of a second data flow.

14. The method of claim 11 , wherein the assigning of the second core is based on a comparison of the size of the data flow to a threshold.

15. The method of claim 11 , wherein the data flow corresponds to a single session associated with a network connection, and the assigning of the second core is based a duration of the network connection.

16. The method of claim 15 , wherein the assigning of the second core is based on a comparison of the duration of the network connection to a threshold.

17. The method of claim 11 , wherein the semiconductor die includes a third core, and further including assigning the third core to process the second data packets of the data flow based on a bandwidth associated with the second core.

18. The method of claim 17 , wherein the assigning of the third core is based on a comparison of the bandwidth associated with the second core to a threshold.

19. An apparatus comprising:

a first core and a second core of a processor; and

circuitry in a die of the processor, the circuitry separate from the first core and the second core, the die including the first core, the second core, and the circuitry, the circuitry to:

enqueue an identifier in a queue in the circuitry, the identifier to reference a data packet obtained by a network interface card;

dequeue the identifier to the first core to cause the first core to execute a first operation on the data packet;

enqueue the identifier in the queue in response to obtaining an indication of completion of the first operation;

dequeue the identifier to the first core or the second core to cause the first core or the second core to execute a second operation on the data packet different from the first operation; and

distribute the data packet.

20. The apparatus of claim 19 , wherein the first operation is a decryption operation of an Internet Protocol security (IPsec) application and the second operation is an encryption operation of the IPsec application.

Continuity (4)
Continuation 17018809 · Sep 11, 2020
Provisional Application 62979963 · Feb 21, 2020
Provisional Application 62899061 · Sep 11, 2019
Related Publication 20230231809A1 · Jul 20, 2023
References Cited (155)
US 7289442B1 · Srinivasan · 2007 [cited by examiner]
US 8327187B1 · Metcalf · 2012 [cited by applicant]
US 9178815B2 · Gasparakis · 2015 [cited by applicant]
US 9313047B2 · Michels et al. · 2016 [cited by applicant]
US 9432298B1 · Smith · 2016 [cited by examiner]
US 9485326B1 · Ulman et al. · 2016 [cited by applicant]
US 10025619B1 · Felter · 2018 [cited by applicant]
US 10469404B1 · Mao et al. · 2019 [cited by applicant]
US 10505849B1 · Iny · 2019 [cited by examiner]
US 10686729B2 · Sindhu · 2020 [cited by examiner]
US 10686763B2 · Kantecki et al. · 2020 [cited by applicant]
US 10826841B2 · Malloy et al. · 2020 [cited by applicant]
US 11575607B2 · Palermo et al. · 2023 [cited by applicant]
US 20070230493A1 · Dravida · 2007 [cited by applicant]
US 20100061245A1 · Larsen · 2010 [cited by applicant]
US 20110044171A1 · Csaszar · 2011 [cited by applicant]
US 20110142064A1 · Dubal · 2011 [cited by applicant]
US 20120033673A1 · Goel · 2012 [cited by applicant]
US 20120182870A1 · Francini · 2012 [cited by applicant]
US 20120314710A1 · Shikano · 2012 [cited by applicant]
US 20130159669A1 · Comparan et al. · 2013 [cited by applicant]
US 20140201305A1 · Dalal · 2014 [cited by examiner]
US 20140281349A1 · Peters · 2014 [cited by applicant]
US 20150016266A1 · Dumitrescu · 2015 [cited by applicant]
US 20150019702A1 · Kancherla · 2015 [cited by applicant]
US 20150055457A1 · Agarwal · 2015 [cited by applicant]
US 20150071076A1 · Izhak-Ratzin · 2015 [cited by applicant]
US 20150082319A1 · Liu · 2015 [cited by applicant]
US 20150163146A1 · Zhang et al. · 2015 [cited by applicant]
US 20150200854A1 · Buchnik · 2015 [cited by applicant]
US 20150263968A1 · Jain · 2015 [cited by applicant]
US 20150263974A1 · Jain · 2015 [cited by applicant]
US 20160036693A1 · Galdy · 2016 [cited by examiner]
US 20160173398A1 · Reinig · 2016 [cited by examiner]
US 20160241482A1 · Tsuruoka · 2016 [cited by applicant]
US 20160285971A1 · Bilas · 2016 [cited by examiner]
US 20160301632A1 · Anand · 2016 [cited by applicant]
US 20160344629A1 · Gray · 2016 [cited by examiner]
US 20170048144A1 · Liu · 2017 [cited by examiner]
US 20170054659A1 · Ergin · 2017 [cited by examiner]
US 20170070356A1 · Dumitrescu et al. · 2017 [cited by applicant]
US 20170118120A1 · Testicioglu · 2017 [cited by examiner]
US 20170192921A1 · Wang et al. · 2017 [cited by applicant]
US 20170195238A1 · Luo · 2017 [cited by applicant]
US 20170223107A1 · Ben Dayan · 2017 [cited by examiner]
US 20170230298A1 · Perry · 2017 [cited by applicant]
US 20170286337A1 · Wang et al. · 2017 [cited by applicant]
US 20170289129A1 · Ghetie et al. · 2017 [cited by applicant]
US 20170318082A1 · Thakur et al. · 2017 [cited by applicant]
US 20170324713A1 · Kantecki et al. · 2017 [cited by applicant]
US 20170366449A1 · Loveless · 2017 [cited by applicant]
US 20180048524A1 · Mendoza et al. · 2018 [cited by applicant]
US 20180054485A1 · Warfield et al. · 2018 [cited by applicant]
US 20180069924A1 · Tumuluru et al. · 2018 [cited by applicant]
US 20180083878A1 · Francini · 2018 [cited by applicant]
US 20180103060A1 · Li · 2018 [cited by applicant]
US 20180107766A1 · Dumitrescu et al. · 2018 [cited by applicant]
US 20180114012A1 · Sood · 2018 [cited by applicant]
US 20180123950A1 · Garg · 2018 [cited by applicant]
US 20180157515A1 · Malloy et al. · 2018 [cited by applicant]
US 20180183860A1 · Majumdar · 2018 [cited by applicant]
US 20180191523A1 · Shah et al. · 2018 [cited by applicant]
US 20180191630A1 · Kenny et al. · 2018 [cited by applicant]
US 20180205656A1 · Atli · 2018 [cited by applicant]
US 20180212889A1 · Xu et al. · 2018 [cited by applicant]
US 20180220008A1 · Reith · 2018 [cited by applicant]
US 20180253084A1 · Diaz et al. · 2018 [cited by applicant]
US 20180278530A1 · Connor · 2018 [cited by applicant]
US 20180278532A1 · Ashner et al. · 2018 [cited by applicant]
US 20180285151A1 · Wang et al. · 2018 [cited by applicant]
US 20180293168A1 · Noureddine et al. · 2018 [cited by applicant]
US 20180300766A1 · Ashner · 2018 [cited by examiner]
US 20180307521A1 · Pinto · 2018 [cited by applicant]
US 20180336067A1 · Lee et al. · 2018 [cited by applicant]
US 20180351692A1 · Rozenboim · 2018 [cited by applicant]
US 20180365176A1 · Finkelstein · 2018 [cited by applicant]
US 20180367460A1 · Gao et al. · 2018 [cited by applicant]
US 20190007347A1 · Chayat · 2019 [cited by applicant]
US 20190042304A1 · Wang · 2019 [cited by applicant]
US 20190042305A1 · McDonnell et al. · 2019 [cited by applicant]
US 20190042319A1 · Sood · 2019 [cited by applicant]
US 20190042331A1 · McDonnell et al. · 2019 [cited by applicant]
US 20190043357A1 · Stinnett · 2019 [cited by applicant]
US 20190044892A1 · Mangan et al. · 2019 [cited by applicant]
US 20190095236A1 · Felter · 2019 [cited by applicant]
US 20190132253A1 · Thubert · 2019 [cited by examiner]
US 20190158371A1 · Dillon et al. · 2019 [cited by applicant]
US 20190179668A1 · Wang et al. · 2019 [cited by applicant]
US 20190182180A1 · Frankel · 2019 [cited by applicant]
US 20190190838A1 · Shpiner · 2019 [cited by applicant]
US 20190253362A1 · Ruthstein · 2019 [cited by applicant]
US 20190260685A1 · Tsirkin · 2019 [cited by examiner]
US 20190303222A1 · Masputra · 2019 [cited by examiner]
US 20190327126A1 · Rivaud · 2019 [cited by applicant]
US 20190327188A1 · Rivaud · 2019 [cited by applicant]
US 20190334828A1 · Fairhurst et al. · 2019 [cited by applicant]
US 20190334837A1 · Fairhurst et al. · 2019 [cited by applicant]
US 20200004584A1 · Burroughs et al. · 2020 [cited by applicant]
US 20200042479A1 · Wang et al. · 2020 [cited by applicant]
US 20200053019A1 · Singh · 2020 [cited by applicant]
US 20200084155A1 · Song · 2020 [cited by applicant]
US 20200104195A1 · Sanghi · 2020 [cited by examiner]
US 20200151120A1 · Thyamagondlu et al. · 2020 [cited by applicant]
US 20200153756A1 · Thyamagondlu · 2020 [cited by examiner]
US 20200260317A1 · Francini et al. · 2020 [cited by applicant]
US 20200264914A1 · Dasgupta · 2020 [cited by applicant]
US 20200267053A1 · Zheng · 2020 [cited by applicant]
US 20200274796A1 · Garg · 2020 [cited by applicant]
US 20200374742A1 · Chong et al. · 2020 [cited by applicant]
US 20200389410A1 · Guim Bernat · 2020 [cited by applicant]
US 20200412659A1 · Arditti Ilitzky · 2020 [cited by examiner]
US 20200412660A1 · Ji et al. · 2020 [cited by applicant]
US 20210011785A1 · Sanghi · 2021 [cited by examiner]
US 20210051117A1 · Yoshida · 2021 [cited by applicant]
US 20210297343A1 · Vegesna · 2021 [cited by applicant]
US 20210297350A1 · Vegesna · 2021 [cited by applicant]
US 20210297351A1 · Vegesna et al. · 2021 [cited by applicant]
US 20210336895A1 · He · 2021 [cited by examiner]
US 20210344648A1 · Patel · 2021 [cited by applicant]
US 20220091977A1 · Grunwald · 2022 [cited by examiner]
US 20220286399A1 · Mcdonnell · 2022 [cited by applicant]
US 20230198912A1 · Mcdonnell · 2023 [cited by applicant]
US 20230231811A1 · Dalal · 2023 [cited by applicant]
DE 112016006154T5 · 2018 [cited by applicant]
DE 112017001808T5 · 2019 [cited by applicant]
EP 3629189A2 · 2020 [cited by applicant]
EP 3637704B1 · 2023 [cited by examiner]
WO WO2014106201A1 · 2014 [cited by examiner]
WO 2017119980A1 · 2017 [cited by applicant]
WO 2017172217A1 · 2017 [cited by applicant]
WO WO2018119153A2 · 2018 [cited by examiner]
WO 2018191355A1 · 2018 [cited by applicant]
WO WO2021026740A1 · 2021 [cited by examiner]
DPDK, “NXP DPAA Eventdev Driver,” last retrieved Sep. 9, 2020, available online: https://doc.dpdk.org/guides/eventdevs/dpaa.html (3 pages). [cited by applicant]
DPDK, “NXP DPAA2 Even dev Driver,” last retrieved Sep. 9, 2020, available online: https://doc.dpdk.org/guides/eventdevs/dpaa2.html (3 pages). [cited by applicant]
DPDK, “OCTEON TX SSOVF Eventdev Driver,” last retrieved Jan. 11, 2021, available online: https://doc.dpdk.org/guides/eventdevs/octeontx.html (3 pages). [cited by applicant]
DPDK, “OCTEON TX2 SSO Eventdev Driver,” last retrieved Jan. 11, 2021, available online: https://doc.dpdk.org/guides/eventdevs/octeontx2.html (4 pages). [cited by applicant]
International Searching Authority, “International Search Report”, mailed in connection with International Application No. PCT/US2020/050506 dated Dec. 23, 2020 (3 pages). [cited by applicant]
International Searching Authority, “Written Opinion”, mailed in connection with International Application No. PCT/US2020/050506 dated Dec. 23, 2020 (9 pages). [cited by applicant]
DPDK, “Octeon TX SSOVF Eventdev Driver,” last retrieved Sep. 9, 2020, available online: https://doc.dpdk.org/guides/eventdevs/octeontx.html (4 pages). [cited by applicant]
DPDK, “Octeon TX2 SSO Eventdev Driver,” last retrieved Sep. 9, 2020, available online: https://doc.dpdk.org/guides/eventdevs/octeontx2.html (4 pages). [cited by applicant]
United States Patent and Trademark Office, “Non-Final Rejection,” mailed in connection with U.S. Appl. No. 17/018,809 dated Jun. 7, 2022, 22 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” mailed in connection with U.S. Appl. No. 17/018,809 dated Sep. 30, 2022, 13 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” mailed in connection with U.S. Appl. No. 17/018,809 dated Oct. 13, 2022, 2 pages. [cited by applicant]
International Searching Authority, “International Preliminary Report on Patentability,” issued in connection with International Patent Application No. PCT/US2020/050506, issued on Mar. 15, 2022, 10 pages. [cited by applicant]
European Patent Office, “Extended European Search Report,” issued in connection with European Patent Application No. 20863993.0, dated Feb. 3, 2023, 10 pages. [cited by applicant]
European Patent Office, “Communication Pursuant to Article 94(3) EPC,” issued in connection with European Patent Application No. 20863993.0-1213, dated Nov. 29, 2023, 8 pages. [cited by applicant]
“Optimizing Mobile Backhaul Using Machine Learning”; Mohammedadem; Master Thesis, School of Electrical Engineering, Aalto University School of Electrical Engineering; May 2019 (Year: 2019). [cited by applicant]
“Queue Management in Network Processors”; Papaefstathiou et al.; IEEE Computer Society, Proceedings of the Design Automation and Test in Europe Conference and Exhibition; 2005 (Year: 2005). [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 17/637,416, dated Apr. 19, 2024, 14 pages. [cited by applicant]
European Patent Office, “Communication under Rule 71(3) EPC—Intention to Grant,” issued in connection with European Patent Application No. 20 863 993.0-1206, dated Jul. 18, 2024, 140 pages. [cited by applicant]
United States Patent and Trademark Office, “Final Office Action,” issued in connection with U.S. Appl. No. 17/637,416, dated Sep. 12, 2024, 11 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 18/621,516, dated Nov. 8, 2024, 16 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 17/637,416, dated Nov. 26, 2024, 8 pages. [cited by applicant]
United States Patent and Trademark Office, “Corrected Notice of Allowability,” issued in connection with U.S. Appl. No. 17/637,416, dated Nov. 29, 2024, 5 pages. [cited by applicant]