IP Library › Granted Patent US 12,375,561
Granted Patent B2
US 12,375,561 · App. 18/652,561 · Granted Jul 29, 2025

Cloud scale multi-tenancy for RDMA over converged ethernet (RoCE)

Inventors: Santosh Narayan Shilimkar (San Jose, CA); David Dale Becker (Seattle, WA); Jagwinder Singh Brar (Bellevue, WA)
Assignee: Oracle International Corporation
H04L67/1097H04L12/4645H04L47/115H04L47/2408H04L67/61H04L69/22H04L69/325H04L2212/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,375,561
App. No.
18/652,561
Granted
Jul 29, 2025
Kind
B2
Abstract

Techniques and apparatus for data networking are described. In one example, a method includes receiving a first Layer-2 Remote Direct Memory Access (RDMA) packet which includes a virtual local area network (VLAN) tag and a quality-of-service (QoS) data field; converting the first Layer-2 RDMA packet to a first Layer-3 encapsulated packet; and forwarding the first Layer-3 encapsulated packet to a switch fabric. In this method, the converting includes adding at least one header to the first Layer-2 RDMA packet, where the at least one header includes: a virtual network identifier that is based on information from the VLAN tag, and a QoS value that is based on information from the QoS data field.

Claims (67)

1. A method of data networking, the method comprising:

receiving, at an ingress switch and from a host machine executing a plurality of compute instances for a plurality of tenants, a first Layer-2 Remote Direct Memory Access (RDMA) packet for a first tenant among the plurality of tenants;

converting the first Layer-2 RDMA packet to a first Layer-3 encapsulated packet having at least one header;

forwarding the first Layer-3 encapsulated packet to a switch fabric,

wherein the first Layer-2 RDMA packet includes a virtual local area network (VLAN) tag and a quality-of-service (QOS) data field, and

wherein the converting includes adding the at least one header to the first Layer-2 RDMA packet, the at least one header added to the first Layer-2 RDMA packet including:

a virtual network identifier that is based on information from the VLAN tag, wherein the virtual network identifier is a virtual local area network identifier (VLAN ID), and wherein the VLAN ID is located in an 802.1Q tag of the first Layer-2 RDMA packet, and

a QoS value that is based on information from the QoS data field; and

translating congestion information in a congestion notification data field of the at least one header of the first Layer-3 encapsulated packet to map the congestion information to a header of the 802.1Q tag of the first Layer-2 RDMA packet.

2. The method according to claim 1 , wherein the VLAN ID comprises tenancy information.

3. The method according to claim 1 , further comprising mapping the VLAN ID to information included in a Layer-3 overlay encapsulation protocol wrapper that is added to the 802.1Q tagged Layer-2 RDMA packet.

4. The method according to claim 1 , wherein the switch fabric is configured to use a network control traffic class for underlying IP routing protocol functions among Top-of-Rack switches.

5. The method according to claim 1 , wherein an Ethernet Virtual Private Network (EVPN) carries Media Access Control (MAC) address information across an underlying layer-3 network.

6. The method according to claim 1 , further comprising:

at the ingress switch, receiving a second Layer-2 RDMA packet which includes a VLAN tag and a QoS data field;

converting the second Layer-2 RDMA packet to a second Layer-3 encapsulated packet having at least one header; and

forwarding the second Layer-3 encapsulated packet to the switch fabric,

wherein the VLAN tag of the second Layer-2 RDMA packet indicates a different VLAN than the VLAN tag of the first Layer-2 RDMA packet.

7. The method according to claim 6 , further comprising, at an intermediate switch of the switch fabric:

based on the QoS value of the at least one header of the first Layer-3 encapsulated packet, queuing the first Layer-3 encapsulated packet in a first queue of the intermediate switch; and

based on the QoS value of the at least one header of the second Layer-3 encapsulated packet, queuing the second Layer-3 encapsulated packet in a second queue of the intermediate switch that is different than the first queue.

8. The method according to claim 6 further comprising:

at an egress switch, receiving the first Layer-3 encapsulated packet;

decapsulating the first Layer-3 encapsulated packet to obtain the first Layer-2 RDMA packet; and

based on the VLAN tag of the first Layer-2 RDMA packet, forwarding the first Layer-2 RDMA packet to a first compute instance;

at the egress switch, receiving the second Layer-3 encapsulated packet;

decapsulating the second Layer-3 encapsulated packet to obtain the second Layer-2 RDMA packet; and

based on the VLAN tag of the second Layer-2 RDMA packet, forwarding the second Layer-2 RDMA packet to a second compute instance that is different than the first compute instance.

9. The method according to claim 1 , further comprising:

at an egress switch, receiving the first Layer-3 encapsulated packet;

decapsulating the first Layer-3 encapsulated packet to obtain the first Layer-2 RDMA packet; and

based on the VLAN tag of the first Layer-2 RDMA packet, forwarding the first Layer-2 RDMA packet to a first compute instance.

10. The method according to claim 9 , further comprising,

based on the information in the congestion notification data field of the at least one header of the first Layer-3 encapsulated packet, setting a value of a congestion notification data field of the first Layer-2 RDMA packet.

11. The method according to claim 1 , wherein the QoS value is a Differentiated Services Code Point (DSCP) field of an outer IP header of the first Layer-3 encapsulated packet, and

wherein the converting includes copying a DSCP field of an IP header of the first Layer-2 RDMA packet to the DSCP field of the outer IP header of the first Layer-3 encapsulated packet.

12. The method according to claim 1 , wherein the first Layer-3 encapsulated packet is a Virtual Extensible Local Area Network (VxLAN) packet, and

wherein the virtual network identifier is a Virtual Network Identifier (VNI) of a VxLAN header of the first Layer-3 encapsulated packet.

13. The method according to claim 1 , further comprising preventing a pause frame from traveling beyond the switch fabric.

14. A system for data networking, the system comprising:

a switch fabric; and

an ingress switch configured to:

receive, from a host machine executing a plurality of compute instances for a plurality of tenants, a first Layer-2 Remote Direct Memory Access (RDMA) packet for a first tenant among the plurality of tenants;

convert the first Layer-2 RDMA packet to a first Layer-3 encapsulated packet having at least one header;

forward the first Layer-3 encapsulated packet to the switch fabric,

wherein the first Layer-2 RDMA packet includes a virtual local area network (VLAN) tag and a quality-of-service (QOS) data field, and

wherein the ingress switch configured to convert the first Layer-2 RDMA packet is configured to:

add the at least one header to the first Layer-2 RDMA packet, the at least one header including:

a virtual network identifier that is based on information from the VLAN tag,

wherein the virtual network identifier is a virtual local area network identifier (VLAN ID), and wherein the VLAN ID is located in an 802.1Q tag of the first Layer-2 RDMA packet, and

a QoS value that is based on information from the QoS data field; and

translate congestion information in a congestion notification data field of the at least one header of the first Layer-3 encapsulated packet to map the congestion information to a header of the 802.1Q tag of the first Layer-2 RDMA packet.

15. The system according to claim 14 , further comprising mapping the VLAN ID to information included in a Layer-3 overlay encapsulation protocol wrapper that is added to the 802.1Q tagged Layer-2 RDMA packet.

16. The system according to claim 14 , wherein the switch fabric is configured to use a network control traffic class for underlying IP routing protocol functions among Top-of-Rack switches.

17. The system according to claim 14 , wherein an Ethernet Virtual Private Network (EVPN) carries Media Access Control (MAC) address information across an underlying layer-3 network.

18. A non-transitory computer-readable medium storing a plurality of instructions executable by one or more processors, the plurality of instructions comprising instructions that when executed by the one or more processors cause the one or more processors to:

receive, from a host machine executing a plurality of compute instances for a plurality of tenants, a first Layer-2 Remote Direct Memory Access (RDMA) packet;

convert the first Layer-2 RDMA packet to a first Layer-3 encapsulated packet having at least one header;

forward the first Layer-3 encapsulated packet to a switch fabric,

wherein the first Layer-2 RDMA packet includes a virtual local area network (VLAN) tag and a quality-of-service (QOS) data field, and

wherein the instructions that when executed by the one or more processors cause the one or more processors to convert cause the one or more processors to add the at least one header to the first Layer-2 RDMA packet, the at least one header including:

a virtual network identifier that is based on information from the VLAN tag,

wherein the virtual network identifier is a virtual local area network identifier (VLAN ID), and wherein the VLAN ID is located in an 802.1Q tag of the first Layer-2 RDMA packet, and

a QoS value that is based on information from the QoS data field; and

translate congestion information in a congestion notification data field of the at least one header of the first Layer-3 encapsulated packet to map the congestion information to a header of the 802.1Q tag of the first Layer-2 RDMA packet.

19. The non-transitory computer-readable medium according to claim 18 , further comprising mapping the VLAN ID to information included in a Layer-3 overlay encapsulation protocol wrapper that is added to the 802.1Q tagged Layer-2 RDMA packet.

20. The non-transitory computer-readable medium according to claim 18 , wherein the switch fabric is configured to use a network control traffic class for underlying IP routing protocol functions among Top-of-Rack switches.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 26, 2024
From: SHILIMKAR, SANTOSH NARAYAN; BECKER, DAVID DALE; BRAR, JAGWINDER SINGH
To: ORACLE INTERNATIONAL CORPORATION
Reel/Frame 068095/0718 →
Continuity (3)
Continuation 17165877 · Feb 2, 2021
Provisional Application 63132417 · Dec 30, 2020
Related Publication 20240291889A1 · Aug 29, 2024
References Cited (62)
US 6963576B1 · Lee · 2005 [cited by applicant]
US 7415035B1 · Muller et al. · 2008 [cited by applicant]
US 8824293B2 · Assarpour · 2014 [cited by examiner]
US 9712440B2 · Koodli et al. · 2017 [cited by applicant]
US 9747249B2 · Cherian et al. · 2017 [cited by applicant]
US 10177936B2 · Banavalikar · 2019 [cited by examiner]
US 10887131B1 · Chin · 2021 [cited by examiner]
US 11042496B1 · BeSerra et al. · 2021 [cited by applicant]
US 11258719B1 · Sommers · 2022 [cited by applicant]
US 11991246B2 · Shilimkar · 2024 [cited by examiner]
US 12010173B2 · Shilimkar et al. · 2024 [cited by applicant]
US 20120063316A1 · Ghanwani et al. · 2012 [cited by applicant]
US 20130223287A1 · Ahmad · 2013 [cited by examiner]
US 20150281099A1 · Banavalikar · 2015 [cited by applicant]
US 20160036774A1 · Chong et al. · 2016 [cited by applicant]
US 20160112305A1 · Djekic et al. · 2016 [cited by applicant]
US 20160188527A1 · Cherian et al. · 2016 [cited by applicant]
US 20160212214A1 · Rahman et al. · 2016 [cited by applicant]
US 20160337234A1 · Duda · 2016 [cited by examiner]
US 20170171075A1 · Sajeepa et al. · 2017 [cited by applicant]
US 20190116126A1 · Shen et al. · 2019 [cited by applicant]
US 20190207868A1 · Chang et al. · 2019 [cited by applicant]
US 20190303345A1 · Zhu et al. · 2019 [cited by applicant]
US 20190354406A1 · Ganguli et al. · 2019 [cited by applicant]
US 20190394067A1 · Yu · 2019 [cited by examiner]
US 20200021532A1 · Borikar et al. · 2020 [cited by applicant]
US 20200021555A1 · Fernando · 2020 [cited by examiner]
US 20200151137A1 · Izenberg et al. · 2020 [cited by applicant]
US 20200192695A1 · Vincent et al. · 2020 [cited by applicant]
US 20200280518A1 · Lee et al. · 2020 [cited by applicant]
US 20200382329A1 · Yuan · 2020 [cited by applicant]
US 20210368029A1 · Zhang et al. · 2021 [cited by applicant]
US 20220058146A1 · Balakrishnan et al. · 2022 [cited by applicant]
US 20220060417A1 · Kfir et al. · 2022 [cited by applicant]
US 20220092021A1 · Cherian · 2022 [cited by examiner]
US 20220171663A1 · Hiregoudar et al. · 2022 [cited by applicant]
CN 102594713A · 2012 [cited by applicant]
CN 110022269A · 2019 [cited by applicant]
JP 2001244987A · 2001 [cited by applicant]
JP 2011024027A · 2011 [cited by applicant]
JP 2011205187A · 2011 [cited by applicant]
JP 2016504820A · 2016 [cited by applicant]
WO 2004112318A1 · 2004 [cited by applicant]
Oracle Introduces Exadata Cloud Service X8M, Oracle, Available online at: https://www.oracle.com/news/announcement/oracle-introduces-exadata-cloud-service-x8m-101520.html, Accessed from Internet on: Feb. 2, 2021, 5 page… [cited by applicant]
RDMA Over Converged Ethernet (RoCE) Design Guide, Hewlett Packard Enterprise, 2017, 22 pages. [cited by applicant]
Supplement to InfiniBandTM Architecture Specification, Annex A17: RoCEv2, vol. 1 Release 1.2.1, Sep. 2, 2014, 23 pages. [cited by applicant]
U.S. Appl. No. 17/165,877, Final Office Action mailed on May 25, 2023, 18 pages. [cited by applicant]
U.S. Appl. No. 17/165,877, Non-Final Office Action mailed on Aug. 4, 2022, 16 pages. [cited by applicant]
U.S. Appl. No. 17/165,877, Non-Final Office Action mailed on Sep. 19, 2023, 19 pages. [cited by applicant]
U.S. Appl. No. 17/166,922, Final Office Action mailed on Mar. 16, 2023, 25 pages. [cited by applicant]
U.S. Appl. No. 17/166,922, Non-Final Office Action mailed on Jul. 20, 2023, 19 pages. [cited by applicant]
U.S. Appl. No. 17/166,922, Non-Final Office Action mailed on Jun. 9, 2022, 20 pages. [cited by applicant]
Guo et al., RDMA Over Commodity Ethernet at Scale, SIGCOMM '16: Proceedings of the 2016 ACM SIGCOMM Conference, Aug. 22, 2016, 14 pages. [cited by applicant]
Harkins, RDMA over Converged Ethernet, Mellanox Technologies, May 2017, 24 pages. [cited by applicant]
Parish, Introducing Exadata X8M: In-Memory Performance with All the Benefits of shared storage for both OLTP and Analytics, Available Online at https://blogs.oracle.com/exadata/exadata-x8m, Sep. 17, 2019, 10 pages. [cited by applicant]
International Application No. PCT/US2021/025459, International Search Report and Written Opinion mailed on Jul. 30, 2021, 11 pages. [cited by applicant]
International Application No. PCT/US2021/027069, International Search Report and Written Opinion mailed on Nov. 15, 2021, 16 pages. [cited by applicant]
International Application No. PCT/US2021/027069, Invitation to Pay Additional Fees and, Where Applicable, Protest Fee mailed on Sep. 23, 2021, 5 pages. [cited by applicant]
Zhu et al., Congestion Control for Large-Scale RDMA Deployments, ACM SIGCOMM Computer Communication Review, Aug. 17-21, 2015, pp. 523-536. [cited by applicant]
International Application No. JP2023-540185, “Notice of Decision to Grant”, mailed Oct. 1, 2024, 3 pages. [cited by applicant]
International Application No. JP2023-540190, “Office Action”, mailed Oct. 1, 2024, 8 pages. [cited by applicant]
International Application No. EP21721693.6, “Office Action”, mailed Jan. 31, 2025, 5 pages. [cited by applicant]