IP Library › Granted Patent US 12,255,734
Granted Patent B2
US 12,255,734 · App. 17/973,575 · Granted Mar 18, 2025

Clock synchronization NIC offload

Inventors: Wojciech Wasko (Młynek, PL); Dotan David Levi (Kiryat Motzkin, IL); Avi Urman (Yokneam, IL); Natan Manevich (Ramat HaSharon, IL)
Assignee: Mellanox Technologies, Ltd
H04J3/0638
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,255,734
App. No.
17/973,575
Granted
Mar 18, 2025
Kind
B2
Abstract

In one embodiment, a system includes a network interface controller including a device interface to connect to a processing device and receive a time synchronization marker message from an application running on the processing device, a network interface to send packets over a network, and packet processing circuitry to process the time synchronization marker message for sending via the network interface over the network to a slave clock device, generate a time synchronization follow-up message including a timestamp indicative of when the synchronization marker message egressed the network interface, and process the time synchronization follow-up message for sending via the network interface over the network to the slave clock device.

Claims (39)

1. A system comprising a network interface controller (NIC) including:

a device interface to connect to a processing device and receive a time synchronization marker message from an application running on the processing device;

a network interface to send packets over a network; and

packet processing circuitry to:

process the time synchronization marker message for sending via the network interface over the network to a slave clock device;

generate a time synchronization follow-up message including a timestamp indicative of when the synchronization marker message egressed the network interface, thereby offloading generation of the time synchronization follow-up message from the application running on the processing device to the NIC; and

process the time synchronization follow-up message for sending via the network interface over the network to the slave clock device.

2. The system according to claim 1 , wherein the packet processing circuitry is configured to generate the time synchronization follow-up message autonomously of the application running on the processing device.

3. The system according to claim 1 , further comprising the processing device, wherein the application is configured to generate the time synchronization marker message but not the time synchronization follow-up message.

4. The system according to claim 1 , wherein the packet processing circuitry is configured to duplicate the time synchronization marker message in a transmission pipeline of the packet processing circuitry as the time synchronization follow-up message for sending to the slave clock device.

5. The system according to claim 1 , wherein the packet processing circuitry is configured to:

perform a packet loopback of the time synchronization marker message in a transmission pipeline of the packet processing circuitry yielding a loop-backed packet directed towards a receive pipeline of the packet processing circuitry with the timestamp added to metadata of the loop-backed packet; and

perform a hairpin operation to move the loop-backed packet from the receive pipeline to the transmission pipeline for sending to the slave clock device as the time synchronization follow-up message.

6. The system according to claim 5 , wherein the packet processing circuitry is configured to change an operation code field of the loop-backed packet from marker message to follow-up message.

7. The system according to claim 5 , wherein the packet processing circuitry is configured to insert the timestamp into a header of the loop-backed packet responsively to the timestamp in the metadata of the loop-backed packet.

8. The system according to claim 7 , wherein the packet processing circuitry is configured to insert the timestamp into a preciseOriginTimestamp field of the loop-backed packet.

9. The system according to claim 1 , further comprising a kernel space driver disposed between the application and the network interface controller.

10. The system according to claim 1 , further comprising the slave clock device including clock synchronization circuitry to:

receive the time synchronization marker message; and

receive the time synchronization follow-up message while being unaware that the time synchronization follow-up message was generated by the network interface controller and not the application running on the processing device.

11. A time synchronization method, comprising:

connecting to a processing device;

receiving a time synchronization marker message from an application running on the processing device;

sending packets over a network;

processing by a network interface controller (NIC) the time synchronization marker message for sending via a network interface over the network to a slave clock device;

generating by the NIC a time synchronization follow-up message including a timestamp indicative of when the synchronization marker message egressed the network interface, thereby offloading generation of the time synchronization follow-up message from the application running on the processing device to the NIC; and

processing by the NIC the time synchronization follow-up message for sending via the network interface over the network to the slave clock device.

12. The method according to claim 11 , wherein the generating includes generating the time synchronization follow-up message autonomously of the application running on the processing device.

13. The method according to claim 11 , further comprising the application generating the time synchronization marker message but not the time synchronization follow-up message.

14. The method according to claim 11 , further comprising duplicating by the network interface controller the time synchronization marker message in a transmission pipeline as the time synchronization follow-up message for sending to the slave clock device.

15. The method according to claim 11 , further comprising:

performing by the network interface controller a packet loopback of the time synchronization marker message in a transmission pipeline yielding a loop-backed packet directed towards a receive pipeline with the timestamp added to metadata of the loop-backed packet; and

performing a hairpin operation to move the loop-backed packet from the receive pipeline to the transmission pipeline for sending to the slave clock device as the time synchronization follow-up message.

16. The method according to claim 15 , further comprising changing an operation code field of the loop-backed packet from marker message to follow-up message.

17. The method according to claim 15 , further comprising inserting the timestamp into a header of the loop-backed packet responsively to the timestamp in the metadata of the loop-backed packet.

18. The method according to claim 17 , wherein the inserting includes inserting the timestamp into a preciseOriginTimestamp field of the loop-backed packet.

19. The method according to claim 11 , further comprising:

receiving by the slave clock device the time synchronization marker message; and

receiving by the slave clock device the time synchronization follow-up message while being unaware that the time synchronization follow-up message was generated by the network interface controller and not the application running on the processing device.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 26, 2022
From: WASKO, WOJCIECH; LEVI, DOTAN DAVID; URMAN, AVI; MANEVICH, NATAN
To: MELLANOX TECHNOLOGIES, LTD.
Reel/Frame 061535/0876 →
Continuity (1)
Related Publication 20240146431A1 · May 2, 2024
References Cited (118)
US 6212197B1 · Christensen et al. · 2001 [cited by applicant]
US 6564267B1 · Lindsay · 2003 [cited by applicant]
US 6816492B1 · Turner et al. · 2004 [cited by applicant]
US 6954432B1 · Bychowsky et al. · 2005 [cited by applicant]
US 7224669B2 · Kagan et al. · 2007 [cited by applicant]
US 7245627B2 · Goldenberg et al. · 2007 [cited by applicant]
US 7676597B2 · Kagan et al. · 2010 [cited by applicant]
US 8370675B2 · Kagan · 2013 [cited by applicant]
US 8879552B2 · Zheng · 2014 [cited by examiner]
US 9390056B1 · Noureddine et al. · 2016 [cited by applicant]
US 9397960B2 · Arad et al. · 2016 [cited by applicant]
US 9450846B1 · Huang · 2016 [cited by examiner]
US 9628209B2 · Kingsley · 2017 [cited by examiner]
US 9667446B2 · Worrell · 2017 [cited by applicant]
US 10019333B2 · Regev · 2018 [cited by examiner]
US 10027601B2 · Narkis et al. · 2018 [cited by applicant]
US 10075253B2 · Olofsson · 2018 [cited by examiner]
US 10084559B1 · Devineni · 2018 [cited by examiner]
US 10187400B1 · Castro et al. · 2019 [cited by applicant]
US 10284383B2 · Bloch et al. · 2019 [cited by applicant]
US 10320508B2 · Shimizu et al. · 2019 [cited by applicant]
US 10727966B1 · Izenberg et al. · 2020 [cited by applicant]
US 10778361B1 · Almog et al. · 2020 [cited by applicant]
US 11271874B2 · Levi et al. · 2022 [cited by applicant]
US 11283454B2 · Ravid et al. · 2022 [cited by applicant]
US 11336383B2 · Mula et al. · 2022 [cited by applicant]
US 11388263B2 · Levi et al. · 2022 [cited by applicant]
US 11476928B2 · Levi et al. · 2022 [cited by applicant]
US 11543852B2 · Levi et al. · 2023 [cited by applicant]
US 11831403B2 · Lee · 2023 [cited by examiner]
US 20030002483A1 · Zwack · 2003 [cited by applicant]
US 20040174820A1 · Ricciulli · 2004 [cited by applicant]
US 20060007862A1 · Sayeedi et al. · 2006 [cited by applicant]
US 20060064579A1 · Aaron et al. · 2006 [cited by applicant]
US 20060095732A1 · Tran et al. · 2006 [cited by applicant]
US 20060153179A1 · Ho et al. · 2006 [cited by applicant]
US 20080298380A1 · Rittmeyer et al. · 2008 [cited by applicant]
US 20120201248A1 · Aoki · 2012 [cited by applicant]
US 20130027611A1 · Closset · 2013 [cited by applicant]
US 20130173950A1 · Banerjea et al. · 2013 [cited by applicant]
US 20140092918A1 · Jost · 2014 [cited by applicant]
US 20140379714A1 · Hankins · 2014 [cited by applicant]
US 20150046741A1 · Yen et al. · 2015 [cited by applicant]
US 20150055508A1 · Ashida et al. · 2015 [cited by applicant]
US 20150318015A1 · Bose et al. · 2015 [cited by applicant]
US 20170125026A1 · Lawrence · 2017 [cited by applicant]
US 20170171362A1 · Bolotov et al. · 2017 [cited by applicant]
US 20170188092A1 · Lawrence · 2017 [cited by applicant]
US 20170331926A1 · Raveh et al. · 2017 [cited by applicant]
US 20180124812A1 · Thubert et al. · 2018 [cited by applicant]
US 20180191629A1 · Biederman et al. · 2018 [cited by applicant]
US 20180309821A1 · Byers et al. · 2018 [cited by applicant]
US 20180351676A1 · Butterworth · 2018 [cited by examiner]
US 20190056972A1 · Zhou et al. · 2019 [cited by applicant]
US 20190087352A1 · Lee et al. · 2019 [cited by applicant]
US 20190087652A1 · Townsend et al. · 2019 [cited by applicant]
US 20190124524A1 · Gormley · 2019 [cited by applicant]
US 20190149365A1 · Chatterjee et al. · 2019 [cited by applicant]
US 20190319730A1 · Webb et al. · 2019 [cited by applicant]
US 20190379714A1 · Levi et al. · 2019 [cited by applicant]
US 20200026656A1 · Liao et al. · 2020 [cited by applicant]
US 20200252320A1 · Zemach et al. · 2020 [cited by applicant]
US 20200259759A1 · Shalev et al. · 2020 [cited by applicant]
US 20210081352A1 · Yang et al. · 2021 [cited by applicant]
US 20210152267A1 · Chan · 2021 [cited by examiner]
US 20210328900A1 · Sattinger et al. · 2021 [cited by applicant]
US 20220116132A1 · Kasichainula · 2022 [cited by examiner]
US 20220360423A1 · Levi et al. · 2022 [cited by applicant]
US 20220416925A1 · Levi et al. · 2022 [cited by applicant]
US 20230262281A1 · Kasichainula · 2023 [cited by examiner]
AU 2012216611B2 · 2013 [cited by applicant]
CN 103354989A · 2013 [cited by applicant]
EP 0607412B1 · 1999 [cited by applicant]
JP 2005529523A · 2005 [cited by applicant]
JP 2007042145A · 2007 [cited by applicant]
WO 9418627A2 · 1994 [cited by applicant]
WO 9857497A1 · 1998 [cited by applicant]
WO 2001037489A1 · 2001 [cited by applicant]
WO 2010117359A1 · 2010 [cited by applicant]
IEEE Standard 1588-2008, “IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems”, pp. 1-289, year 2008. [cited by applicant]
Weibel et al., “Implementation and Performance of Time Stamping Techniques”, 2004 Conference on IEEE 1588, pp. 1-29, Sep. 28, 2004. [cited by applicant]
InfiniBandTM Architecture Specification vol. 1, Release 1.2.1, pp. 1-1727, Nov. 2007. [cited by applicant]
Lu et al., “A Fast CRC Update Implementation”, Computer Engineering Laboratory, Electrical Engineering Department, pp. 113-120, Oct. 8, 2003. [cited by applicant]
Mellette et al., “Toward Optical Switching in the Data Center”, IEEE 19th International Conference on High Performance Switching and Routing (HPSR), pp. 1-6, Bucharest, Romania, Jun. 18-20, 2018. [cited by applicant]
Bakopoulos et al., “NEPHELE: an end-to-end scalable and dynamically reconfigurable optical architecture for application-aware SDN cloud datacenters”, IEEE Communications Magazine, vol. 56, issue 2, pp. 1-26, Feb. 2018. [cited by applicant]
O-RAN Alliance, “O-RAN Fronthaul Working Group; Control, User and Synchronization Plane Specification”, ORAN-WG4.CUS.0-v01.00, Technical Specification, pp. 1-189, year 2019. [cited by applicant]
Vattikonda et al., “Practical TDMA for Datacenter Ethernet”, EuroSys conference, Bern, Switzerland, pp. 225-238, Apr. 10-13, 2012. [cited by applicant]
Ericsson Ab et al., “Common Public Radio Interface: eCPRI Interface Specification”, V2.0, pp. 1-109, May 10, 2019. [cited by applicant]
Xilinx Inc., “Radio over Ethernet Framer v2.1”, PB056 (v2.1), pp. 1-9, Oct. 30, 2019. [cited by applicant]
Weibel, H., “High Precision Clock Synchronization according to IEEE 1588 Implementation and Performance Issues”, Zurich University of Applied Sciences, pp. 1-9, Jan. 17, 2005. [cited by applicant]
Sanchez-Palencia, J., “[RFC,v3,net-next,00/18] Time based packet transmission”, pp. 1-14, Mar. 7, 2018. [cited by applicant]
IEEE Std 802.1Qaz™, “IEEE Standard for Local and metropolitan area networks—Media Access Control (MAC) Bridges and Virtual Bridged Local Area Networks—Amendment 18: Enhanced Transmission Selection for Bandwidth Sharing … [cited by applicant]
Crupnicoff et al., “Deploying Quality of Service and Congestion Control in InfiniBand-based Data Center Networks” White Paper, Mellanox Technologies Inc, Rev 1.0, pp. 1-19, Oct. 15, 2005. [cited by applicant]
Mathey et al., “Scalable Deadlock-Free Deterministic Minimal-Path Routing Engine for InfiniBand-Based Dragonfly Networks”, IEEE Transactions on Parallel and Distributed Systems, vol. 29, No. 1, pp. 183-197, Jan. 2018. [cited by applicant]
Wikipedia, Precision Time Protocol, pp. 1-9, Apr. 20, 2020. [cited by applicant]
SMPTE Standard, “Professional Media Over Managed IP Networks:Traffic Shaping and Delivery Timing for Video”, The Society of Motion Picture and Television Engineers, pp. 1-17, Nov. 22, 2017. [cited by applicant]
Wikipedia, “Time-Sensitive Networking”, pp. 1-12, Mar. 5, 2020. [cited by applicant]
IEEE Standard 1588-2002, “IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems”, pp. 1-154, year 2002. [cited by applicant]
Yau et al., “Development of Situation-Aware Application Software for Ubiquitous Computing Environments,” Proceedings of the 26th Annual International Computer Software and Applications Conference (COMSAC'02), IEEE Compu… [cited by applicant]
Davie et al., “An Expedited Forwarding PHB (Per-Hop-Behavior)”, RFC 3246, Network Working Group, pp. 1-32, Mar. 2002. [cited by applicant]
Mills, “Network Time Protocol (NTP),” RFC 0958, pp. 2-15, Sep. 1985. [cited by applicant]
Mills, “Network Time Protocol (Version 1): Specification and Implementation,” RFC 1059, pp. 2-59, Jul. 1988. [cited by applicant]
Mills, “Internet Time Synchronization: The Network Time Protocol,” IEEE Transactions on Communication, vol. 39, No. 10, pp. 1482-1493, Oct. 1991. [cited by applicant]
Mills, “Network Time Protocol (Version 3): Specification, Implementation and Analysis,” RFC 1305, pp. 1-96, Mar. 1992. [cited by applicant]
Baboescu et al., “Scalable Packet Classification,” Proceedings of SIGCOMM'01, ACM, pp. 199-210, year 2001. [cited by applicant]
Bader et al., “Survey and Comparison of Open Source Time Series Databases,” BTW 2017, Workshopband, Lecture Notes in Informatics (LNI), Gesellschaft fuer Informatik, Bonn, pp. 249-268, year 2017. [cited by applicant]
Li et al., “Sundial: Fault-Tolerant Clock Synchronization for Datacenters,” Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation, Open Access, USENIX Association, pp. 1171-1186, Nov. 2… [cited by applicant]
Levi et al., U.S. Appl. No. 17/667,600, filed Feb. 9, 2022. [cited by applicant]
“Time-Appliance-Project / Open-Time-Server,” Spec Revision No. 1.0, GitHub, Inc., pp. 1-9, last updated Oct. 3, 2022, as downloaded from http://www.opentimeserver.com/. [cited by applicant]
Manevich et al., U.S. Appl. No. 17/579,630, filed Jan. 20, 2022. [cited by applicant]
Levi et al., U.S. Appl. No. 17/871,937, filed Jul. 24, 2022. [cited by applicant]
Levi et al., U.S. Appl. No. 17/868,841, filed Jul. 20, 2022. [cited by applicant]
Levi et al., U.S. Appl. No. 17/983,427, filed Nov. 9, 2022. [cited by applicant]
Manevich et al., U.S. Appl. No. 17/885,604, filed Aug. 11, 2022. [cited by applicant]
Manevich et al., U.S. Appl. No. 18/110,788, filed Feb. 16, 2023. [cited by applicant]
Wertheimer et al., U.S. Appl. No. 17/886,606, filed Aug. 12, 2022. [cited by applicant]
U.S. Appl. No. 17/667,600 Office Action dated Oct. 23, 2024. [cited by applicant]
U.S. Appl. No. 17/667,600 Office Action dated Dec. 16, 2024. [cited by applicant]