IP Library › Granted Patent US 12,511,161
Granted Patent B2
US 12,511,161 · App. 17/667,600 · Granted Dec 30, 2025

Time-based synchronization descriptors

Inventors: Dotan David Levi (Kiryat Motzkin, IL); Daniel Marcovitch (Yokneam Illit, IL); Natan Manevich (Migdal HaEmek, IL); Wojciech Wasko (Pomeranian, PL); Igor Voks (Kfar Yehoshua, IL)
Assignee: Mellanox Technologies, Ltd
G06F9/4887
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,511,161
App. No.
17/667,600
Granted
Dec 30, 2025
Kind
B2
Abstract

In one embodiment, a system includes a peripheral device including a hardware clock, and processing circuitry to read a given work request entry stored with a plurality of work request entries in at least one work queue in a memory, the given work request entry including timing data and an operator, the timing data being indicative of a time at which a work request should be executed, retrieve a clock value from the hardware clock, and execute the work request with a workload while execution of the work request is timed responsively to the timing data and the operator and the retrieved clock value.

Claims (52)

1 . A peripheral device, comprising:

a hardware clock; and

processing circuitry to:

read a given work request entry stored with a plurality of work request entries in at least one work queue in a memory, respective ones of the work request entries being fence entries without work requests, the fence entries including timing data that defines execution timing for other work requests, the given work request entry including timing data and an operator, the timing data being indicative of a time at which a work request should be executed;

retrieve a clock value from the hardware clock; and

execute the work request with a workload while execution of the work request is timed responsively to the timing data and the operator and the retrieved clock value, wherein:

the operator defines a condition for executing the work request based on the timing data and retrieved clock value; and

the processing circuitry includes a packet processing pipeline to perform a steering decision process based on packets associated with the work request entries.

2 . A system comprising the peripheral device according to claim 1 , and further comprising multiple host devices, each host device including:

an interface to connect with the peripheral device and transfer data with the peripheral device; and

a processor to run at least one software application, which is configured to write host-specific work request entries of the work request entries into at least one respective work queue of the at least one work queue.

3 . A system comprising the peripheral device according to claim 1 , and further comprising a host device including:

an interface to connect with the peripheral device and transfer data with the peripheral device; and

a processor to run at least one software application, which is configured to write the work request entries into the at least one work queue in the memory, wherein the peripheral device includes an interface configured to connect with the host device and transfer the data with the host device.

4 . The system according to claim 3 , wherein the host device includes the memory.

5 . The system according to claim 3 , wherein the at least one software application is configured to write the timing data according to Coordinated Universal Time (UTC) format.

6 . The system according to claim 3 , wherein the at least one software application is configured to generate respective timing data of respective ones of the work request entries for writing into the at least one work queue in the memory responsively to a time division multiplexing schedule.

7 . The system according to claim 6 , wherein the processing circuitry is configured to process packets to be sent over the network timed according to the time division multiplexing schedule responsively to the respective timing data of the respective ones of the work request entries and respective clock values retrieved from the hardware clock.

8 . The device according to claim 1 , wherein the operator is selected from any one or more of the following: a conditional operator; a comparison operator; is less than; is greater than; includes a pattern; includes a periodic pattern; or is in a range.

9 . The device according to claim 1 , wherein the operator includes a mask, which masks out most significant bits of the retrieved clock value.

10 . The device according to claim 1 , wherein respective ones of the work request entries define execution timing for respective work requests included in the respective ones of the work request entries.

11 . The device according to claim 1 , wherein at least some work request entries of the work request entries include references to locations in the memory where respective workloads are stored or should be scattered.

12 . The device according to claim 1 , further comprising a network interface controller and a network interface to share packets with remote devices over a packet data network.

13 . The device according to claim 12 , further comprising a data processing unit (DPU) comprising the network interface controller, the network interface, and at least one processing core.

14 . The device according to claim 12 , wherein:

the network interface is configured to receive packets over the network; and

the processing circuitry is configured to process the received packets timed responsively to respective timing data and respective operators of respective ones of the work request entries.

15 . The system according to claim 12 , wherein:

the processing circuitry is configured to process packets to be sent over the network timed responsively to respective timing data and respective operators of respective ones of the work request entries; and

the network interface is configured to send the packets over the network.

16 . The system according to claim 1 , wherein the processing circuitry comprises a clock controller to discipline the hardware clock responsively to a Precision Time Protocol (PTP).

17 . The system according to claim 1 , wherein the processing circuitry comprises a hardware accelerator to execute at least one of the work requests.

18 . The system according to claim 17 , wherein the hardware accelerator is configured to perform any one or more of the following:

encrypt the workload;

decrypt the workload;

compute a cyclic redundancy check (CRC) of the workload; or

compute a cryptographic hash of the workload.

19 . A system, comprising:

a plurality of peripheral devices to connect with each other over a network, and comprising respective hardware clocks to be synchronized with each other over the network; and

multiple host devices connected to the network via the peripheral devices, the host devices comprising respective processors to run respective software applications to share data with each other over the network, each of the respective software applications being configured to write into a respective work queue a plurality of work request entries including a respective work request entry including a respective operator and respective timing data indicative of a time at which a respective work request should be executed respective ones of the work request entries being fence entries without work requests, the fence entries including timing data that defines execution timing for other work requests, wherein each of the peripheral devices comprises processing circuitry to:

read the respective work request entry from the respective work queue;

retrieve a respective clock value from a respective one of the hardware clocks; and

execute the respective work request with a workload timed responsively to the respective timing data, the respective operator, and the retrieved respective clock value wherein the respective operator defines a condition for executing the respective work request based on the respective timing data and respective retrieved clock value, wherein the processing circuitry of each of the peripheral devices includes a packet processing pipeline to perform a steering decision process based on a packet associated with the respective work request entry.

20 . The system according to claim 19 , wherein the respective software applications are configured to share the data with each other over the network and generate respective work requests entries responsively to a time division multiplexing schedule.

21 . A method comprising:

reading a given work request entry stored with a plurality of work request entries in at least one work queue in a memory, respective ones of the work request entries being fence entries without work requests, the fence entries including timing data that defines execution timing for other work requests, the given work request entry including timing data and an operator, the timing data being indicative of a time at which a work request should be executed;

retrieving a clock value from a hardware clock;

executing the work request with a workload while execution of the work request is timed responsively to the timing data and the operator and the retrieved clock value wherein the operator defines a condition for executing the work request based on the timing data and retrieved clock value; and

performing a steering decision process with a packet processing pipeline based on packets associated with the work request entries.

22 . The peripheral device according to claim 1 , wherein the packet processing pipeline is to perform the steering decision process based on the packets yielding results indicating which of the work request entries the packets belong to.

23 . The peripheral device according to claim 22 , further comprising a network interface controller including a physical layer and a medium access control (MAC) chip and to process the packets which are received, or for sending, over a network.

24 . The peripheral device according to claim 23 , further comprising a clock controller to discipline the hardware clock responsively to a clock synchronization protocol.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 9, 2022
From: LEVI, DOTAN DAVID; MARCOVITCH, DANIEL; MANEVICH, NATAN; WASKO, WOJCIECH; VOKS, IGOR
To: MELLANOX TECHNOLOGIES, LTD.
Reel/Frame 058934/0502 →
Continuity (1)
Related Publication 20230251899A1 · Aug 10, 2023
References Cited (113)
US 6212197B1 · Christensen et al. · 2001 [cited by applicant]
US 6381283B1 · Bhardwaj · 2002 [cited by examiner]
US 6564267B1 · Lindsay · 2003 [cited by applicant]
US 6816492B1 · Turner et al. · 2004 [cited by applicant]
US 6954432B1 · Bychowsky et al. · 2005 [cited by applicant]
US 7224669B2 · Kagan et al. · 2007 [cited by applicant]
US 7245627B2 · Goldenberg et al. · 2007 [cited by applicant]
US 7676597B2 · Kagan et al. · 2010 [cited by applicant]
US 8370675B2 · Kagan · 2013 [cited by applicant]
US 8879552B2 · Zheng et al. · 2014 [cited by applicant]
US 9390056B1 · Noureddine et al. · 2016 [cited by applicant]
US 9397960B2 · Arad et al. · 2016 [cited by applicant]
US 9450846B1 · Huang et al. · 2016 [cited by applicant]
US 9628209B2 · Kingsley · 2017 [cited by applicant]
US 9667446B2 · Worrell · 2017 [cited by applicant]
US 10019333B2 · Regev · 2018 [cited by applicant]
US 10027601B2 · Narkis et al. · 2018 [cited by applicant]
US 10075253B2 · Olofsson et al. · 2018 [cited by applicant]
US 10084559B1 · Devineni et al. · 2018 [cited by applicant]
US 10187400B1 · Castro et al. · 2019 [cited by applicant]
US 10284383B2 · Bloch et al. · 2019 [cited by applicant]
US 10320508B2 · Shimizu et al. · 2019 [cited by applicant]
US 10727966B1 · Izenberg et al. · 2020 [cited by applicant]
US 11271874B2 · Levi et al. · 2022 [cited by applicant]
US 11831403B2 · Lee et al. · 2023 [cited by applicant]
US 20030002483A1 · Zwack · 2003 [cited by applicant]
US 20040174820A1 · Ricciulli · 2004 [cited by applicant]
US 20060007862A1 · Sayeedi et al. · 2006 [cited by applicant]
US 20060064579A1 · Aaron · 2006 [cited by examiner]
US 20060095732A1 · Tran · 2006 [cited by examiner]
US 20060153179A1 · Ho et al. · 2006 [cited by applicant]
US 20080298380A1 · Rittmeyer et al. · 2008 [cited by applicant]
US 20120201248A1 · Aoki · 2012 [cited by applicant]
US 20130027611A1 · Closset · 2013 [cited by applicant]
US 20130173950A1 · Banerjea et al. · 2013 [cited by applicant]
US 20140092918A1 · Jost · 2014 [cited by applicant]
US 20140379714A1 · Hankins · 2014 [cited by applicant]
US 20150046741A1 · Yen et al. · 2015 [cited by applicant]
US 20150055508A1 · Ashida et al. · 2015 [cited by applicant]
US 20150318015A1 · Bose et al. · 2015 [cited by applicant]
US 20160127246A1 · Ashwood-Smith · 2016 [cited by examiner]
US 20170125026A1 · Lawrence · 2017 [cited by applicant]
US 20170171362A1 · Bolotov et al. · 2017 [cited by applicant]
US 20170188092A1 · Lawrence · 2017 [cited by applicant]
US 20170331926A1 · Raveh et al. · 2017 [cited by applicant]
US 20180124812A1 · Thubert et al. · 2018 [cited by applicant]
US 20180191629A1 · Biederman et al. · 2018 [cited by applicant]
US 20180309821A1 · Byers et al. · 2018 [cited by applicant]
US 20180351676A1 · Butterworth et al. · 2018 [cited by applicant]
US 20190056972A1 · Zhou et al. · 2019 [cited by applicant]
US 20190087352A1 · Lee et al. · 2019 [cited by applicant]
US 20190087652A1 · Townsend et al. · 2019 [cited by applicant]
US 20190104075A1 · Li · 2019 [cited by examiner]
US 20190124524A1 · Gormley · 2019 [cited by applicant]
US 20190149365A1 · Chatterjee et al. · 2019 [cited by applicant]
US 20190319730A1 · Webb et al. · 2019 [cited by applicant]
US 20190379714A1 · Levi · 2019 [cited by examiner]
US 20200026656A1 · Liao et al. · 2020 [cited by applicant]
US 20200252320A1 · Zemach et al. · 2020 [cited by applicant]
US 20200259759A1 · Shalev et al. · 2020 [cited by applicant]
US 20210081352A1 · Yang et al. · 2021 [cited by applicant]
US 20210152267A1 · Chan et al. · 2021 [cited by applicant]
US 20210297151A1 · Levi et al. · 2021 [cited by applicant]
US 20210409137A1 · Mula et al. · 2021 [cited by applicant]
US 20220029858A1 · Jiang · 2022 [cited by examiner]
US 20220116132A1 · Kasichainula et al. · 2022 [cited by applicant]
US 20220116473A1 · Levi et al. · 2022 [cited by applicant]
US 20230262281A1 · Kasichainula et al. · 2023 [cited by applicant]
AU 2012216611B2 · 2013 [cited by applicant]
CN 101517944A · 2009 [cited by applicant]
CN 103354989A · 2013 [cited by applicant]
EP 0607412B1 · 1999 [cited by applicant]
JP 2005529523A · 2005 [cited by applicant]
JP 2007042145A · 2007 [cited by applicant]
WO 9418627A2 · 1994 [cited by applicant]
WO 9857497A1 · 1998 [cited by applicant]
WO 2001037489A1 · 2001 [cited by applicant]
WO 2010117359A1 · 2010 [cited by applicant]
WO 2017063322A1 · 2017 [cited by applicant]
IEEE Standard 1588-2008, “IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems”, pp. 1-289, year 2008. [cited by applicant]
Weibel et al., “Implementation and Performance of Time Stamping Techniques”, 2004 Conference on IEEE 1588, pp. 1-29, Sep. 28, 2004. [cited by applicant]
InfiniBand™ Architecture Specification vol. 1, Release 1.2.1,pp. 1-1727, Nov. 2007. [cited by applicant]
Lu et al., “A Fast CRC Update Implementation”, Computer Engineering Laboratory, Electrical Engineering Department, pp. 113-120, Oct. 8, 2003. [cited by applicant]
Mellette et al., “Toward Optical Switching in the Data Center”, IEEE 19th International Conference on High Performance Switching and Routing (HPSR), pp. 1-6, Bucharest, Romania, Jun. 18-20, 2018. [cited by applicant]
Bakopoulos et al., “NEPHELE: an end-to-end scalable and dynamically reconfigurable optical architecture for application-aware SDN cloud datacenters”, IEEE Communications Magazine, vol. 56, issue 2, pp. 1-26, Feb. 2018. [cited by applicant]
O-RAN Alliance, “O-RAN Fronthaul Working Group; Control, User and Synchronization Plane Specification”, ORAN-WG4.CUS.0-v01.00, Technical Specification, pp. 1-189, year 2019. [cited by applicant]
Vattikonda et al., “Practical TDMA for Datacenter Ethernet”, EuroSys conference, Bern, Switzerland, pp. 225-238, Apr. 10-13, 2012. [cited by applicant]
Ericsson AB et al., “Common Public Radio Interface: eCPRI Interface Specification”, V2.0, pp. 1-109, May 10, 2019. [cited by applicant]
Xilinx Inc., “Radio over Ethernet Framer v2.1”, PB056 (v2.1), pp. 1-9, Oct. 30, 2019. [cited by applicant]
Weibel, H., “High Precision Clock Synchronization according to IEEE 1588 Implementation and Performance Issues”, Zurich University of Applied Sciences, pp. 1-9, Jan. 17, 2005. [cited by applicant]
Sanchez-Palencia, J., “[RFC,v3, net-next,00/18] Time based packet transmission”, pp. 1-14, Mar. 7, 2018. [cited by applicant]
IEEE Std 802.1Qaz™, “IEEE Standard for Local and metropolitan area networks-Media Access Control (MAC) Bridges and Virtual Bridged Local Area Networks-Amendment 18: Enhanced Transmission Selection for Bandwidth Sharing … [cited by applicant]
Crupnicoff et al., “Deploying Quality of Service and Congestion Control in InfiniBand-based Data Center Networks” White Paper, Mellanox Technologies Inc, Rev 1.0, pp. 1-19, Oct. 15, 2005. [cited by applicant]
Mathey et al., “Scalable Deadlock-Free Deterministic Minimal-Path Routing Engine for InfiniBand-Based Dragonfly Networks”, IEEE Transactions on Parallel and Distributed Systems, vol. 29, No. 1, pp. 183-197, Jan. 2018. [cited by applicant]
Wikipedia, Precision Time Protocol, pp. 1-9, Apr. 20, 2020. [cited by applicant]
SMPTE Standard, “Professional Media Over Managed IP Networks:Traffic Shaping and Delivery Timing for Video”, The Society of Motion Picture and Television Engineers, pp. 1-17, Nov. 22, 2017. [cited by applicant]
Wikipedia, “Time-Sensitive Networking”, pp. 1-12, Mar. 5, 2020. [cited by applicant]
IEEE Standard 1588-2002, “IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems”, pp. 1-154, year 2002. [cited by applicant]
Levi et al., U.S. Appl. No. 17/315,396, filed May 10, 2021. [cited by applicant]
Levi et al., U.S. Appl. No. 17/359,667, filed Jun. 28, 2021. [cited by applicant]
Mills, “Network Time Protocol (NTP),” RFC 0958, pp. 2-15, Sep. 1985. [cited by applicant]
Mills, “Network Time Protocol (Version 1): Specification and Implementation,” RFC 1059, pp. 2-59, Jul. 1988. [cited by applicant]
Mills, “Internet Time Synchronization: The Network Time Protocol,” IEEE Transactions on Communication, vol. 39, No. 10, pp. 1482-1493, Oct. 1991. [cited by applicant]
Mills, “Network Time Protocol (Version 3): Specification, Implementation and Analysis,” RFC 1305, pp. 1-96, Mar. 1992. [cited by applicant]
Baboescu et al., “Scalable Packet Classification,” Proceedings of SIGCOMM'01, ACM, pp. 199-210, year 2001. [cited by applicant]
Bader et al., “Survey and Comparison of Open Source Time Series Databases,” BTW 2017, Workshopband, Lecture Notes in Informatics (LNI), Gesellschaft fuer Informatik, Bonn, pp. 249-268, year 2017. [cited by applicant]
Davie et al., “An Expedited Forwarding PHB (Per-Hop-Behavior)”, RFC 3246, Network Working Group, pp. 1-32, Mar. 2002. [cited by applicant]
Yau et al., “Development of Situation-Aware Application Software for Ubiquitous Computing Environments,” Proceedings of the 26th Annual International Computer Software and Applications Conference (COMSAC'02), IEEE Compu… [cited by applicant]
U.S. Appl. No. 17/359,667 Office Action dated Jul. 5, 2022. [cited by applicant]
U.S. Appl. No. 17/973,575 Office Action dated Jul. 18, 2024. [cited by applicant]
“Time-Appliance-Project / Open-Time-Server,” Spec Revision No. 1.0, GitHub, Inc., pp. 1-9, last updated Oct. 3, 2022, as downloaded from http://www.opentimeserver.com/. [cited by applicant]
Li et al., “Sundial: Fault-Tolerant Clock Synchronization for Datacenters,” Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation, Open Access, USENIX Association, pp. 1171-1186, Nov. 2… [cited by applicant]
Chinese Notification to Grant Patent Right and Notice of Registration Formalities for application No. 202310034356.6, dated Oct. 16, 2025. [cited by applicant]