IP Library Granted Patent US 12,547,875
Granted Patent B2
US 12,547,875 · App. 17/072,709 · Granted Feb 10, 2026

Automated setup and communication coordination for training and utilizing massively parallel neural networks

Inventor: Bradley David Safnuk (Victoria, CA)
Assignee: Ford Global Technologies, LLC
G06N3/045G06N3/08
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,547,875
App. No.
17/072,709
Granted
Feb 10, 2026
Kind
B2
Abstract

A method is disclosed for training and utilizing massively parallel neural networks. A distributed computing system may be configured to perform various operations. The distributed computing system may divide a directed acyclic graph (“DAG”) that comprises a plurality of vertices linked in pairwise relationships via a plurality of edges among a plurality of nodes. Each node may comprise a computing device. The distributed computing system may provide a map of the DAG that described a flow of data through the vertices to each of the vertices of the DAG. The distributed computing system may perform a topological sort of the vertices of the DAG and may traverse the DAG.

Claims (54)

1 . A method of distributed computing, the method comprising:

dividing a directed acyclic graph (DAG) among a plurality of nodes, the directed acyclic graph comprising a plurality of vertices linked in pairwise relationships via a plurality of edges, wherein each of the plurality of nodes comprises a computing device;

performing a topological sort of the vertices of the directed acyclic graph, the topological sort comprising a deterministic topological sort controlling an order of operations for traversing the directed acyclic graph;

generating a DAG map of the directed acyclic graph that comprises the deterministic topological sort and that comprises an indication of the order of operations for traversing the directed acyclic graph among the plurality of nodes;

providing the DAG map of the directed acyclic graph to each vertex of the plurality of vertices of the directed acyclic graph such that each vertex of the plurality of vertices receives (i) a description of a flow of data through the plurality of vertices represented by the deterministic topological sort and (ii) a processing order of traversal of the plurality of vertices of the directed acyclic graph, wherein when a particular vertex of the plurality of vertices receives the DAG map, the particular vertex prepares to execute an operation to be performed by the particular vertex as indicated by the DAG map prior to an actual execution of the operation by the particular vertex;

training a neural network associated with a vehicle system by using a distributed computing system to traverse the directed acyclic graph; and

executing the trained neural network to facilitate a human machine interface of the vehicle system by generating one or more predictions associated with the vehicle system to simulate a behavior or an attribute of the vehicle system.

2 . The method of claim 1 , wherein the directed acyclic graph forms a part of the neural network.

3 . The method of claim 2 , further comprising: identifying an edge linking separate nodes; and inserting a data exchange vertex between the separate nodes.

4 . The method of claim 3 , wherein identifying the edge linking separate nodes comprises: tracing an edge from a first vertex to a second vertex, wherein the edge coupled the first vertex to the second vertex; comparing a node of the first vertex to a node of the second vertex; and determining that the edge links separate nodes when the node of the first vertex is different than the node of the second vertex.

5 . The method of claim 4 , wherein the data exchange vertex comprises a recursive directed acyclic graph structure comprising:

a send vertex located on the node of the first vertex; and

a receive vertex located on the node of the second vertex, wherein when the data exchange vertex is activated, both the send vertex and the receive vertex run in concert.

6 . The method of claim 5 , further comprising:

activating a data vertex; and

running the send vertex and the receive vertex in concert while passing a tensor from the send vertex to the receive vertex.

7 . The method of claim 6 , wherein passing the tensor from the send vertex to the receive vertex comprises:

passing a series of communications characterizing the tensor from the send vertex to the receive vertex; and

subsequently passing the tensor from the send vertex to the receive vertex.

8 . The method of claim 7 , wherein passing the series of communications characterizing the tensor from the send vertex to the receive vertex comprises: passing a first value indicating a rank of the tensor from the send vertex to the receive vertex; passing a vector of values indicating a shape of the tensor from the send vertex to the receive vertex; passing at least one second value characterizing data contained in the tensor from the send vertex to the receive vertex; and passing a third value directing tracking of a gradient of the Neural Network.

9 . The method of claim 1 , further comprising:

creating at least one clone directed acyclic graph identical to the directed acyclic graph, the clone directed acyclic graph comprising a plurality of clone vertices;

for each of the clone vertices, identifying a corresponding vertex in the directed acyclic graph;

during training of the directed acyclic graph and the clone directed acyclic graph, calculating aggregate gradient data based on gradient data from each of the clone vertices and its corresponding vertex in the directed acyclic graph; and

updating at least one weight of the directed acyclic graph and the clone directed acyclic graph based on the aggregate gradient data.

10 . The method of claim 1 , wherein one of the plurality of vertices of the directed acyclic graph comprises an entry vertex, the method further comprising:

identifying the nodes underlying the directed acyclic graph;

generating a subordinate directed acyclic graph in the entry vertex, the subordinate directed acyclic graph comprising a plurality of subordinate vertices, each of the plurality of subordinate vertices corresponding to a one of the nodes underlying the directed acyclic graph;

receiving data and metadata at the entry vertex;

delivering the data to a next vertex in the directed acyclic graph; and

communicating the metadata to nodes underlying the directed acyclic graph via the subordinate directed acyclic graph.

11 . A system comprising:

one or more processors; and

one or more memories storing computer-executable instructions that, when executed by the one or more processors, configure the one or more processors to:

divide a directed acyclic graph among a plurality of nodes, the directed acyclic graph comprising a plurality of vertices linked in pairwise relationships via a plurality of edges, wherein each of the plurality of nodes comprises a computing device;

perform a topological sort of the vertices of the directed acyclic graph, the topological sort comprising a deterministic topological sort controlling an order of operations for traversing the directed acyclic graph;

generate a DAG map of the directed acyclic graph that comprises the deterministic topological sort and that comprises an indication of the order of operations for traversing the directed acyclic graph among the plurality of nodes;

provide the DAG map of the directed acyclic graph to each vertex of the plurality of vertices of the directed acyclic graph such that each vertex of the plurality of vertices receives (i) a description of a flow of data through the plurality of vertices represented by the deterministic topological sort and (ii) a processing order of traversal of the plurality of vertices of the directed acyclic graph, wherein when a particular vertex of the plurality of vertices receives the DAG map, the particular vertex prepares to execute an operation to be performed by the particular vertex as indicated by the DAG map prior to an actual execution of the operation by the particular vertex;

train a neural network associated with a vehicle system by traversing the directed acyclic graph; and

execute the trained neural network to facilitate a human machine interface of the vehicle system by generating one or more predictions associated with the vehicle system to simulate a behavior or an attribute of the vehicle system.

12 . The system of claim 11 , wherein the directed acyclic graph forms a part of the neural network.

13 . The system of claim 12 , further comprising: identifying an edge linking separate nodes; and inserting a data exchange vertex between the separate nodes.

14 . The system of claim 13 , wherein identifying the edge linking separate nodes comprises: tracing an edge from a first vertex to a second vertex, wherein the edge coupled the first vertex to the second vertex; comparing a node of the first vertex to a node of the second vertex; and determining that the edge links separate nodes when the node of the first vertex is different than the node of the second vertex.

15 . The system of claim 14 , wherein the data exchange vertex comprises:

a recursive directed acyclic graph structure;

a send vertex located on the node of the first vertex; and

a receive vertex located on the node of the second vertex, wherein when the data exchange vertex is activated, both the send vertex and the receive vertex run in concert.

16 . The system of claim 15 , further comprising:

activating a data vertex; and

running the send vertex and the receive vertex in concert while passing a tensor from the send vertex to the receive vertex.

17 . The system of claim 16 , wherein passing the tensor from the send vertex to the receive vertex comprises:

passing a series of communications characterizing the tensor from the send vertex to the receive vertex; and

subsequently passing the tensor from the send vertex to the receive vertex.

18 . The system of claim 17 , wherein passing the series of communications characterizing the tensor from the send vertex to the receive vertex comprises: passing a first value indicating a rank of the tensor from the send vertex to the receive vertex; passing a vector of values indicating a shape of the tensor from the send vertex to the receive vertex; passing at least one second value characterizing data contained in the tensor from the send vertex to the receive vertex; and passing a third value directing tracking of a gradient of the Neural Network.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 16, 2020
From: SAFNUK, BRADLEY DAVID
To: FORD GLOBAL TECHNOLOGIES, LLC
Reel/Frame 054080/0352 →
Continuity (1)
Related Publication 20220121904A1 · Apr 21, 2022
References Cited (38)
US 8229968B2 · Wang · 2012 [cited by examiner]
US 9811775B2 · Krizhevsky et al. · 2017 [cited by applicant]
US 10210860B1 · Ward et al. · 2019 [cited by applicant]
US 10325340B2 · Wu · 2019 [cited by examiner]
US 10467528B2 · Golovashkin · 2019 [cited by examiner]
US 10540588B2 · Burger et al. · 2020 [cited by applicant]
US 10846245B2 · Islam · 2020 [cited by examiner]
US 11875261B2 · Safnuk · 2024 [cited by applicant]
US 20030220906A1 · Chickering · 2003 [cited by applicant]
US 20170038919A1 · Moss et al. · 2017 [cited by applicant]
US 20180227218A1 · Thubert · 2018 [cited by examiner]
US 20190042946A1 · Sur et al. · 2019 [cited by applicant]
US 20190114534A1 · Teng et al. · 2019 [cited by applicant]
US 20190208039A1 · Song et al. · 2019 [cited by applicant]
US 20200097333A1 · Jain · 2020 [cited by examiner]
CN 110941494A · 2020 [cited by examiner]
CN 110941494A2 · 2020 [cited by examiner]
WO 2016186801A1 · 2016 [cited by applicant]
WO 2019042571A1 · 2019 [cited by applicant]
Mayer, R., Mayer, C., & Laich, L. (Dec. 2017). The tensorflow partitioning and scheduling problem: it's the critical path!. In Proceedings of the 1st Workshop on Distributed Infrastructures for Deep Learning (pp. 1-6). … [cited by examiner]
Costa, Marly Guimaraes Fernandes, et al. “Evaluating the performance of convolutional neural networks with direct acyclic graph architectures in automatic segmentation of breast lesion in US images.” BMC Medical Imaging… [cited by examiner]
Abadi et al., Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv: 1603.04467 (Year: 2016). [cited by examiner]
Suresh et al., Data flow and distributed deep neural network based low latency IoT-edge computation model for big data environment. Engineering Applications of Artificial Intelligence (Year: 2020). [cited by examiner]
Shi, S., et al., “A DAG Model of Synchronous Stochastic Gradient Descent in Distributed Deep Learning”, 2018 IEEE 24th International Conference on Parallel and Distributed Systems (ICPADS), Last revised on Oct. 31, 2018… [cited by applicant]
Eustace, P., et al., “SpiNNaker: A 1-W 18-Core System-on-Chip for Massively-Parallel Neural Network Simulation,” in IEEE Journal of Solid-State Circuits, vol. 48, No. 8, pp. 1943-1953, Aug. 2013, DOI: 10.1109/JSSC.2013.… [cited by applicant]
Takahiro, S., et al., “Structure discovery of deep neural network based on evolutionary algorithms,” 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brisbane, QLD, 2015, pp. 4979-… [cited by applicant]
Hedge, V., et al., “Parallel and Distributed Deep Learning”, (2016), 8 pages. [cited by applicant]
Ben-Nun, T., et al., “Demystifying Parallel and Distributed Deep Learning: An In-depth Concurrency Analysis”, ACM Computing Surveys, vol. 52, Issue 4, Article 65, (Last revised Sep. 15, 2018), 43 pages. arXiv:1802.09941… [cited by applicant]
Zhang, J., et al., “A Parallel Strategy for Convolutional Neural Network Based on Heterogeneous Cluster for Mobile Information System”, Hindawi, Mobile InformationSystems, vol. 2017, Article ID 3824765, published Mar. 2… [cited by applicant]
U.S. Appl. No. 17/072,757, filed Oct. 16, 2020, Non-Final Office Action mailed Aug. 9, 2023, all pages. [cited by applicant]
U.S. Appl. No. 17/072,803, filed Oct. 16, 2020, Notice of Allowance mailed Nov. 4, 2023, all pages. [cited by applicant]
U.S. Appl. No. 17/072,757 , Advisory Action, Mailed On Apr. 8, 2024, 5 pages. [cited by applicant]
U.S. Appl. No. 17/072,757 , Final Office Action, Mailed On Feb. 2, 2024, 15 pages. [cited by applicant]
Lazo , “Visual Intuition on Ring-allreduce for Distributed Deep Learning”, Medium, Available Online at: https://towardsdatascience.com/visual-intuition-on-ring-allreduce-for-distributed-deep-learning-d1f34b4911da, 2019,… [cited by applicant]
Qi , “Intro Distributed Deep Learning”, Available Online at: https://xiandong79.github.io/, Jun. 19, 2017, 6 pages. [cited by applicant]
Zhao et al., “Sparse Allreduce: Efficient Scalable Communication for Power-Law Data”, arXiv:1312.3020, Dec. 11, 2013, 10 pages. [cited by applicant]
U.S. Appl. No. 17/072,757 , Non-Final Office Action, Mailed On Sep. 17, 2024, 16 pages. [cited by applicant]
Naumov et al., “Parallel Depth-First Search for Directed Acyclic Graphs”, IA3'17: Proceedings of the Seventh Workshop on Irregular Applications: Architectures and Algorithms, vol. 4, Nov. 12-17, 2017, pp. 1-8. [cited by applicant]