Parallelized machine learning with distributed lockless training
Systems and methods are disclosed for providing distributed learning over a plurality of parallel machine network nodes by allocating a per-sender receive queue at every machine network node and performing distributed in-memory training; and training each unit replica and maintaining multiple copies of the unit replica being trained, wherein all unit replicas train, receive unit updates and merge in parallel in a peer-to-peer fashion, wherein each receiving machine network node merges updates at later point in time without interruption and wherein the propagating and synchronizing unit replica updates are lockless and asynchronous.
1. A method for providing distributed learning over a plurality of parallel machine network nodes, comprising:
providing a vector object library with vector abstractions to create unit parameters or gradients to be synchronized across the parallel machine network nodes;
training the parallel machine network nodes in a peer-to-peer fashion and allocating a per-sender receive queue at every machine network node and performing distributed in-memory training;
training each unit replica and maintaining multiple copies of the unit replica being trained, wherein all unit replicas train, receive unit updates and merge in parallel in a peer-to-peer fashion, wherein each receiving machine network node merges updates at a later point in time without interruption; and
propagating and synchronizing unit replica updates as lockless and asynchronous operations using wait-free queues, wherein multiple per-sender queues are combined with remote direct memory access (RDMA) circuit.
2. The method of claim 1 , wherein each queue stores received units and wherein each unit merges in a fully asynchronous fashion.
3. The method of claim 1 , comprising receiving a unit from another machine into a queue or a buffer.
4. The method of claim 1 , comprising using hardware support that directly maps remote memory.
5. The method of claim 1 , wherein when parallel unit replicas train their units and send their updates, incoming units are stored in receive queues on every network node.
6. The method of claim 1 , wherein at the end of every iteration, each unit processes the incoming units and updates its current unit.
7. The method of claim 1 , wherein unit replicas send updates and proceed to a next iteration.
8. The method of claim 1 , wherein unit replicas train in parallel and send updates to one another, in a fully asynchronous fashion.
9. The method of claim 1 , wherein a receiver network node only looks up the receive queue at the end of an iteration cycle, and performs a merge of all received updates.
10. The method of claim 2 , wherein no replicas wait for one-another to merge or receive units.
11. A plurality of learning machine network nodes, comprising:
a processor in communication with a plurality of parallel machine network nodes;
computer readable code for providing a vector object library with vector abstractions to create unit parameters or gradients to be synchronized across the parallel machine network nodes;
computer readable code for allocating a per-sender receive queue at every machine network node and performing distributed in-memory training; and
computer readable code for training a unit replica and maintaining multiple copies of the unit replica being trained, wherein all unit replicas train, receive unit updates and merge in parallel in a peer-to-peer fashion, wherein each receiving machine network node merges updates at later point in time without interruption and wherein the propagating and synchronizing unit replica updates are lockless and asynchronous operations using wait-free queues, wherein multiple per-sender queues are combined with remote direct memory access (RDMA) circuit.
12. The network nodes of claim 11 , wherein each queue stores received units, and unit merges in a fully asynchronous fashion.
13. The network nodes of claim 11 , wherein no replicas wait for one-another to merge or receive units.
14. The network nodes of claim 11 , comprising removing a burden of receiving a unit from another machine into a queue or a buffer.
15. The network nodes of claim 11 , comprising using hardware support that directly maps remote memory to receive and store an update in a receive queue.
16. The network nodes of claim 11 , wherein when parallel unit replicas train their units and send their updates, incoming units are stored in receive queues on every network node.
17. The network nodes of claim 11 , wherein at the end of every iteration, each unit processes the incoming units and updates its current unit.
18. The network nodes of claim 11 , wherein unit replicas send updates and proceed to a next iteration.
19. The network nodes of claim 11 , wherein unit replicas train in parallel and send updates to one another, in a fully asynchronous fashion.