Fine-grain synchronization in data-parallel jobs for distributed machine learning
A computer-implemented method and computer processing system are provided. The method includes synchronizing, by a processor, respective ones of a plurality of data parallel workers with respect to an iterative distributed machine learning process. The synchronizing step includes individually continuing, by the respective ones of the plurality of data parallel workers, from a current iteration to a subsequent iteration of the iterative distributed machine learning process, responsive to a satisfaction of a predetermined condition thereby. The predetermined condition includes individually sending a per-receiver notification from each sending one of the plurality of data parallel workers to each receiving one of the plurality of data parallel workers, responsive to a sending of data there between. The predetermined condition further includes individually sending a per-receiver acknowledgement from the receiving one to the sending one, responsive to a consumption of the data thereby.
1. A computer-implemented method, comprising:
synchronizing, by a processor, respective ones of a plurality of data parallel workers with respect to an iterative distributed machine learning process,
wherein said synchronizing step includes individually continuing, by the respective ones of the plurality of data parallel workers, from a current iteration to a subsequent iteration of the iterative distributive machine learning process, responsive to a satisfaction of a predetermined condition thereby, and
wherein the predetermined condition includes:
individually sending a per-receiver notification from each sending one of the plurality of data parallel workers to each receiving one of the plurality of data parallel workers, responsive to a sending of data there between; and
individually sending a per-receiver acknowledgement from the receiving one to the sending one, responsive to a consumption of model parameters of the data thereby;
wherein at least some of the respective ones of the plurality of data parallel workers continue to the subsequent iteration at different times; and
wherein the different times are based on respective times at which the predetermined condition is satisfied by the at least some of the respective ones of the plurality of data parallel workers.
2. A computer program product for data synchronization, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:
synchronizing, by a processor, respective ones of a plurality of data parallel workers with respect to an iterative distributed machine learning process,
wherein said synchronizing step includes individually continuing, by the respective ones of the plurality of data parallel workers, from a current iteration to a subsequent iteration of the iterative distributed machine learning process, responsive to a satisfaction of a predetermined condition thereby, and
wherein the predetermined condition includes:
individually sending a per-receiver notification from each sending one of the plurality of data parallel workers to each receiving one of the plurality of data parallel workers, responsive to a sending of data there between; and
individually sending a per-receiver acknowledgement from the receiving one to the sending one, responsive to a consumption of model parameters of the data thereby;
wherein at least some of the respective ones of the plurality of data parallel workers continue to the subsequent iteration at different times; and
wherein the different times are based on respective times at which the predetermined condition is satisfied by the at least some of the respective ones of the plurality of data parallel workers.