Fine-grain synchronization in data-parallel jobs
A computer-implemented method and computer processing system are provided. The method includes synchronizing, by a processor, respective ones of a plurality of data parallel workers with respect to an iterative process. The synchronizing step includes individually continuing, by the respective ones of the plurality of data parallel workers, from a current iteration to a subsequent iteration of the iterative process, responsive to a satisfaction of a predetermined condition thereby. The predetermined condition includes individually sending a per-receiver notification from each sending one of the plurality of data parallel workers to each receiving one of the plurality of data parallel workers, responsive to a sending of data there between. The predetermined condition further includes individually sending a per-receiver acknowledgement from the receiving one to the sending one, responsive to a consumption of the data thereby.
1. A computer-implemented method, comprising:
synchronizing, by a processor, respective ones of a plurality of data parallel workers with respect to an iterative process,
wherein said synchronizing step includes individually continuing, by the respective ones of the plurality of data parallel workers, from a current iteration to a subsequent iteration of the iterative process, responsive to a satisfaction of a predetermined condition thereby,
wherein the predetermined condition includes:
individually sending a per-receiver notification from each sending one of the plurality of data parallel workers to each receiving one of the plurality of data parallel workers, responsive to a sending of data there between; and
individually sending a per-receiver acknowledgement from the receiving one to the sending one, responsive to a consumption of the data thereby; and
wherein the each receiving one consumes model parameters, performs a reduce operation, and sends an acknowledgement to the each sending one indicating that a gradient has been consumed;
wherein at least some of the respective ones of the plurality of data parallel workers continue to the subsequent iteration at different times; and
wherein the different times are based on respective times at which the predetermined condition is satisfied by the at least some of the respective ones of the plurality of data parallel workers.
2. A computer program product for data synchronization, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause to computer to perform a method comprising:
synchronizing, by a processor, respective ones of a plurality of data parallel workers with respect to an iterative process,
wherein said synchronizing step includes individually continuing, by the respective ones of the plurality of data parallel workers, from a current iteration to a subsequent iteration of the iterative process, responsive to a satisfaction of a predetermined condition thereby,
wherein the predetermined condition includes:
individually sending a per-receiver notification from each sending one of the plurality of data parallel workers to each receiving one of the plurality of data parallel workers, responsive to a sending of data there between; and
individually sending a per-receiver acknowledgement from the receiving one to the sending one, responsive to a consumption of the data thereby;
wherein the each receiving one consumes model parameters, performs a reduce operation, and sends an acknowledgement to the each sending one indicating that a gradient has been consumed;
wherein at least some of the respective ones of the plurality of data parallel workers continue to the subsequent iteration at different times; and
wherein the different times are based on respective times at which the predetermined condition is satisfied by the at least some of the respective ones of the plurality of data parallel workers.