Method and apparatus for performing distributed training on deep learning model, device and storage medium
The present disclosure provides a method and apparatus for performing distributed training on a deep learning model. The method may include: generating a distributed computation view based on data information of a to-be-trained deep learning model; generating a cluster resource view based on property information of a cluster hardware resource corresponding to the to-be-trained deep learning model; determining a target segmentation strategy of a distributed training task based on the distributed computation view and the cluster resource view; and performing distributed training on the to-be-trained deep learning model based on the target segmentation strategy.
1 . A computer-implemented method for performing distributed training on a deep learning model, comprising:
generating, by computer, a distributed computation view based on data information from a data flow graph of a to-be-trained deep learning model, comprising
acquiring an operator and a tensor of a to-be-trained deep learning model,
acquiring a distributed property of the operator and a distributed property of the tensor, and
generating the distributed computation view based on the operator, the distributed property of the operator, the tensor, and the distributed property of the tensor, wherein the distributed properties comprise at least: logical process topology information, segmentation mapping information of each dimension of the tensor, slice size information of the each dimension of the tensor, and slice physical device placement information;
generating, by computer, a cluster resource view based on property information of a cluster hardware resource corresponding to the to-be-trained deep learning model;
determining, by computer, a target segmentation strategy of a distributed training task based on the distributed computation view and the cluster resource view, comprising:
generating a different segmentation strategy based on the distributed computation view and the cluster resource view, the segmentation strategy comprising a segmentation approach of the deep learning model and a resource configuration approach of the cluster hardware resource,
calculating an overhead corresponding to each segmentation strategy, wherein the overhead comprises a running time overhead and a storage space occupation overhead, and the running time overhead comprises an operator overhead and a communication overhead,
determining a segmentation strategy with a minimum overhead as the target segmentation strategy of the distributed training task; and
performing, by computer, distributed training on the to-be-trained deep learning model based on the target segmentation strategy.
2 . The method according to claim 1 , further comprising:
inserting, in response to detecting that a definition for an input tensor and an output tensor in the distributed computation view does not match a definition for an operator using the input tensor and the output tensor, a recombination transformation operator; and
inserting, in response to detecting that a device corresponding to the input tensor and the output tensor in the distributed computation view is not the same as a device corresponding to the operator using the input tensor and the output tensor, a communication operator.
3 . The method according to claim 1 , wherein generating a cluster resource view based on property information of a cluster hardware resource corresponding to the to-be-trained deep learning model comprises:
acquiring topology information between devices in the cluster hardware resource corresponding to the to-be-trained deep learning model and processing capability information of the devices in the cluster hardware resource, the processing capability information comprising at least one of: processor information, bandwidth information, or video memory information; and
generating the cluster resource view based on the topology information and the processing capability information.
4 . The method according to claim 1 , wherein performing distributed training on the to-be-trained deep learning model based on the target segmentation strategy comprises:
performing, based on the target segmentation strategy, the distributed training on the to-be-trained deep learning model by using an asynchronous pipeline operation mechanism.
5 . The method according to claim 4 , further comprising:
triggering, during the distributed training, an elastic resource management mechanism in response to an environment of the cluster hardware resource changing, to re-determine the target segmentation strategy.
6 . The method according to claim 1 , wherein the distributed property of a tensor is different from the distributed property corresponding to the operator using the tensor.
7 . The method according to claim 1 , wherein calculating an overhead corresponding to each segmentation strategy further comprises using a reduction algorithm combining dynamic information and static information to calculate the overhead.
8 . The method according to claim 1 , wherein the information of a cluster hardware resource comprises information of heterogenous devices in the cluster.
9 . An electronic device, comprising:
at least one processor; and
a memory, communicatively connected with the at least one processor,
wherein the memory stores an instruction executable by the at least one processor, and the instruction when executed by the at least one processor, causes the at least one processor to perform operations, the operations comprising:
generating a distributed computation view based on data information from a data flow graph of a to-be-trained deep learning model, comprising:
acquiring an operator and a tensor of a to-be-trained deep learning model,
acquiring a distributed property of the operator and a distributed property of the tensor, and
generating the distributed computation view based on the operator, the distributed property of the operator, the tensor, and the distributed property of the tensor, wherein the distributed properties comprise at least: logical process topology information, segmentation mapping information of each dimension of the tensor, slice size information of the each dimension of the tensor, and slice physical device placement information;
generating a cluster resource view based on property information of a cluster hardware resource corresponding to the to-be-trained deep learning model;
determining a target segmentation strategy of a distributed training task based on the distributed computation view and the cluster resource view, comprising:
generating a different segmentation strategy based on the distributed computation view and the cluster resource view, the segmentation strategy comprising a segmentation approach of the deep learning model and a resource configuration approach of the cluster hardware resource,
calculating an overhead corresponding to each segmentation strategy, wherein the overhead comprises a running time overhead and a storage space occupation overhead, and the running time overhead comprises an operator overhead and a communication overhead,
determining a segmentation strategy with a minimum overhead as the target segmentation strategy of the distributed training task; and
performing distributed training on the to-be-trained deep learning model based on the target segmentation strategy.
10 . The electronic device according to claim 9 , wherein the operations further comprise:
inserting, in response to detecting that a definition for an input tensor and an output tensor in the distributed computation view does not match a definition for an operator using the input tensor and the output tensor, a recombination transformation operator; and
inserting, in response to detecting that a device corresponding to the input tensor and the output tensor in the distributed computation view is not the same as a device corresponding to the operator using the input tensor and the output tensor, a communication operator.
11 . The electronic device according to claim 9 , wherein generating a cluster resource view based on property information of a cluster hardware resource corresponding to the to-be-trained deep learning model comprises:
acquiring topology information between devices in the cluster hardware resource corresponding to the to-be-trained deep learning model and processing capability information of the devices in the cluster hardware resource, the processing capability information comprising at least one of: processor information, bandwidth information, or video memory information; and
generating the cluster resource view based on the topology information and the processing capability information.
12 . The electronic device according to claim 9 , wherein performing distributed training on the to-be-trained deep learning model based on the target segmentation strategy comprises:
performing, based on the target segmentation strategy, the distributed training on the to-be-trained deep learning model by using an asynchronous pipeline operation mechanism.
13 . The electronic device according to claim 12 , wherein the operations further comprise:
triggering, during the distributed training, an elastic resource management mechanism in response to an environment of the cluster hardware resource changing, to re-determine the target segmentation strategy.
14 . The electronic device according to claim 9 , wherein the distributed property of a tensor is different from the distributed property corresponding to the operator using the tensor.
15 . The electronic device according to claim 9 , wherein calculating an overhead corresponding to each segmentation strategy further comprises using a reduction algorithm combining dynamic information and static information to calculate the overhead.
16 . The electronic device according to claim 9 , wherein the distributed property of a tensor is different from the distributed property corresponding to the operator using the tensor.
17 . A non-transitory computer readable storage medium, storing a computer instruction, wherein the computer instruction when executed by a processor, causes the processor to perform operations, the operations comprising:
generating, by the processor, a distributed computation view based on data information from a data flow graph of a to-be-trained deep learning model, comprising
acquiring, by the processor, an operator and a tensor of a to-be-trained deep learning model,
acquiring, by the processor, a distributed property of the operator and a distributed property of the tensor, and
generating, by the processor, the distributed computation view based on the operator, the distributed property of the operator, the tensor, and the distributed property of the tensor, wherein the distributed properties comprise at least: logical process topology information, segmentation mapping information of each dimension of the tensor, slice size information of the each dimension of the tensor, and slice physical device placement information;
generating, by the processor, a cluster resource view based on property information of a cluster hardware resource corresponding to the to-be-trained deep learning model;
determining, by the processor, a target segmentation strategy of a distributed training task based on the distributed computation view and the cluster resource view, comprising:
generating, by the processor, a different segmentation strategy based on the distributed computation view and the cluster resource view, the segmentation strategy comprising a segmentation approach of the deep learning model and a resource configuration approach of the cluster hardware resource,
calculating, by the processor, an overhead corresponding to each segmentation strategy, wherein the overhead comprises a running time overhead and a storage space occupation overhead, and the running time overhead comprises an operator overhead and a communication overhead,
determining, by the processor, a segmentation strategy with a minimum overhead as the target segmentation strategy of the distributed training task; and
performing, by the processor, distributed training on the to-be-trained deep learning model based on the target segmentation strategy.