Training task optimization system, training task optimization method and non-transitory computer readable medium for operating the same

1. A training task optimization system, comprising:

a processor, configured to receive training environment information of a training task, wherein the training environment information carries information corresponding to training samples in the training task,

wherein a neural network comprises a plurality of layers, and the processor is configured to calculate a first required space that represents spaces allocated to the training samples and output data corresponding to each of the plurality of layers,

the processor is configured to obtain model parameters from the neural network and calculate a second required space that represents spaces allocated to the model parameters and gradients,

the processor is configured to calculate an available memory space by deducting the first required space and the second required space from memory factors,

each of the layers corresponds to a plurality of algorithm candidates,

the processor is configured to calculate a third required space that represents buffers prepared for algorithms in the layers, and the third required space is smaller than the available memory space, thereby calculating, by the processor, a memory distribution for the training task,

the processor is configured to determine a mini-batch size that is fit to the memory distribution, wherein the mini-batch size is an amount of the training samples being fetched in one mini-batch,

in response to the training environment information, the processor is configured to output the mini-batch size for execution of the training task, wherein the training task is executed according to the mini-batch size so as to optimize training efficiency.

2. The training task optimization system of claim 1 , further comprising:

a memory, configured to store data of correspondences between a plurality of mini-batch sizes and computation efficiencies, wherein the mini-batch size is selected from the memory based on the computation efficiencies.

3. The training task optimization system of claim 1 , further comprising:

an interaction interface, configured to input the training environment information, wherein the training environment information further carries information corresponding to the memory factors and the neural network.

4. The training task optimization system of claim 1 , wherein the training environment information further carries information corresponding to a plurality of graphic processors provided for the training task and an expected efficiency, and

the processor is configured to determine a number of the plurality of graphic processors to be employed based on the expected efficiency, and output the number of the graphic processors for execution of the training task.

5. The training task optimization system of claim 1 , wherein the training environment information further carries information corresponding to a number of graphic processors provided for the training task and a plurality of parameter servers provided for the training task and a network bandwidth, and

the processor is configured to determine a number of the plurality of parameter servers to be employed based on the number of graphic processors and the network bandwidth, and output the number of the parameter servers for execution of the training task.

6. A training task optimization method, executed by a processor, comprising:

receiving training environment information of a training task, wherein the training environment information carries information corresponding training samples in the training task, wherein a neural network comprises a plurality of layers, each of the layers corresponds to a plurality of algorithm candidates;

calculating a first required space that represents spaces allocated to the training samples and output data corresponding to each of the plurality of layers;

obtaining model parameters from the neural network and calculating a second required space that represents spaces allocated to the model parameters and gradients;

calculating an available memory space by deducting the first required space and the second required space from memory factors;

calculating a third required space that represents buffers prepared for algorithms in the layers, and the third required space is smaller than the available memory space, thereby calculating a memory distribution for the training task;

determining a mini-batch size that is fit to the memory distribution, wherein the mini-batch size is an amount of the training samples being fetched in one mini-batch; and

in response to the training environment information, outputting the mini-batch size for execution of the training task, wherein the training task is executed according to the mini-batch size so as to optimize training efficiency.

7. The training task optimization method of claim 6 , further comprising:

retrieving correspondences between a plurality of mini-batch sizes and computation efficiencies, wherein the mini-batch size is selected based on the computation efficiencies.

8. The training task optimization method of claim 6 , wherein the training environment information further carries information corresponding to a plurality of graphic processors provided for the training task and an expected efficiency, and the training task optimization method further comprising:

determining a number of the plurality of graphic processors to be employed based on the expected efficiency; and

outputting the number of the plurality of graphic processors for execution of the training task.

9. The training task optimization method of claim 6 , wherein the training environment information further carries information corresponding to a number of graphic processors provided for the training task, a plurality of parameter servers provided for the training task and a network bandwidth, and the training task optimization method further comprising:

determining a number of the plurality of parameter servers to be employed based on the number of graphic processors and the network bandwidth; and

outputting the number of the plurality of parameter servers for execution of the training task.

10. A non-transitory computer readable medium for training task optimization, and the non-transitory computer readable medium containing program instructions for causing a processor to perform a method of:

receiving training environment information of a training task, wherein the training environment information carries information corresponding to training samples in the training task, wherein a neural network comprises a plurality of layers, each of the layers corresponds to a plurality of algorithm candidates;

calculating a first required space that represents spaces allocated to the training samples and output data corresponding to each of the plurality of layers;

obtaining model parameters from the neural network and calculating a second required space that represents spaces allocated to the model parameters and gradients;

calculating an available memory space by deducting the first required space and the second required space from memory factors;

calculating a third required space that represents buffers prepared for algorithms in the layers, and the third required space is smaller than the available memory space, thereby calculating a memory distribution for the training task;

determining a mini-batch sizes that is fit to the memory distribution, wherein the mini-batch size is an amount of the training samples being fetched in one mini-batch; and

in response to the training environment information, outputting the mini-batch size for execution of the training task, wherein the training task is executed according to the mini-batch size so as to optimize training efficiency.