Deep neural network training accelerator and operation method thereof
A deep neural network training accelerator includes an operational unit sequentially performing first and second operations on a plurality of input data of a sub-set according to a mini-batch gradient descent, a determination unit determining each of the input data as one of skip data and training data based on a confidence matrix obtained by the first operation, and a control unit controlling the operational unit to skip the second operation with respect to the skip data.
1. A deep neural network training accelerator implemented in hardware and configured to increase learning speed and reduce learning energy, the deep neural network training accelerator comprising:
an operational unit, executed by the deep neural network training accelerator, to sequentially perform first and second operations on a plurality of input data of a sub-set according to a mini-batch gradient descent;
a determination unit, executed by the deep neural network training accelerator, to determine each of the plurality of input data as one of skip data and training data based on a confidence matrix obtained by the first operation; and
a control unit, executed by the deep neural network training accelerator, to control the operational unit to skip the second operation with respect to the skip data,
wherein the determination unit comprises a comparator, the comparator executed by the deep neural network training accelerator to:
compare a largest element among elements of the confidence matrix with a predetermined threshold,
output a low signal corresponding to the skip data to the control unit when a value of the largest element is equal to or greater than the predetermined threshold, and output a high signal corresponding to the training data to the control unit when a value of the largest element is smaller than the predetermined threshold, and
wherein the control unit, executed by the deep neural network training accelerator, to output a parallelization control signal to the operation unit in response to the low signal being received to control the operation unit to skip the second operation on the skip data and perform the second operation on the training data based on the parallelization control signal,
wherein the performing of the second operation comprises:
initializing, by the operational unit, any one operational device corresponding to the skip data among operational devices in response to the parallelization control signal;
reassigning, by the operational unit, a portion of each of the training data assigned to the other operational devices to the any one operational device; and
processing, by the operational unit, the second operation with respect to the training data in parallel using the operational devices after a predetermined time has elapsed from a time at which the first operation is performed.
2. The deep neural network training accelerator of claim 1 , wherein the operational unit performs the second operation with respect to the training data after a predetermined time has elapsed from a time at which the first operation is performed.
3. The deep neural network training accelerator of claim 1 , wherein the first operation is a first training stage of the mini-batch gradient descent, which uses a forward propagation algorithm.
4. The deep neural network training accelerator of claim 1 , wherein the second operation is a second training stage of the mini-batch gradient descent, which sequentially uses a backward propagation algorithm and a weight update algorithm.
5. The deep neural network training accelerator of claim 1 , wherein the control unit parallelizes the second operation with respect to the training data in response to the low signal.
6. The deep neural network training accelerator of claim 1 , wherein a number of the low signals is inversely proportional to an operation time of the second operation.
7. The deep neural network training accelerator of claim 1 , further comprising: an input unit assigning each of the input data arbitrarily selected from total input data to the operational unit; and an output unit summing each variation in weight output through the operational unit to output a variation in output weight corresponding to a gradient of the sub-set.
8. The deep neural network training accelerator of claim 1 , wherein the operational unit has a systolic array structure and comprises operational devices that sequentially perform the first and second operations.
9. The deep neural network training accelerator of claim 8 , wherein the operational unit initializes any one operational device corresponding to the skip data among the operational devices in response to the parallelization control signal applied thereto from the control unit.
10. The deep neural network training accelerator of claim 9 , wherein the operational unit reassigns a portion of the training data assigned to the other operational devices among the operational devices to the any one operational device.
11. The deep neural network training accelerator of claim 8 , wherein the control unit reassigns a plurality of sub-data divided from each of the training data to the operational devices according to a data flow.
12. The deep neural network training accelerator of claim 11 , wherein the data flow refers to a data movement path for reading and storing data.
13. A method of operating a deep neural network training accelerator implemented in hardware and configured to increase learning speed and reduce learning energy, the method comprising:
performing, by an operational unit, first and second operations on a plurality of input data of a sub-set according to a mini-batch gradient descent;
determining, by a determination unit, each of the plurality of input data as one of skip data and training data based on a confidence matrix obtained by the first operation;
controlling, by a control unit, the operational unit to skip the second operation with respect to the skip data;
wherein the determination unit comprises a comparator, configured to the comparator executed by the deep neural network training accelerator to:
compare a largest element among elements of the confidence matrix with a predetermined threshold,
output a low signal corresponding to the skip data to the control unit when a value of the largest element is equal to or greater than the predetermined threshold, and
output a high signal corresponding to the training data to the control unit when a value of the largest element is smaller than the predetermined threshold; and
output, by the control unit, a parallelization control signal to the operation unit in response to the low signal being received to control the operation unit to skip the second operation on the skip data and perform the second operation on the training data based on the parallelization control signal,
wherein the performing of the second operation comprises:
initializing, by the operational unit, any one operational device corresponding to the skip data among operational devices in response to the parallelization control signal;
reassigning, by the operational unit, a portion of each of the training data assigned to the other operational devices to the any one operational device; and
processing, by the operational unit, the second operation with respect to the training data in parallel using the operational devices after a predetermined time has elapsed from a time at which the first operation is performed.
14. The method of claim 13 , wherein the first operation is a first training stage of the mini-batch gradient descent, which uses a forward propagation algorithm.
15. The method of claim 13 , wherein the second operation is a second training stage of the mini-batch gradient descent, which sequentially uses a backward propagation algorithm and a weight update algorithm.