System and method for designing efficient super resolution deep convolutional neural networks by cascade network training, cascade network trimming, and dilated convolutions
Apparatuses and methods of manufacturing same, systems, and methods for generating a convolutional neural network (CNN) are described. In one aspect, a minimal CNN having, e.g., three or more layers is trained. Cascade training may be performed on the trained CNN to insert one or more intermediate layers until a training error is less than a threshold. When cascade training is complete, cascade network trimming of the CNN output from the cascade training may be performed to improve computational efficiency. To further reduce network parameters, convolutional filters may be replaced with dilated convolutional filters with the same receptive field, followed by additional training/fine-tuning.
1. A method of generating a convolutional neural network (CNN), comprising:
training a CNN having three or more layers; and
performing cascade training on the trained CNN to insert one or more intermediate layers into the CNN until a training error is less than a threshold,
wherein cascade training comprises an iterative process of one or more stages, in which each stage comprises:
training the current CNN;
determining whether the training error is converging; and
if the training error is converging,
inserting a preset number of intermediate layers in the CNN, the weights of each new layer being set to, a predetermined setting; and
starting a new stage.
2. The method of claim 1 , wherein each stage of the cascade training iterative process further comprises:
if the training error is not converging, determining whether the current training error is less than the threshold;
if the current training error is less than the threshold, outputting the cascade trained CNN; and
if the current training error is not less than the threshold, starting a new stage.
3. The method of claim 2 , wherein the predetermined setting of each new layer is randomly initialized with a Gaussian distribution with zero mean and standard deviation U.
4. The method of claim 1 , wherein the CNN is a super-resolution CNN (SRCNN) for processing at least one of images or video.
5. The method of claim 1 , further comprising:
performing cascade network trimming after performing cascade training.
6. The method of claim 5 , wherein cascade network trimming comprises an iterative process of one or more stages, in which each stage comprises:
trimming a set number of layers of the current CNN by reducing dimensions of filters at one or more intermediate layers;
determining whether the training error is converging; and
if the training error is converging, determining whether all of the layers of the current CNN have been trimmed;
if all of the layers of the current CNN have been trimmed, outputting the network trimmed CNN; and
if all of the layers of the current CNN have not been trimmed, starting a new stage.
7. The method of claim 6 , wherein the cascade network trimming iterative process further comprises:
if the training error is not converging, outputting the CNN at a cascade trimming stage where the training error was last converging.
8. The method of claim 1 , wherein the cascade training is performed using dilated convolutional filters.
9. A method of generating a convolutional neural network (CNN), comprising:
training a CNN having three or more layers; and
performing cascade network trimming of the trained CNN,
wherein the cascade network trimming comprises an iterative process of one or more stages, in which each stage comprises:
trimming a set number of layers of the current CNN by reducing dimensions of filters at one or more intermediate layers;
determining whether the training error is converging; and
if the training error is converging, determining whether all of the layers of the current CNN have been trimmed;
if all of the layers of the current CNN have been trimmed, outputting the network trimmed CNN; and
if all of the layers of the current CNN have not been trimmed, starting a new stage.
10. The method of claim 9 , wherein the cascade network trimming iterative process further comprises:
if the training error is not converging, outputting the CNN at a cascade trimming stage where the training error was last converging.
11. The method of claim 9 , wherein each stage of the cascade network trimming iterative process further comprises:
fine tuning before determining whether the current training error is less than the threshold.
12. The method of claim 9 , wherein trimming the set number of layers of the current CNN comprises, for each layer:
trimming filters that do not meet a certain criteria.
13. The method of claim 12 , wherein the certain criteria comprises a measure of relative importance.
14. The method of claim 9 , wherein the CNN is a super-resolution CNN (SRCNN) for processing at least one of images or video.
15. The method of claim 9 , further comprising:
performing cascade training before cascade network trimming.
16. The method of claim 15 , wherein cascade training comprises an iterative process of one or more stages, in which each stage comprises:
training the current CNN;
determining whether the training error is converging; and
if the training error is converging,
inserting a preset number of intermediate layers in the CNN, the weights of each new layer being set to a predetermined setting; and
starting a new stage.
17. The method of claim 16 , wherein each stage of the cascade training iterative process further comprises:
if the training error is not converging, determining whether the current training error is less than the threshold;
if the current training error is less than the threshold, outputting the cascade trained CNN; and
if the current training error is not less than the threshold, starting a new stage.
18. The method of claim 16 , wherein the predetermined setting of each new layer is randomly initialized with a Gaussian distribution with zero mean and standard deviation a.
19. The method of claim 16 , wherein the cascade training is performed using dilated convolutional filters.
20. An apparatus for generating a convolutional neural network (CNN), comprising:
one or more non-transitory computer-readable media; and
at least one processor which, when executing instructions stored on one or more non-transitory computer readable media, performs the steps of:
training a CNN having three or more layers;
performing cascade training on the trained CNN to add one or more intermediate layers until a training error is less than a threshold; and
performing network trimming of the CNN output from the cascade training.
21. The apparatus of claim 20 , wherein cascade training comprises an iterative process of one or more stages, in which each stage comprises:
training the current CNN;
determining whether the training error is converging; and
if the training error is converging,
inserting a preset number of intermediate layers in the CNN, the weights of each new layer being set to a predetermined setting; and
starting a new stage.
22. The apparatus of claim 20 , wherein the cascade training is performed using dilated convolutional filters.
23. The apparatus of claim 20 , wherein cascade network trimming comprises an iterative process of one or more stages, in which each stage comprises:
trimming a set number of layers of the current CNN by reducing dimensions of filters at one or more intermediate layers;
determining whether the training error is converging; and
if the training error is converging, determining whether all of the layers of the current CNN have been trimmed;
if all of the layers of the current CNN have been trimmed, outputting the network trimmed CNN; and
if all of the layers of the current CNN have not been trimmed, starting a new stage.
24. The apparatus of claim 21 , wherein the CNN is a super-resolution CNN (SRCNN) for processing at least one of images or video.
25. A method, comprising:
manufacturing a chipset comprising:
at least one processor which, when executing instructions stored on one or more non-transitory computer readable media, performs the steps of:
training a CNN having three or more layers;
performing cascade training on the trained CNN to add one or more intermediate layers until a training error is less than a threshold; and
performing cascade network trimming of the CNN output from the cascade training; and
the one or more non-transitory computer-readable media which store the instructions.
26. A method of testing an apparatus, comprising:
testing whether the apparatus has at least one processor which, when executing instructions stored on one or more non-transitory computer readable media, performs the steps of:
training a CNN having three or more layers;
performing cascade training on the trained CNN to add one or more intermediate layers until a training error is less than a threshold; and
performing cascade network trimming of the CNN output from the cascade training;
testing whether the apparatus has the one or more non-transitory computer-readable media which store the instructions.