Knowledge distillation for neural networks using multiple augmentation strategies

1. A computer-implemented method for distilling knowledge from source neural networks to distilled neural networks, the computer-implemented method comprising:

generating heavily augmented digital images by augmenting a set of digital images using a heavy augmentation strategy comprising a heavy augmentation range with an upper limit;

sampling a light augmentation range to select augmentation measures to apply to the set of digital images, wherein the light augmentation range comprises an additional upper limit lower than the upper limit of the heavy augmentation range;

generating lightly augmented digital images by augmenting the set of digital images using a light augmentation strategy to apply the augmentation measures selected from the light augmentation range;

determining a first set of classifications for the heavily augmented digital images utilizing a source neural network with parameters learned from the lightly augmented digital images;

determining a second set of classifications for the heavily augmented digital images utilizing a distilled neural network; and

learning parameters of the distilled neural network from the parameters of the source neural network based on comparing the first set of classifications and the second set of classifications.

2. The computer-implemented method of claim 1 , further comprising generating the lightly augmented digital images using the light augmentation strategy by randomly sampling the light augmentation range according to a normal distribution across the light augmentation range.

3. The computer-implemented method of claim 2 , wherein generating the heavily augmented digital images using the heavy augmentation strategy comprises:

sampling the heavy augmentation range to select additional augmentation measures to apply to the set of digital images; and

augmenting the set of digital images by applying the additional augmentation measures.

4. The computer-implemented method of claim 1 , further comprising learning parameters to transfer from the source neural network to the distilled neural network based on comparing classifications of the lightly augmented digital images with ground truth labels utilizing a classification loss function.

5. The computer-implemented method of claim 1 , wherein comparing the first set of classifications with the second set of classifications comprises utilizing a knowledge distillation loss function to transfer learned parameters of the source neural network to the distilled neural network.

6. The computer-implemented method of claim 1 , wherein generating the heavily augmented digital images utilizing the heavy augmentation strategy comprises using an augmentation range that is mutually exclusive from the light augmentation range associated with the light augmentation strategy.

7. The computer-implemented method of claim 1 , wherein learning the parameters of the distilled neural network further comprises comparing the second set of classifications from the distilled neural network with ground truth labels utilizing a classification loss function.

8. A non-transitory computer readable medium comprising instructions that, when executed by at least one processor, cause a computing device to:

sample a first augmentation range to select augmentation measures to apply to a set of digital images, wherein the first augmentation range comprises a first upper limit;

generate a first set of augmented digital images by augmenting the set of digital images using a first augmentation strategy to apply the augmentation measures selected from the first augmentation range;

learn parameters of a source neural network based on comparing classifications of the first set of augmented digital images with ground truth labels;

generate a second set of augmented digital images by augmenting the set of digital images using a second augmentation strategy to apply augmentation measures selected from a second augmentation range with a second upper limit greater than the first upper limit; and

learn parameters of a distilled neural network from the parameters learned for the source neural network utilizing the second set of augmented digital images based on comparing a first set of classifications of the second set of augmented digital images generated by the source neural network with a second set of classifications of the second set of augmented digital images generated by the distilled neural network.

9. The non-transitory computer readable medium of claim 8 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:

generate the second set of augmented digital images by sampling the second augmentation range to select a second set of augmentation measures to apply to the set of digital images.

10. The non-transitory computer readable medium of claim 8 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:

determine a classification for a digital image utilizing the distilled neural network in accordance with the parameters learned for the source neural network; and

provide the digital image and an indication of the classification for display on a client device.

11. The non-transitory computer readable medium of claim 8 , further comprising instructions that, when executed by the at least one processor, cause the computing device to learn the parameters of the distilled neural network based on comparing the classifications of the first set of augmented digital images from the source neural network with classifications of the first set of augmented digital images from the distilled neural network utilizing a knowledge distillation loss function.

12. The non-transitory computer readable medium of claim 8 , further comprising instructions that, when executed by the at least one processor, cause the computing device to learn the parameters of the distilled neural network by utilizing a classification loss function to compare the second set of classifications with the ground truth labels.

13. The non-transitory computer readable medium of claim 8 , further comprising instructions that, when executed by the at least one processor, cause the computing device to learn the parameters of the source neural network by:

generating a third set of classifications for the first set of augmented digital images utilizing the distilled neural network; and

comparing, utilizing a classification loss function, the third set of classifications with the ground truth labels without comparing the second set of classifications with the ground truth labels.

14. A system comprising:

one or more memory devices comprising a set of digital images, a source neural network, and a distilled neural network; and

one or more computing devices that are configured to cause the system to:

sample a light augmentation range to select augmentation measures to apply to the set of digital images, wherein the light augmentation range comprises a first upper limit

generate lightly augmented digital images by augmenting the set of digital images using a light augmentation strategy to apply the augmentation measures selected from the light augmentation range;

learn parameters for the source neural network utilizing the lightly augmented digital images; and

transfer learned parameters from the source neural network to the distilled neural network by:

generating heavily augmented digital images by augmenting the set of digital images using a heavy augmentation strategy to apply augmentation measures selected from a heavy augmentation range with a second upper limit greater than the first upper limit;

determining a first set of classifications utilizing the source neural network and a second set of classifications utilizing the distilled neural network from the heavily augmented digital images; and

learning parameters of the distilled neural network based on comparing the first set of classifications and the second set of classifications.

15. The system of claim 14 , wherein the one or more computing devices are further configured to cause the system to learn the parameters of the source neural network based on comparing classifications of the lightly augmented digital images with ground truth labels utilizing a classification loss function.

16. The system of claim 14 , wherein the one or more computing devices are further configured to cause the system to learn the parameters of the distilled neural network based on comparing the first set of classifications and the second set of classifications utilizing a knowledge distillation loss function.

17. The system of claim 14 , wherein the one or more computing devices are further configured to cause the system to generate the heavily augmented digital images using the heavy augmentation range that is mutually exclusive from the light augmentation range of the lightly augmented digital images.

18. The system of claim 14 , wherein the one or more computing devices are further configured to cause the system to determine the first set of classifications utilizing the source neural network to classify the heavily augmented digital images in accordance with the parameters learned for the source neural network learned from the lightly augmented digital images.

19. The system of claim 14 , wherein the one or more computing devices are further configured to cause the system to:

determine a third set of classifications from the lightly augmented digital images utilizing the distilled neural network; and

learn the parameters of the distilled neural network by further comparing the third set of classifications with the first set of classifications utilizing a knowledge distillation loss function.

20. The system of claim 14 , wherein the one or more computing devices are further configured to cause the system to:

generate the lightly augmented digital images using the light augmentation strategy by rotating digital images within the set of digital images according to a first rotation range; and

generate the heavily augmented digital images using the heavy augmentation strategy by rotating digital images within the set of digital images according to a second rotation range greater than the first rotation range.