Selective knowledge distillation

1 . A computer-implemented method comprising:

providing, to a teacher model in a knowledge distillation training system, first data representing an image to cause the teacher model to generate teacher output data, the teacher output data comprising (i) features of the teacher model and (ii) predictions that indicate whether the image depicts an object of interest;

providing, to a student model in the knowledge distillation training system, second data representing the image to cause the student model to generate student output data that indicate whether the image depicts an object of interest;

determining whether an accuracy of the teacher output data satisfies an accuracy threshold; and

in response to determining that the accuracy of the teacher output data does not satisfy the accuracy threshold:

determining to skip updating the student model using a feature-based distillation loss for the teacher model, the feature-based distillation loss determined based on the features of the teacher model; and

updating the student model using the student output data and ground truth data for the image.

2 . The method of claim 1 , comprising:

providing, to the teacher model in a knowledge distillation training system, third data representing a second image to cause the teacher model to generate second teacher output data, the second teacher output data comprising (i) second features of the teacher model and (ii) second predictions that indicate whether the second image depicts an object of interest;

providing, to the student model in the knowledge distillation training system, fourth data representing the second image to cause the student model to generate second student output data that indicates whether the second image depicts an object of interest;

determining whether a second accuracy of the second teacher output data satisfies the accuracy threshold; and

in response to determining that the second accuracy of the second teacher output data satisfies the accuracy threshold, updating the student model using the second student output data, second ground truth data for the second image, and a second feature-based distillation loss for the teacher model determined using the second teacher output data, the second feature-based distillation loss determined based on the second features of the teacher model.

3 . The method of claim 1 , wherein:

the features of the teacher output data comprise data that represents each of a plurality of bounding boxes;

determining whether the accuracy of the teacher output data satisfies the accuracy threshold comprises determining, for each of the plurality of bounding boxes, whether a corresponding accuracy of a corresponding portion of the teacher output data that represents a bounding box of the plurality of bounding boxes satisfies the accuracy threshold; and

determining to skip updating the student model comprises determining, for the bounding box of the plurality of bounding boxes that have the corresponding accuracy that does not satisfy the accuracy threshold, to skip updating the student model using a corresponding distillation loss for the bounding box.

4 . The method of claim 1 , comprising determining the accuracy of the teacher output data using the ground truth data for the image and the teacher output data.

5 . The method of claim 1 , wherein the predictions of the teacher output data and the student output data comprise predicted labels and bounding box locations.

6 . The method of claim 1 , wherein updating the student model using the student output data and ground truth data for the image comprises:

determining a cross entropy loss using the ground truth and the student output data; and

updating the student model using the cross entropy loss.

7 . The method of claim 1 , wherein the first data and second data comprise different data.

8 . The method of claim 1 , wherein the first data and second data comprise the same data.

9 . The method of claim 1 , wherein the teacher model and the student model are object detection models.

10 . The method of claim 1 , wherein:

determining whether the accuracy of the teacher output data satisfies the accuracy threshold comprises:

determining whether the accuracy for a first portion of the teacher output data satisfies the accuracy threshold; and

determining whether a second accuracy for a second, different portion of the teacher output data satisfies the accuracy threshold; and

determining to skip updating the student model using the feature-based distillation loss for the teacher model comprises determining to skip updating the student model using a first feature-based distillation loss corresponding to the first portion of the features of the teacher output data in response to determining that the accuracy for the first portion of the features of the teacher output data does not satisfy the accuracy threshold, and

wherein the method comprises:

in response to determining that the second accuracy satisfies the accuracy threshold, updating the student model using the second, different portion of the student output data, second ground truth data for the second, different portion, and a second feature-based distillation loss for the teacher model determined using the second portion of the features of the teacher output data.

11 . The computer-implemented method of claim 1 , wherein the features of the teacher model comprise one or more of a weight of the teacher model, a bounding box associated with an image from the first data, or an identified portion of an image from the first data.

12 . A system comprising one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

providing, to a teacher model in a knowledge distillation training system, first data representing an image to cause the teacher model to generate teacher output data, the teacher output data comprising (i) features of the teacher model and (ii) predictions that indicate whether the image depicts an object of interest;

providing, to a student model in the knowledge distillation training system, second data representing the image to cause the student model to generate student output data, the student output data comprising (i) features of the student model and (ii) predictions that indicate whether the image depicts an object of interest;

determining whether an accuracy of the teacher output data satisfies an accuracy threshold; and

in response to determining that the accuracy of the teacher output data does not satisfy the accuracy threshold:

determining to skip updating the student model using a feature-based distillation loss for the teacher model, the feature-based distillation loss determined based on the features of the student and teacher models;

determining to skip updating the student model using a prediction-based distillation loss for the teacher model, the prediction-based distillation loss determined based on the predictions of the student and teacher models; and

updating the student model using the student output data and ground truth data for the image.

13 . The system of claim 12 , wherein the operations further comprise:

providing, to the teacher model in a knowledge distillation training system, third data representing a second image to cause the teacher model to generate second teacher output data, the second teacher output data comprising (i) second features of the teacher model and (ii) second predictions that indicate whether the second image depicts an object of interest;

providing, to the student model in the knowledge distillation training system, fourth data representing the second image to cause the student model to generate second student output data, the second student output data comprising (i) second features of the student model and (ii) second predictions that indicate whether the second image depicts an object of interest;

determining whether a second accuracy of the second teacher output data satisfies the accuracy threshold; and

in response to determining that the second accuracy of the second teacher output data satisfies the accuracy threshold, updating the student model using the second student output data, second ground truth data for the second image, a second prediction-based distillation loss, and a second feature-based distillation loss,

wherein the second prediction-based distillation loss is determined based on the second predictions of the student and teacher models, and the second feature-based distillation loss is determined based on the second features of the student and the teacher model.

14 . The system of claim 12 , wherein:

the features of the teacher output data comprises data that represents each of a plurality of bounding boxes;

determining whether the accuracy of the teacher output data satisfies the accuracy threshold comprises determining, for each of the plurality of bounding boxes, whether a corresponding accuracy of a corresponding portion of the features of the teacher output data that represents the bounding box satisfies the accuracy threshold; and

determining to skip updating the student model comprises determining, for the bounding boxes from the plurality of bounding boxes that have the corresponding accuracy that does not satisfy the accuracy threshold, to skip updating the student model using a corresponding distillation loss for the bounding box.

15 . The system of claim 12 , wherein the operations further comprise determining the accuracy of the teacher output data using the ground truth data for the image and the teacher output data.

16 . The system of claim 12 , wherein the features of the teacher output data and the student output data comprise predicted labels and bounding box locations.

17 . The system of claim 12 , wherein updating the student model using the student output data and ground truth data for the image comprises: determining a cross entropy loss using the ground truth and the student output data; and updating the student model using the cross entropy loss.

18 . The system of claim 12 , wherein the first data and second data comprise different data.

19 . The system of claim 12 , wherein the teacher model and the student model are object detection models.

20 . One or more non-transitory computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

providing, to a teacher model in a knowledge distillation training system, first data representing an image to cause the teacher model to generate teacher output data, the teacher output data comprising (i) features of the teacher model and (ii) predictions that indicate whether the image depicts an object of interest;

providing, to a student model in the knowledge distillation training system, second data representing the image to cause the student model to generate student output data, the student output data comprising (i) features of the student model and (ii) predictions that indicate whether the image depicts an object of interest;

determining whether an accuracy of the teacher output data satisfies an accuracy threshold; and

in response to determining that the accuracy of the teacher output data does not satisfy the accuracy threshold:

determining to skip updating the student model using a feature-based distillation loss for the teacher model, the feature-based distillation loss determined based on the features of the student and teacher models; and

updating the student model using the student output data and ground truth data for the image.