Prediction of classification of an unknown input by a trained neural network

1 . A method of classifying an object by a trained neural network (NN) trained to classify input data into one of a set of candidate classes, the method comprising:

receiving unknown input data that is not classified;

operating the trained NN on the unknown input data;

receiving, by the trained NN, output indicating that the trained NN cannot classify the unknown input data because a prediction threshold is not met;

responsive to receiving the output and for each class of a set of candidate classes for which the trained NN has been trained:

accessing the trained NN in a state it is in before retraining;

retraining, using backpropagation, the trained NN from its trained state until a prediction can be made, by the retrained NN and because the prediction threshold is met for a current class, for classifying the unknown input data to the current class by an inference procedure;

determining an amount of effort exerted for retraining the trained NN to classify the unknown input data to the current class including counting a number of retraining iterations performed; and

selecting a next class of the candidate classes to be the current class after determining the amount of effort;

selecting a class of the candidate classes for which a least amount of effort was exerted in retraining the trained NN by selecting the class for which a fewest number of retraining iterations were performed; and

outputting the selected class as the predicted class to which the unknown input data is predicted to be classified.

2 . The method of claim 1 , wherein retraining the trained NN comprises:

accessing a copy of the trained NN as trained; and

iteratively performing backpropagation in the trained NN as copied using the class as a target class until the prediction can be made for classifying the unknown input data to the class.

3 . The method of claim 1 , wherein:

retraining the trained NN further comprises determining when the prediction can be made for classifying the unknown input data to the class;

the inference procedure is a forward propagation procedure that outputs a predicted class, and

it is determined the prediction can be made for classifying the unknown input data to the class when the inference procedure outputs the class as the predicted class.

4 . The method of claim 3 , wherein the inference procedure further outputs a prediction strength associated with the predicted class that indicates a level of confidence in the predicted class, and wherein determining when the prediction can be made for classifying the unknown input data to the class includes determining whether the output prediction strength satisfies a threshold criterion.

5 . The method of claim 4 , wherein the predicted class is associated with a neuron of the trained NN after retraining for which a highest value was output by the forward propagation and the prediction strength is a difference between the highest value and a second highest value output for another neuron of the trained NN after retraining by the forward propagation, and the threshold criterion is satisfied when the difference exceeds a predetermined threshold.

6 . The method of claim 2 , wherein the copy of the trained NN includes a copy of weights associated with the trained NN once trained and before weights are changed by backpropagation during retraining.

7 . A computer system for classifying an object by a trained neural network (NN) trained to classify input data into one of a set of candidate classes, the computer system comprising:

a memory configured to store a plurality of programmable instructions; and

a processing device in communication with the memory, wherein the processing device, upon execution of the plurality of programmable instructions is configured to:

receive unknown input data that is not classified;

operate the trained NN on the unknown input data;

receive, by the trained NN, output indicating that the trained NN cannot classify the unknown input data because a prediction threshold is not met;

responsive to receiving the output and for each class of a set of candidate classes for which the trained NN has been trained:

access the trained NN in a state it is in before retraining;

retrain, using backpropagation, the trained NN from its trained state until a prediction can be made, by the retrained NN and because the prediction threshold is met for a current class of the candidate classes, for classifying the unknown input data to the current class by an inference procedure;

determine an amount of effort exerted for retraining the trained NN to classify the unknown input data to the current class including counting a number of retraining iterations performed; and

select a next class of the candidate classes to be the current class after determining the amount of effort;

select a class of the candidate classes for which a least amount of effort was exerted by selecting the class for which a fewest number of retraining iterations were performed; and

output the selected class as the predicted class to which the unknown input data is predicted to be classified.

8 . The computer system of claim 7 , wherein retraining the trained NN comprises:

accessing a copy of the trained NN as trained; and

iteratively performing backpropagation in the trained NN as copied using the class as a target class until the prediction can be made for classifying the unknown input data to the class.

9 . The computer system of claim 7 , wherein:

retraining the trained NN further comprises determining when the prediction can be made for classifying the unknown input data to the class;

the inference procedure is a forward propagation procedure that outputs a predicted class, and

it is determined the prediction can be made for classifying the unknown input data to the class when the inference procedure outputs the class as the predicted class.

10 . The computer system of claim 9 , wherein the inference procedure further outputs a prediction strength associated with the predicted class that indicates a level of confidence in the predicted class, and wherein determining when the prediction can be made for classifying the unknown input data to the class includes determining whether the output prediction strength satisfies a threshold criterion.

11 . The computer system of claim 10 , wherein the predicted class is associated with a neuron of the trained NN after retraining for which a highest value was output by the forward propagation and the prediction strength is a difference between the highest value and a second highest value output for another neuron of the trained NN after retraining by the forward propagation, and the threshold criterion is satisfied when the difference exceeds a predetermined threshold.

12 . The computer system of claim 8 , wherein the copy of the trained NN includes a copy of weights associated with the trained NN once trained and before weights are changed by backpropagation during retraining.

13 . A non-transitory computer readable storage medium and one or more computer programs embedded therein, the computer programs comprising instructions, which when executed by a computer system, cause the computer system to:

receive unknown input data that is not classified;

operate a trained neural network (NN) on the unknown input data;

receive, by the trained NN, output indicating that the trained NN cannot classify the unknown input data because a prediction threshold is not met;

responsive to receiving the output and for each class of a set of candidate classes for which the trained NN has been trained:

access the trained NN in a state it is in before retraining;

retrain, using backpropagation, the trained NN from its trained state until a prediction can be made, by the retrained NN and because the prediction threshold is met for a current class, for classifying the unknown input data to the current class by an inference procedure;

determine an amount of effort exerted for retraining the trained NN to classify the unknown input data to the current class including counting a number of retraining iterations performed; and

select a next class of the candidate classes to be the current class after determining the amount of effort;

select a class of the candidate classes for which a least amount of effort was exerted by selecting the class for which a fewest number of retraining iterations were performed; and

output the selected class as the predicted class to which the unknown input data is predicted to be classified.

14 . The non-transitory computer readable storage medium of claim 13 , wherein retraining the network comprises, for each class of the candidate classes:

accessing a copy of the trained NN as trained; and

iteratively performing backpropagation in the trained NN as copied using the class as a target class until the prediction can be made for classifying the unknown input data to the class.