Systems, techniques, and interfaces for obtaining and annotating training instances
A previously trained classification model associated with the machine learning system is configured to process an input to generate i) a first prediction that represents a characteristic associated with the input, and ii) a representation of accuracy associated with the prediction. A retraining subsystem is configured to receive the input, the first prediction, and the representation of accuracy. The retraining subsystem processes the input to generate a prediction representing a characteristic. A sufficiency of certainty of the first prediction is determined based on at least the input, the first prediction, the measure of accuracy, and the second prediction. Based at least on the determined sufficiency the retraining subsystem causes the machine learning system to be automatically retrained, be retrained using the input with active learning or not retrained.
1. A method for retraining a machine learning system, the method comprising:
processing, by a previously trained classification model associated with the machine learning system, an input to generate i) a first prediction that represents a characteristic associated with the input, and ii) a representation of accuracy associated with the prediction;
receiving, by a retraining subsystem associated with the machine learning system, the input, the first prediction, and the representation of accuracy;
comparing, by a selection agent of the retraining subsystem, the representation of accuracy to a first threshold value and a second threshold value; and
based at least on the comparison:
i) causing, by the retraining subsystem, the machine learning system to be automatically retrained using the input and the first prediction in the case the representation of accuracy is greater than the first threshold value;
ii) causing, by the retraining subsystem, the machine learning system to be retrained using the input with active learning in the case the representation of accuracy is less than the first threshold value and greater than the second threshold value; and
iii) causing, by the retraining subsystem, the machine learning system not to be retrained using the input in the case the representation of accuracy is less than the second threshold value.
2. The method of claim 1 , wherein causing the machine learning system to be retrained using the input with active learning further comprises:
transmitting, by at least one computing device comprised in the machine learning system, to a user device configured with a graphical user interface, information associated with the input and the first prediction to present a representation of the input and the characteristic via the graphical user interface to a user.
3. The method of claim 2 , further comprising:
training, by the at least one computing device, the machine learning system as a function of an acceptance or rejection of the characteristic received from the user device.
4. The method of claim 2 , wherein the information associated with the input is a copy of the input.
5. The method of claim 1 , wherein the representation of accuracy associated with the first prediction is a Softmax confidence value.
6. The method of claim 2 , the method further comprising receiving, upon a single action of a user operating the user device, an acceptance or a rejection of the characteristic.
7. The method of claim 1 , wherein the retraining subsystem comprises at least one of a second classification model.
8. The method of claim 1 , wherein determining the sufficiency of certainty of the first prediction includes determining that the first prediction is out of domain.
9. The method of claim 1 , wherein the input is received from a computing device associated with the input, and further comprising:
transmitting, by a computing device associated with the machine learning system, the first prediction to the computing device associated with the input.
10. A system for retraining a machine learning system, the system comprising:
a previously trained classification model comprising at least one computing device associated with the machine learning system, wherein the previously trained classification model is configured to process an input to generate i) a first prediction that represents a characteristic associated with the input, and ii) a representation of accuracy associated with the prediction, wherein the representation of accuracy is a value;
a retraining subsystem comprising at least one computing device that is associated with the machine learning system, wherein the retraining subsystem is configured by executing code to:
receive the input, the first prediction, and the representation of accuracy;
determine a sufficiency of certainty of the first prediction based on at least the input, the first prediction, the value and by a comparison of the value to a first threshold value and a second threshold value; and
based at least on the determined sufficiency of the first prediction, the retraining subsystem is configured by executing code to:
i) cause the machine learning system to be automatically retrained using the input and the first prediction in the case the value is greater than the first threshold value;
ii) cause the machine learning system to be retrained using the input with active learning in the case the value is less than the first threshold value and greater than the second threshold value; or
iii) cause the machine learning system to be not trained in the case the value is less than the second threshold value.
11. The system of claim 10 , wherein causing the machine learning system to be retrained using the input with active learning further comprises:
at least one computing device comprised in the machine learning system that is configured by executing code to:
transmit to a user device configured with a graphical user interface, information associated with the input and the first prediction to present a representation of the input and the characteristic via the graphical user interface to a user; and
receive, from the user device, an acceptance or a rejection of characteristic.
12. The system of claim 11 , further comprising:
at least one computing device comprised in the machine learning system that is configured by executing code to train the machine learning system as a function of the acceptance or rejection of the characteristic received from the user device.
13. The system of claim 11 , wherein the information associated with the input is a copy of the input.
14. The system of claim 10 , wherein the representation of accuracy associated with the first prediction is a Softmax confidence value.
15. The system of claim 10 , further comprising at least one computing device associated with the machine learning system that is configured by executing code to store the input for future use by the machine learning system.
16. The system of claim 10 , wherein the retraining subsystem comprises at least one of a second classification model and selection agent.
17. The system of claim 10 , wherein the retraining subsystem is further configured to determine the sufficiency of certainty of the first prediction by determining that the first prediction is out of domain.
18. The system of claim 10 , wherein the input is received from a computing device associated with the input, and further comprising:
a computing device associated with the machine learning system which is configured to transmit the first prediction to the computing device associated with the input.