Unsupervised data drift detection for classification neural networks
Methods, systems, and apparatuses for unsupervised data drift detection for classification neural networks are disclosed. One method includes providing the data stream of images to a neural network, generating, by the neural network, class wise probabilities, storing each image of the data stream of images, storing the class wise probabilities generated by the neural network, comparing artifacts of images of the data stream at a first time with artifacts of images of the data stream at a second time, comparing artifacts produced by the class wise probabilities of the data stream retrieved from the stored class wise probabilities at a third time with artifacts produced by the class wise probabilities of the data stream retrieved from the stored class wise probabilities at a fourth time, and generating an informative communication based on the comparisons.
1. A method, comprising:
sensing a data stream of images;
providing the data stream of images to a neural network;
generating, by the neural network, class wise probabilities;
storing each image of the data stream of images in a first database;
storing the class wise probabilities generated by the neural network in a second database;
comparing artifacts of images of the data stream at a first time with artifacts of images of the data stream at a second time;
comparing artifacts produced by the class wise probabilities of the data stream retrieved from the stored class wise probabilities at a third time with artifacts produced by the class wise probabilities of the data stream retrieved from the stored class wise probabilities at a fourth time; and
generating an informative communication based on the compared artifacts of images of the data stream at the first time with artifacts of images of the data stream at the second time and based on the compared artifacts produced by the class wise probabilities of the data stream retrieved from the stored class wise probabilities at a third time with artifacts produced by the class wise probabilities of the data stream retrieved from the stored class wise probabilities at a fourth time;
wherein the informative communication is generated when a weighted combination of a weight W 1 of the compared artifacts of images of the data stream at the first time with artifacts of images of the data stream at the second time, and a weight W 2 of the compared artifacts produced by the class wise probabilities of the data stream retrieved from the stored class wise probabilities at a third time with artifacts produced by the class wise probabilities of the data stream retrieved from the stored class wise probabilities at a fourth time, is greater than a weighted threshold.
2. The method of claim 1 , wherein the weights W 1 , W 2 of the weighted combination are adaptively adjusted.
3. The method of claim 2 , wherein the weights W 1 , W 2 are adaptively adjusted based on a value V 1 of the compared artifacts of images of the data stream at the first time with artifacts of images of the data stream at the second time and based on a value V 2 of the compared images of the data stream retrieved from the stored class wise probabilities at a third time with images of the data stream retrieved from the stored class wise probabilities at a fourth time.
4. The method of claim 3 , wherein a one of the weights W 1 , W 2 is greatest based on which of the values V 1 , V 2 is relatively greater.
5. The method of claim 1 , further comprising encoding the images of the data stream at the first time and encoding the images of the data stream at the second time before comparing, wherein the encoding provides the artifacts of images of the data stream at the first time with images of the data stream at the second time.
6. The method of claim 1 , wherein the comparing artifacts of the images includes determining linear interpolation distances between stacks of encoded images.
7. The method of claim 1 , wherein the informative communication provides an alert to a system operator that models of a neural network need to be updated.
8. The method of claim 1 , further comprising:
providing the data stream of images to one or more retired neural network models; and
comparing outputs of the one or more retired network models with outputs of the class wise probabilities;
wherein the informative communication is generated further based on the compared outputs of the one or more retired neural network models with outputs of the class wise probabilities.
9. The method of claim 8 , wherein the informative communication is generated when the compared artifacts of images of the data stream at the first time with artifacts of images of the data stream at the second time is greater than the first change threshold, the compared artifacts produced by the class wise probabilities of the data stream retrieved from the stored class wise probabilities at a third time with artifacts produced by the class wise probabilities of the data stream retrieved from the stored class wise probabilities at a fourth time is greater than a second change threshold, or the compared outputs of the one or more retired network models with outputs of the class wise probabilities is greater than a third change threshold.
10. The method of claim 9 , wherein the informative communication is generated when a weighted combination of the weight W 1 of the compared artifacts of images of the data stream at the first time with artifacts of images of the data stream at the second time, the weight W 2 of the compared artifacts produced by the class wise probabilities of the data stream retrieved from the stored class wise probabilities at a third time with artifacts produced by the class wise probabilities of the data stream retrieved from the stored class wise probabilities at a fourth time, and a weight W 3 of the compared outputs of the one or more retired network models with outputs of the class wise probabilities is greater than a third change threshold.
11. The method of claim 10 , wherein the weights W 1 , W 2 , W 3 of the weighted combination are adaptively adjusted.
12. The method of claim 11 , wherein the weights W 1 , W 2 , W 3 are adaptively adjusted based on a value V 1 of the compared artifacts of images of the data stream at the first time with artifacts of images of the data stream at the second time, based on a value V 2 of the compared artifacts produced by the class wise probabilities of the data stream retrieved from the stored class wise probabilities at a third time with artifacts produced by the class wise probabilities of the data stream retrieved from the stored class wise probabilities at a fourth time, and based on a value V 3 of the compared outputs of the one or more retired network models with outputs of the class wise probabilities.
13. The method of claim 12 , wherein a one of the weights W 1 , W 2 , W 3 is greatest based on which of the values V 1 , V 2 , V 3 is relatively greater.
14. The method of claim 8 , wherein the one or more retired neural network models comprise models of the neural network that were previously used.
15. A system for detection of unsupervised data drift, comprising:
an image sensor configured to sense a data stream of images; and
a processor, the processor configured to:
receive the data stream of images;
provide the data stream of images to a neural network;
a neural network configured to generate class wise probabilities;
the processor further configured to:
store each image of the data stream of images in a database;
store the class wise probabilities generated by the neural network;
compare artifacts of images of the data stream at a first time with artifacts of images of the data stream at a second time;
compare artifacts produced by the class wise probabilities of the data stream retrieved from the stored class wise probabilities at a third time with artifacts produced by the class wise probabilities of the data stream retrieved from the stored class wise probabilities at a fourth time; and
generate an informative communication based on the compared artifacts of images of the data stream at the first time with images of the data stream at the second time and based on the compared images of the data stream retrieved from the stored class wise probabilities at the third time with images of the data stream retrieved from the stored class wise probabilities at the fourth time;
wherein the informative communication is generated when a weighted combination of a weight W 1 of the compared artifacts of images of the data stream at the first time with artifacts of images of the data stream at the second time, and a weight W 2 of the compared artifacts produced by the class wise probabilities of the data stream retrieved from the stored class wise probabilities at a third time with artifacts produced by the class wise probabilities of the data stream retrieved from the stored class wise probabilities at a fourth time, is greater than a weighted threshold.
16. The system of claim 15 , wherein the processor is further configured to:
provide the data stream of images to one or more retired neural network models; and
compare outputs of the one or more retired network models with outputs of the class wise probabilities;
wherein the informative communication is generated further based on the compared outputs of the one or more retired network models with outputs of the class wise probabilities.