Safety monitor for image misclassification
Systems, apparatuses, and methods for implementing a safety monitor framework for a safety-critical inference application are disclosed. A system includes a safety-critical inference application, a safety monitor, and an inference accelerator engine. The safety monitor receives an input image, test data, and a neural network specification from the safety-critical inference application. The safety monitor generates a modified image by adding additional objects outside of the input image. The safety monitor provides the modified image and neural network specification to the inference accelerator engine which processes the modified image and provides outputs to the safety monitor. The safety monitor determines the likelihood of erroneous processing of the original input image by comparing the outputs for the additional objects with a known good result. The safety monitor complements the overall fault coverage of the inference accelerator engine and covers faults only observable at the network level.
1 . A system comprising:
a processor comprising circuitry configured to:
track a frequency of detection of one or more objects in previously received image data;
add at least one object of the one or more objects to current image data received subsequent to the previously received image data by inserting pixel data representing the at least one object into the current image data to generate modified image data, responsive to a frequency of detection of the at least one object in the previously received image data exceeding a first threshold; and
generate an indication corresponding to a calculated likelihood of misclassification of the current image data, based at least in part on processing the modified image data; and
perform an action in response to the indication.
2 . The system as claimed in claim 1 , wherein the processor is configured to add the at least one object outside a boundary of an image corresponding to the current image data, to generate the modified image data.
3 . The system as claimed in claim 1 , wherein the processor is configured to generate the indication based at least in part on an output of a machine learning model.
4 . The system as claimed in claim 1 , wherein the processor is further configured to:
cause the modified image data to be processed by transmitting the modified image data to an inference accelerator; and
receive, from the inference accelerator, at least one output indicative of processing of the modified image data by the inference accelerator.
5 . The system as claimed in claim 4 , wherein the current image data is deemed misclassified, responsive to the at least one output not being consistent with a given test vector data corresponding to the current image data.
6 . The system as claimed in claim 1 , wherein the at least one object, represented by the pixel data, is selected based at least in part on a given test vector data corresponding to the previously received image data.
7 . The system as claimed in claim 1 , wherein the action comprises causing a safety-critical application to perform at least one of:
terminating operation;
re processing a same image;
rebooting a system;
generating a warning signal;
reducing a speed of a vehicle; or
changing an operating mode.
8 . A method comprising:
tracking, by a processor comprising circuitry, a frequency of detection of one or more objects in previously received image data; and
adding, by the processor, at least one object of the one or more objects to current image data received subsequent to the previously received image data by inserting pixel data representing the at least one object into the current image data to generate modified image data, responsive to a frequency of detection of the at least one object in the previously received image data exceeding a first threshold;
generating an indication corresponding to a calculated likelihood of misclassification of the current image data, based at least in part on processing the modified image data; and
performing an action in response to the indication.
9 . The method as claimed in claim 8 , further comprising adding, by the processor, the at least one object outside a boundary of an image corresponding to the current image data, to generate the modified image data.
10 . The method as claimed in claim 8 , further comprising generating the indication based at least in part on an output of a machine learning model.
11 . The method as claimed in claim 8 , further comprising:
causing, by the processor, the modified image data to be transmitted to an inference accelerator; and
receiving, by the processor from the inference accelerator, at least one output indicative of processing of the modified image data by the inference accelerator.
12 . The method as claimed in claim 11 , wherein the current image data is deemed misclassified, responsive to the at least one output not being consistent with a given test vector data corresponding to the current image data.
13 . The method as claimed in claim 8 , wherein the at least one object, represented by the pixel data, is selected based at least in part on a given test vector data corresponding to the previously received image data.
14 . The method as claimed in claim 11 , wherein the at least one object is identified as an object having a calculated probability of being correctly identified by the inference accelerator in the previously received image data, exceeding a second threshold.
15 . A system comprising:
an inference accelerator comprising circuitry; and
a processor comprising circuitry configured to:
track a frequency of detection of one or more objects in previously received image data;
add at least one object of the one or more objects to current image data received subsequent to the previously received image data by inserting pixel data representing the at least one object into the current image data to generate modified image data, responsive to a frequency of detection of the at least one object in the previously received image data exceeding a first threshold;
generate an indication corresponding to a calculated likelihood of misclassification of the current image data, based at least in part on processing of the modified image data by the inference accelerator; and
perform an action in response to the indication.
16 . The system as claimed in claim 15 , wherein the processor is configured to add the at least one object outside a boundary of an image corresponding to the current image data, to generate modified image data.
17 . The system as claimed in claim 15 , wherein the inference accelerator is configured to execute a machine learning model.
18 . The system as claimed in claim 15 , where in the processor is further configured to:
transmit the modified image data to the inference accelerator; and
receive, from the inference accelerator, at least one output indicative of processing of the modified image data by the inference accelerator.
19 . The system as claimed in claim 15 , wherein the at least one object is selected based at least in part on a given test vector data corresponding to the previously received image data.
20 . The system as claimed in claim 15 , wherein the at least one object is identified as an object having a calculated probability of being correctly identified by the inference accelerator in previously processed image data.