Reinforcement learning agent to measure robustness of black-box image classification models
Systems and methods are provided for reinforcement Learning agents for adversarial black-box attacks to determine and refine robustness of a machine learning (ML) model. Examples include receiving an image corresponding to a ground truth and computing sensitivity of an ML model in classifying the image as the ground truth to added and removed distortions. An RL agent determines to add distortions to and remove distortions from the image based on the sensitivities. The ML Model classifies the image based on the added and removed distortions, and the process is repeated until the machine learning model misclassifies the image. Based on the misclassification, a measure of robustness is determined and/or the ML model can be retrained.
1 . A method, comprising:
receiving an image comprising an object, the object corresponding to a ground truth;
computing a first sensitivity of a machine learning model in classifying the object as the ground truth with an addition of distortion and a second sensitivity of the machine learning model in classifying the object as the ground truth with a removal of distortion;
adding a first one or more distortions to the image based on the first sensitivity;
removing a second one or more distortions from the image based on the second sensitivity; and
classifying, by the machine learning model, the object based on the added and removed distortions, wherein the computing of the first and second sensitivities, adding the first one or more distortions, and removing the second one or more distortions, are repeated until the machine learning model misclassifies the object.
2 . The method of claim 1 , further comprising:
dividing the image into a plurality of regions;
determining a first one or more regions of the plurality of regions into which the first one or more distortions are added based on the first sensitivity; and
determining a second one or more regions of the plurality of regions from which the second one or more distortions are removed based on the second sensitivity,
wherein computing the first sensitivity and second sensitivity are computed for each region of the plurality of regions.
3 . The method of claim 1 , wherein classifying the object comprises computing a ground truth probability that the object is the ground truth and a measure of total distortion added to the image, wherein the total distortion added to the image is the second one or more distortions subtracted from the first one or more distortions.
4 . The method of claim 1 , further comprising:
computing a reward for each iteration based on a change in ground truth probability with respect to a preceding iteration and a change in total distortion added with respect to the preceding iteration,
wherein the classifying by the machine learning model is based on the reward.
5 . The method of claim 1 , wherein classifying the object comprises computing a ground truth probability that the object is the ground truth and a probability that the object is another object, wherein misclassifying the object comprising the probability being greater than the ground truth probability.
6 . The method of claim 1 , further comprising:
computing a measure of robustness of the machine learning model based on a total distortion when the machine learning model misclassifies the object.
7 . A system, comprising:
a memory configured to store instructions; and
a processor coupled to the memory and configured to execute the instructions to:
receive an image comprising an object, the object corresponding to a ground truth;
compute a first sensitivity of a machine learning model in classifying the object as the ground truth with an addition of distortion and a second sensitivity of the machine learning model in classifying the object as the ground truth with a removal of distortion;
add a first one or more distortions to the image based on the first sensitivity;
remove a second one or more distortions from the image based on the second sensitivity; and
classify, by the machine learning model, the object based on the added and removed distortions, wherein the computing of the first and second sensitivities, adding the first one or more distortions, and removing the second one or more distortions, are repeated until the machine learning model misclassifies the object.
8 . The system of claim 7 , wherein the processor is further configured to:
divide the image into a plurality of regions;
determine a first one or more regions of the plurality of regions into which the first one or more distortions are added based on the first sensitivity; and
determine a second one or more regions of the plurality of regions from which the second one or more distortions are removed based on the second sensitivity,
wherein computing the first sensitivity and second sensitivity are computed for each region of the plurality of regions.
9 . The system of claim 7 , wherein classifying the object comprises computing a ground truth probability that the object is the ground truth and a measure of total distortion added to the image, wherein the total distortion added to the image is the second one or more distortions subtracted from the first one or more distortions.
10 . The system of claim 7 , wherein the processor is further configured to:
compute a reward for each iteration based on a change in ground truth probability with respect to a preceding iteration and a change in total distortion added with respect to the preceding iteration,
wherein the classifying by the machine learning model is based on the reward.
11 . The system of claim 7 , wherein classifying the object comprises computing a ground truth probability that the object is the ground truth and a probability that the object is another object, wherein misclassifying the object comprising the probability being greater than the ground truth probability.
12 . The system of claim 7 , wherein the processor is further configured to:
computing a measure of robustness of the machine learning model based on a total distortion when the machine learning model misclassifies the object.
13 . A non-transitory computer readable storage medium having instructions that when executed cause a processor to:
receive an image comprising an object, the object corresponding to a ground truth;
compute a first sensitivity of a machine learning model in classifying the object as the ground truth with an addition of distortion and a second sensitivity of the machine learning model in classifying the object as the ground truth with a removal of distortion;
add a first one or more distortions to the image based on the first sensitivity;
remove a second one or more distortions from the image based on the second sensitivity; and
classify, by the machine learning model, the object based on the added and removed distortions, wherein the computing of the first and second sensitivities, adding the first one or more distortions, and removing the second one or more distortions, are repeated until the machine learning model misclassifies the object.
14 . The computer readable storage medium of claim 13 , further comprising the processor to:
divide the image into a plurality of regions;
determine a first one or more regions of the plurality of regions into which the first one or more distortions are added based on the first sensitivity; and
determine a second one or more regions of the plurality of regions from which the second one or more distortions are removed based on the second sensitivity,
wherein computing the first sensitivity and second sensitivity are computed for each region of the plurality of regions.
15 . The computer readable storage medium of claim 13 , wherein classifying the object comprises computing a ground truth probability that the object is the ground truth and a measure of total distortion added to the image, wherein the total distortion added to the image is the second one or more distortions subtracted from the first one or more distortions.
16 . The computer readable storage medium of claim 13 , further comprising the processor to:
compute a reward for each iteration based on a change in ground truth probability with respect to a preceding iteration and a change in total distortion added with respect to the preceding iteration,
wherein the classifying by the machine learning model is based on the reward.
17 . The computer readable storage medium of claim 13 , wherein classifying the object comprises computing a ground truth probability that the object is the ground truth and a probability that the object is another object, wherein misclassifying the object comprising the probability being greater than the ground truth probability.
18 . The computer readable storage medium of claim 13 , further comprising the processor to:
compute a measure of robustness of the machine learning model based on a total distortion when the machine learning model misclassifies the object.