Interpretable deep learning-based defect detection and classification
An explanation of a detection/classification algorithm made using a deep learning neural network clarifies the results that are formed and helps a user to identify the root cause of defect detection/classification model performance issues. A relevance map is determined based on a layer-wise relevance propagation algorithm. A mean intersection over union score between the relevance map and a ground truth is determined. A part of one of the semiconductor images that contributed to the classification using the deep learning model based on the relevance map and the mean intersection over union score is determined.
1 . A method comprising:
directing an electron beam at a wafer using an electron beam source in a wafer inspection tool;
receiving electrons returned from the wafer at a detector in the wafer inspection tool;
generating a plurality of semiconductor images of the wafer using data from the detector about the electrons returned from the wafer;
performing defect detection and classification on the plurality of semiconductor images using a processor with a deep learning model;
applying, using the processor, a layer-wise relevance propagation algorithm on the semiconductor images from the deep learning model;
determining, using the processor, a relevance map based on the layer-wise relevance propagation algorithm, wherein the relevance map is based on a relevance of an output layer and relevance of early layers, wherein the relevance of the early layers is determined for each layer using an equation
R
i
=
∑
j
a
i
w
i
j
∑
i
a
i
w
i
j
R
j
,
and wherein R i is the relevance of a neuron i at a first layer, R j is a relevance of the neuron j at a second layer, a i is an activation of the neuron i, and w ij is a weight between the neuron i and the neuron j;
determining, using the processor, a mean intersection over union score between the relevance map and a ground truth;
determining, using the processor, a part of one of the semiconductor images that contributed to the classification using the deep learning model based on the relevance map and the mean intersection over union score, wherein determining the part of the one of the semiconductor images further uses at least one of an overlay of the mean intersection over union score and the one of the semiconductor images or an overlay of the relevance map and the one of the semiconductor images;
changing an annotation of a training image for the deep learning model and/or a class code of a training image for the deep learning model based on the part of the semiconductor images that contributed to the classification; and
performing a second defect detection and classification on the plurality of semiconductor images using the processor with the deep learning model after changing the annotation and/or the class code.
2 . The method of claim 1 , wherein the relevance map is normalized.
3 . The method of claim 1 , further comprising smoothing the plurality of semiconductor images using the processor.
4 . The method of claim 1 , wherein the method includes changing the annotation of the training image for the deep learning model when a mean intersection over union score is less than a threshold.
5 . The method of claim 1 , wherein the method includes changing the class code of the training image for the deep learning model.
6 . A wafer inspection tool comprising:
an electron beam source that generates an electron beam;
a stage configured to hold a wafer in a path of the electron beam;
a detector that receives electrons returned from the wafer; and
a processor in electronic communication with the detector, wherein the processor is configured to:
generate a plurality of semiconductor images of the wafer using data received from the detector about the electrons returned from the wafer;
perform defect detection and classification on the plurality of semiconductor images using a deep learning model;
apply a layer-wise relevance propagation algorithm on the semiconductor images from the deep learning model;
determine a relevance map based on the layer-wise relevance propagation algorithm, wherein the relevance map is based on a relevance of an output layer and relevance of early layers, wherein the relevance of the early layers is determined for each layer using an equation
R
i
=
∑
j
a
i
w
i
j
∑
i
a
i
w
i
j
R
j
,
and wherein R i is the relevance of a neuron i at a first layer, R j is a relevance of the neuron j at a second layer, a i is an activation of the neuron i, and w ij is a weight between the neuron i and the neuron j;
determine a mean intersection over union score between the relevance map and a ground truth;
determine a part of one of the semiconductor images that contributed to the classification using the deep learning model based on the relevance map and the mean intersection over union score, wherein determining the part of the one of the semiconductor images further uses at least one of an overlay of the mean intersection over union score and the one of the semiconductor images or an overlay of the relevance map and the one of the semiconductor images;
change an annotation of a training image for the deep learning model and/or a class code of a training image for the deep learning model based on the part of the semiconductor images that contributed to the classification; and
perform a second defect detection and classification on the plurality of semiconductor images using the processor with the deep learning model after changing the annotation and/or the class code.
7 . The wafer inspection tool of claim 6 , wherein the relevance map is normalized.
8 . The wafer inspection tool of claim 6 , wherein the processor is configured to change the annotation of the training image for the deep learning model when a mean intersection over union score is less than a threshold.
9 . The wafer inspection tool of claim 6 , wherein the processor is configured to change the class code of the training image for the deep learning model.
10 . A non-transitory computer-readable storage medium, comprising one or more programs for executing the following steps on one or more computing devices:
sending instructions to a wafer inspection tool to direct an electron beam at a wafer using an electron beam source;
generating a plurality of semiconductor images of the wafer using data from a detector that received electrons returned from the wafer;
performing defect detection and classification on the plurality of semiconductor images using a deep learning model;
applying a layer-wise relevance propagation algorithm on the semiconductor images from the deep learning model;
determining a relevance map based on the layer-wise relevance propagation algorithm, wherein the relevance map is based on a relevance of an output layer and relevance of early layers, wherein the relevance of the early layers is determined for each layer using an equation
R
i
=
∑
j
a
i
w
i
j
∑
i
a
i
w
i
j
R
j
,
and wherein R i is the relevance of a neuron i at a first layer, R j is a relevance of the neuron j at a second layer, a i is an activation of the neuron i, and w ij is a weight between the neuron i and the neuron j;
determining a mean intersection over union score between the relevance map and a ground truth;
determining a part of one of the semiconductor images that contributed to the classification using the deep learning model based on the relevance map and the mean intersection over union score, wherein determining the part of the one of the semiconductor images further uses at least one of an overlay of the mean intersection over union score and the one of the semiconductor images or an overlay of the relevance map and the one of the semiconductor images;
changing an annotation of a training image for the deep learning model and/or a class code of a training image for the deep learning model based on the part of the semiconductor images that contributed to the classification; and
performing a second defect detection and classification on the plurality of semiconductor images using the processor with the deep learning model after changing the annotation and/or the class code.
11 . The non-transitory computer-readable storage medium of claim 10 , wherein the relevance map is normalized.
12 . The non-transitory computer-readable storage medium of claim 10 , wherein the steps further comprise generating the semiconductor images using data from a wafer inspection tool that uses an electron beam.
13 . The non-transitory computer-readable storage medium of claim 10 , wherein the steps include changing the annotation of the training image for the deep learning model when a mean intersection over union score is less than a threshold.
14 . The non-transitory computer-readable storage medium of claim 10 , wherein the steps include changing the class code of the training image for the deep learning model.