Granted Patent B2

US 12,725,047 · App. 18/167,881 · Granted Sep 1, 2026

Inventors: Hon Chung Lee (Hong Kong, HK); Jiayu Shu (Hong Kong, HK)

Assignee: Hong Kong Applied Science and Technology Research Institute Company Limited

G06N3/0895

Abstract

Claims (65)

1 . A method for classifying and localizing surface defects for a target defect classification task with limited number of labeled real defect images as training data to classification model used, comprising:

predicting a defect class for an input image using a convolutional neural network (CNN) classification model;

generating a coarse localization map for the input image using the CNN classification model to highlight one or more anomaly regions in the input image by using gradient information flowing into a final convolutional layer of the CNN classification model's neural network;

aligning the input image before generating an anomaly map, comprising:

transforming, by a Spatial Transformer Network (STN), a main part of the input image to match a pre-selected golden sample so to reduce variance between the input image and pre-selected golden sample for easing detection of outliers in the input image;

wherein the STN is trained with the real non-defect images for image alignment;

generating the anomaly map for the input image using an anomaly detection model, wherein the anomaly map contains an indication of probability of defect at each of one or more locations in the input image;

predicting a defect location in the input image via location ensemble using the coarse localization map and the anomaly map for enhancing prediction accuracy, comprising:

taking a weighted sum of the coarse localization map generated by the CNN classification model and the anomaly map generated by the anomaly detection model to obtain an intermediate defect location map; and

applying a binary threshold to the intermediate defect location map to obtain the final defect location; and

outputting the defect class and the defect location for the target defect classification task;

wherein the CNN classification model is built based on a CNN encoder and is finetuned with one or more labeled real defect images for the target defect classification task;

wherein the CNN encoder is trained by multi-task self-supervised learning with one or more real non-defect images and one or more synthetic defect images;

wherein number of the labeled real defect images is significantly fewer than number of the real non-defect images;

wherein the anomaly detection model is built by learning a normal distribution of the real non-defect images, comprising;

dividing the real non-defect images into real non-defect image patches;

extracting real non-defect image patch embeddings from the real non-defect image patches; and

estimating the normal distribution of the real non-defect images from the real non-defect image patch embeddings: and

wherein anomaly detection model comprises:

dividing the input image into input image patches;

extracting input image patch embeddings from the input image patches; and

comparing the input image patch embeddings with the normal distribution of the real non-defect images to determine the probability of defect at each of the one or more locations in the input image.

2 . The method of claim 1 , wherein the synthetic defect images are generated using the anomaly detection model identifying one or more anomaly regions in one or more unlabeled real images and overlaying the anomaly regions on the real non-defect images.

3 . The method of claim 1 , wherein the CNN classification model is built and trained by:

adding the CNN encoder with a randomly initialized linear classifier; and

finetuning the CNN classification model with the labeled real defect images.

4 . The method of claim 1 , wherein the multi-task self-supervised learning comprises:

performing a binary classification task with the real non-defect and the synthetic defect images using a classification head to calculate a cross-entropy loss;

performing a contrastive learning task with the real non-defect and the synthetic defect images using a contrastive head to calculate a contrastive loss; and

updating the weights of the CNN encoder and two heads to minimize a weighted sum of the cross-entropy loss and the contrastive loss.

5 . A system for classifying and localizing surface defects for a target defect classification task with limited number of labeled real defect images as training data to classification model used, comprising:

an anomaly learning module having at least one processor configured to perform a learning of an anomaly detection model;

a self-supervised pretraining module having at least one processor configured to perform multi-task self-supervised learning to train a CNN encoder;

a finetuning module having at least one processor configured to build and train a convolutional neural network (CNN) classification model; and

an inference module having at least one processor configured to perform:

predicting a defect class for an input image using the CNN classification model;

generating a coarse localization map for the input image using the CNN classification model by using gradient information flowing into a final convolutional layer of the CNN classification model's neural network, wherein the coarse localization map highlights one or more anomaly regions in the input image;

align the input image before generating an anomaly map, the alignment comprising:

transforming, by a Spatial Transformer Network (STN), a main part of the input image to match a pre-selected golden sample so to reduce variance between the input image and pre-selected golden sample for easing detection of outliers in the input image;

wherein the STN is trained with the real non-defect images for image alignment;

generating the anomaly map for the input image using the anomaly detection model, wherein the anomaly map contains an indication of probability of defect at each of one or more locations in the input image; and

predicting a defect location in the input image via location ensemble using the coarse localization map and the anomaly map for enhancing prediction accuracy, comprising:

taking a weighted sum of the coarse localization map generated by the CNN classification model and the anomaly map generated by the anomaly detection model to obtain an intermediate defect location map:

and applying a binary threshold to the intermediate defect location map to obtain the final defect location;

wherein the CNN classification model is built based on a CNN encoder and is finetuned with one or more labeled real defect images for the target defect classification task;

wherein the CNN encoder is trained by the multi-task self-supervised learning with one or more real non-defect images and one or more synthetic defect images;

wherein number of the labeled real defect images is significantly fewer than number of the real non-defect images;

wherein the anomaly detection model is built by learning a normal distribution of the real non-defect images, comprising:

dividing the real non-defect images into real non-defect image patches;

extracting real non-defect image patch embeddings from the real non-defect image patches; and

estimating the normal distribution of the real non-defect images from the real non-defect image patch embeddings; and

wherein anomaly detection model comprises:

dividing the input image into input image patches;

extracting input image patch embeddings from the input image patches; and

comparing the input image patch embeddings with the normal distribution of the real non-defect images to determine the probability of defect at each of the one or more locations in the input image.

6 . The system of claim 5 , wherein the synthetic defect images are generated using the anomaly detection model identifying one or more anomaly regions in one or more unlabeled real images and overlaying the anomaly regions on the real non-defect images.

7 . The system of claim 5 , wherein at least one processing unit of the finetuning module further performs the following for building and training the CNN classification model:

adding the CNN encoder with a randomly initialized linear classifier; and

finetuning the CNN classification model with the labeled real defect images.

8 . The system of claim 5 , wherein the multi-task self-supervised learning comprises:

performing a binary classification task with the real non-defect and the synthetic defect images using a classification head to calculate a cross-entropy loss;

performing a contrastive learning task with the real non-defect and the synthetic defect images using a contrastive head to calculate a contrastive loss; and

updating the weights of the CNN encoder and two heads to minimize a weighted sum of the cross-entropy loss and the contrastive loss.

9 . The method of claim 1 , wherein the target defect classification task is semiconductor packaging defect detection.

10 . The system of claim 5 , wherein the target defect classification task is semiconductor packaging defect detection.

Assignments (1)

Continuity (1)

References Cited (29)