IP Library › Granted Patent US 11,538,146
Granted Patent B2
US 11,538,146 · App. 17/110,131 · Granted Dec 27, 2022

Automated machine vision-based defect detection

Inventors: Rajen Bhatt (McDonald, PA); Shitong Mao (Pittsburgh, PA); Raviprakash Kandury (Cupertino, CA); Michelle Tai (Fremont, CA); Geoffrey Newman (Pittsburgh, PA)
Assignee: QEEXO, CO.
G06T7/0004G06N3/08G06T7/11G06T7/194G06T2207/20076G06T2207/20081G06T2207/20084G06T2207/30164
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,538,146
App. No.
17/110,131
Granted
Dec 27, 2022
Kind
B2
Abstract

Provided are various mechanisms and processes for automatic computer vision-based defect detection using a neural network. A system is configured for receiving historical datasets that include training images corresponding to one or more known defects. Each training image is converted into a corresponding matrix representation for training the neural network to adjust weighted parameters based on the known defects. Once sufficiently trained, a test image of an object that is not part of the historical dataset is obtained. Portions of the test image are extracted as input patches for input into the neural network as respective matrix representations. A probability score indicating the likelihood that the input patch includes a defect is automatically generated for each input patch using the weighted parameters. An overall defect score for the test image is then generated based on the probability scores to indicate the condition of the object.

Claims (54)

1. A method comprising:

training a neural network to detect defects, wherein training the neural network includes:

receiving a plurality of historical datasets including a plurality of training images corresponding to one or more known defects,

converting each training image into a corresponding matrix representation, and

inputting each corresponding matrix representation into the neural network to adjust weighted parameters based on the one or more known defects, wherein the weighted parameters correspond to dimensions of the matrix representations;

obtaining a test image of an object, wherein the test image is not part of the historical dataset;

extracting portions of the test image as a plurality of patches for input into the neural network,

each patch corresponding to an extracted portion of the test image;

determining whether each patch contains a portion of the object; and

inputting each patch that is determined to contain a portion of the object, while preventing input of patches that are determined to not contain a portion of the object, into the neural network as a respective matrix representation to automatically generate a probability score for each patch using the weighted parameters;

wherein the probability score for each patch input into the neural network indicates the probability that the patch includes a predicted defect, wherein a defect score for the test image is generated based on the probability scores for each patch, and wherein the defect score indicates a condition of the object.

2. The method of claim 1 , wherein the neural network comprises one or more of the following: a convolution layer, a max pooling layer, a flattening layer, and a fully connected layer.

3. The method of claim 1 , wherein the patches are input into the neural network in parallel.

4. The method of claim 1 , wherein the patches include overlapping portions of the test image.

5. The method of claim 1 , wherein the patches are aligned such that each patch is immediately adjacent to one or more other patches of the plurality of patches.

6. The method of claim 1 , wherein the neural network is trained to accurately output a probability score for a defect in each patch input into the neural network using the weighted parameters.

7. The method of claim 6 , further comprising generating a heat map of the patches based on the probability scores.

8. The method of claim 6 , wherein the neural network is trained to accurately output the defect score for the object based on a maximum or average of the probability scores of the patches input into the neural network.

9. The method of claim 1 , further comprising, prior to inputting the patches of the test image into the neural network, removing background and other noise from the test image and removing any color components from the image to only retain a luma component.

10. A server system comprising:

an interface configured to receive:

a plurality of historical data sets including a plurality of training images corresponding to one or more levels of known defects, and

a test image of an object, wherein the test image is not part of the historical dataset;

memory configured to store:

the historical datasets and the test image; and

a processor associated with a neural network, wherein the processor is configured for:

training a neural network to detect defects by:

converting each training image into a corresponding matrix representation,

inputting each corresponding matrix representation into the neural network to adjust weighted parameters based on the one or more known defects, wherein the weighted parameters correspond to dimensions of the matrix representations;

wherein the processor is further configured for:

extracting portions of the test image as a plurality of patches for input into the neural network, each patch corresponding to an extracted portion of the test image;

determining whether each patch contains a portion of the object; and

inputting each patch that is determined to contain a portion of the object, while preventing input of patches that are determined to not contain a portion of the object, into the neural network as a respective matrix representation to automatically generate a probability score for each patch using the weighted parameters;

wherein the probability score for each patch input into the neural network indicates the probability that the patch includes a predicted defect, wherein a defect score for the test image is generated based on the probability scores for each patch, and wherein the defect score indicates a condition of the object.

11. The server system of claim 10 , wherein the neural network comprises one or more of the following: a convolution layer, a max pooling layer, a flattening layer, and a fully connected layer.

12. The server system of claim 10 , wherein the patches are input into the neural network in parallel.

13. The server system of claim 10 , wherein the patches include overlapping portions of the test image.

14. The server system of claim 10 , wherein the patches are aligned such that each patch is immediately adjacent to one or more other patches of the plurality of patches.

15. The server system of claim 10 , wherein the neural network is trained to accurately output a probability score for a defect in each patch input into the neural network using the weighted parameters.

16. The server system of claim 15 , further comprising generating a heat map of the patches based on the probability scores.

17. The server system of claim 10 , wherein the processor is further configured for, prior to inputting the patches of the test image into the neural network, removing background and other noise from the test image and removing any color components from the image to only retain a luma component.

18. The server system of claim 10 , wherein the neural network is trained to accurately output the defect score for the object based on a maximum or average of the probability scores of the patches input into the neural network.

19. A non-transitory computer readable medium storing one or more programs configured for execution by a computer, the one or more programs comprising instructions for:

training a neural network to detect defects by:

receiving a plurality of historical datasets including a plurality of training images corresponding to one or more known defects,

converting each training image into a corresponding matrix representation, and

inputting each corresponding matrix representation into the neural network to adjust weighted parameters based on the one or more known defects, wherein the weighted parameters correspond to dimensions of the matrix representations;

obtaining a test image of an object, wherein the test image is not part of the historical dataset;

extracting portions of the test image as a plurality of patches for input into the neural network, each patch corresponding to an extracted portion of the test image;

determining whether each patch contains a portion of the object; and

inputting each patch that is determined to contain a portion of the object, while preventing input of patches that are determined to not contain a portion of the object, into the neural network as a respective matrix representation to automatically generate a probability score for each patch using the weighted parameters;

wherein the probability score for each patch input into the neural network indicates the probability that the patch includes a predicted defect, wherein a defect score for the test image is generated based on the probability scores for each patch, and wherein the defect score indicates a condition of the object.

20. The non-transitory computer readable medium of claim 19 , wherein the neural network comprises one or more of the following: a convolution layer, a max pooling layer, a flattening layer, and a fully connected layer.

21. The non-transitory computer readable medium of claim 19 , wherein the patches are input into the neural network in parallel.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 22, 2025
From: QEEXO, CO.
To: TDK SENSEI PTE. LTD.
Reel/Frame 072640/0516 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 4, 2020
From: NEWMAN, GEOFFREY; KANDURY, RAVIPRAKASH; TAI, MICHELLE; BHATT, RAJEN; MAO, SHITONG
To: QEEXO, CO.
Reel/Frame 054541/0909 →
Continuity (2)
Provisional Application 62950440 · Dec 19, 2019
Related Publication 20210192714A1 · Jun 24, 2021
Cited By (1)
US 12,456,185