IP Library › Granted Patent US 11,727,052
Granted Patent B2
US 11,727,052 · App. 17/011,909 · Granted Aug 15, 2023

Inspection systems and methods including image retrieval module

Inventors: Xiao Bian (Santa Clara, CA); Bernard Patrick Bewlay (Niskayuna, NY); Colin James Parris (Brookfield, CT); Feng Xue (Clifton Park, NY); Shaopeng Liu (Clifton Park, NY); Arpit Jain (Dublin, CA); Shourya Sarcar (Niskayuna, NY)
Assignee: General Electric Company
G06F16/583G06F16/51G06F16/9014G06N3/045G06V10/25
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,727,052
App. No.
17/011,909
Granted
Aug 15, 2023
Kind
B2
Abstract

A method of inspecting a component using an image retrieval module includes storing an inspection image file in a memory and identifying a region of interest in the inspection image file. The method further includes accessing a database storing image files and determining feature vectors associated with the image files. The method also includes determining a hash code for each image file based on the feature vectors and classifying a subset of image files as relevant based on the hash codes. The method further includes sorting the subset of image files based on the feature vectors and generating search results based on the sorted subset of image files. The image retrieval module includes a convolutional neural network configured to learn from the determination of the feature vectors and increase the accuracy of the image retrieval module in classifying the image files.

Claims (58)

1. A method of inspecting a component using an image inspection controller that includes a processor communicatively coupled to a memory and configured to operate in accordance with an image retrieval module, said method comprising:

storing, using the processor, at least one inspection image file in the memory;

identifying, using the processor, a region of interest in the at least one inspection image file;

determining, using the processor, at least one foreground feature vector associated with the region of interest in the at least one inspection image file;

accessing, using the processor, a database storing a plurality of image files;

determining, using the processor, a plurality of feature vectors associated with the plurality of image files, wherein each image file of the plurality of image files is associated with at least one feature vector of the plurality of feature vectors, and wherein the image retrieval module includes at least one convolutional neural network configured to learn from the determination of the plurality of feature vectors and increase an accuracy of the image retrieval module in classifying the plurality of image files;

determining, using the processor, at least one hash code for each image file of the plurality of image files based on the plurality of feature vectors;

classifying, using the processor, a subset of the plurality of image files as relevant based on hash codes of the plurality of image files;

sorting, using the processor, the subset of the plurality of image files based on the plurality of feature vectors; and

generating, using the processor, search results based on the sorted subset of the plurality of image files.

2. A method in accordance with claim 1 , wherein determining, using the processor, the plurality of feature vectors associated with the plurality of image files comprises determining, using the processor, a plurality of foreground feature vectors and a plurality of background feature vectors, wherein each image file of the plurality of image files is associated with the at least one foreground feature vector and at least one background feature vector.

3. A method in accordance with claim 2 further comprising:

determining, using the processor, the at least one background feature vector associated with a background of the at least one inspection image file;

comparing, using the processor, the at least one foreground feature vector of the at least one inspection image file and the plurality of foreground feature vectors of the plurality of image files; and

comparing, using the processor, the at least one background feature vector of the at least one inspection image file and the plurality of background feature vectors of the plurality of image files.

4. A method in accordance with claim 3 further comprising identifying the region of interest in each image file of the subset of the plurality of image files based on a comparison of the at least one foreground feature vector of the at least one inspection image file and the plurality of foreground feature vectors of the plurality of image files.

5. A method in accordance with claim 1 further comprising determining hamming distances between a hash code of the at least one inspection image file and the hash codes of the plurality of image files.

6. A method in accordance with claim 5 further comprising classifying the plurality of image files according to the hamming distances, wherein each image file in the subset of the plurality of image files is associated with a hamming distance less than a threshold value.

7. A method in accordance with claim 1 , further comprising:

comparing the at least one foreground feature vector of the at least one inspection image file and the plurality of feature vectors of the plurality of image files; and

determining similarities between the at least one inspection image file and the plurality of image files based on a comparison of the at least one foreground feature vector of the at least one inspection image file and the plurality of feature vectors of the plurality of image files.

8. A method in accordance with claim 7 , wherein sorting the subset of the plurality of image files based on the plurality of feature vectors of the plurality of image files comprises sorting the subset of the plurality of image files according to the similarities between the at least one inspection image file and the plurality of image files determined using the plurality of feature vectors.

9. A method in accordance with claim 1 further comprising generating the hash codes for the plurality of image files using a hash net based on the plurality of feature vectors associated with the plurality of image files, wherein the at least one convolutional neural network is configured to learn from the generation of the hash codes to increase the accuracy of retrieval module in classifying the plurality of image files.

10. A method in accordance with claim 1 further comprising identifying at least one region of interest in each image file of the plurality of image files, wherein the search results include an inspection report highlighting the at least one region of interest in each image file of the plurality of image files.

11. A method in accordance with claim 1 further comprising determining, using the processor, a learning loss of the at least one convolutional neural network based on the plurality of feature vectors of the plurality of image files.

12. A system for inspecting a component, said system comprising:

at least one memory comprising a database storing a plurality of image files; and

at least one processor configured to access said at least one memory and operate in accordance with an image retrieval module, wherein said at least one processor is programmed to:

store at least one inspection image file in the memory;

identify a region of interest in the at least one inspection image file;

determine at least one foreground feature vector associated with the region of interest in the at least one inspection image file;

determine a plurality of feature vectors associated with the plurality of image files, wherein each image file of the plurality of image files is associated with at least one feature vector of the plurality of feature vectors, and wherein the image retrieval module includes at least one convolutional neural network configured to learn from the determination of the plurality of feature vectors and increase an accuracy of the image retrieval module in classifying the plurality of image files;

determine at least one hash code for each image file of the plurality of image files based on the plurality of feature vectors;

classify a subset of the plurality of image files as relevant based on the hash codes of the plurality of image files;

sort the subset of the plurality of image files based on the plurality of feature vectors for the plurality of image files; and

generate search results based on the sorted subset of the plurality of image files.

13. A system in accordance with claim 12 , wherein the plurality of feature vectors includes a plurality of foreground feature vectors and a plurality of background feature vectors, and wherein each image file of the plurality of image files is associated with the at least one foreground feature vector and at least one background feature vector.

14. A system in accordance with claim 13 wherein said processor is programmed to:

determine the at least one background feature vector associated with a background of the at least one inspection image file;

compare the at least one foreground feature vector of the at least one inspection image file and the plurality of foreground feature vectors of the plurality of image files; and

compare the at least one background feature vector of the at least one inspection image file and the plurality of background feature vectors of the plurality of image files.

15. A system in accordance with claim 12 , wherein said processor is configured to:

determine hamming distances between a hash code of the at least one inspection image file and the hash codes of the plurality of image files; and

sort the plurality of image files according to the hamming distances, wherein each image file in the subset of the plurality of image files is associated with a hamming distance less than a threshold value.

16. A system in accordance with claim 12 further comprising a user interface configured to receive a search request for the database storing the plurality of image files, and display the search results.

17. A system in accordance with claim 12 , wherein said processor is configured to identify at least one region of interest in each image file of the plurality of image files, and wherein the search results include an inspection report highlighting the at least one region of interest in each image file of the plurality of image files.

18. A non-transitory computer readable medium comprising instructions that, when executed by a processor, cause the processor to implement a technique for identifying at least one feature in at least one image of a component, and in implementing the technique, the processor is configured to:

store at least one inspection image file in a memory;

identify a region of interest in the at least one inspection image file;

determine at least one foreground feature vector associated with the region of interest in the at least one inspection image file;

access a database storing a plurality of image files;

determine a plurality of feature vectors associated with the plurality of image files, wherein each image file of the plurality of image files is associated with at least one feature vector of the plurality of feature vectors, and wherein an image retrieval module, when executed by the processor, implements at least one convolutional neural network configured to learn from the determination of the plurality of feature vectors and increase an accuracy of said image retrieval module in classifying the plurality of image files;

determine at least one hash code for each image file of the plurality of image files based on the plurality of feature vectors;

classify a subset of the plurality of image files as relevant based on the hash codes of the plurality of image files;

sort the subset of the plurality of image files based on the plurality of feature vectors for the plurality of image files; and

generate search results based on the sorted subset of the plurality of image files.

19. A non-transitory computer readable medium in accordance with claim 18 , wherein the plurality of feature vectors includes a plurality of foreground feature vectors and a plurality of background feature vectors, and wherein said image retrieval module, when executed by the processor, is configured to compare the at least one foreground feature vector to the plurality of foreground feature vectors.

20. A non-transitory computer readable medium in accordance with claim 19 , wherein said image retrieval module, when executed by the processor, is configured to identify at least one region of interest in each image file of the plurality of image files, and wherein the search results include an inspection report highlighting the at least one region of interest in each image file of the plurality of image files.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 21, 2021
From: BIAN, XIAO; BEWLAY, BERNARD PATRICK; PARRIS, COLIN JAMES; XUE, FENG; LIU, SHAOPENG; JAIN, ARPIT; SARCAR, SHOURYA
To: GENERAL ELECTRIC COMPANY
Reel/Frame 055991/0130 →
Continuity (1)
Related Publication 20220067082A1 · Mar 3, 2022