IP Library Granted Patent US 11,798,270
Granted Patent B2
US 11,798,270 · App. 17/216,746 · Granted Oct 24, 2023

Systems and methods for image classification

Inventors: Yousef Al-Kofahi (San Jose, CA); Michael MacDonald (San Jose, CA); Asha Singanamalli (San Jose, CA); Mohammed Yousefhussien (San Jose, CA); Will Marshall (San Jose, CA)
Assignee: Molecular Devices, LLC
G06V10/82G06F18/2411G06F18/2413G06T7/11G06V10/764G06V10/776G06V10/945
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,798,270
App. No.
17/216,746
Granted
Oct 24, 2023
Kind
B2
Abstract

In one aspect, a method for inspecting features of an image using an image inspection controller that includes a processor communicatively coupled to a memory is described. The method includes receiving, at the processor, an input image, performing, on the input image, one of a semantic segmentation process and an object classification process to generate an output image, and prompting a user to select between approving the displayed output image, and at least one of i) performing an additional semantic segmentation process on the displayed output image, and ii) performing an additional object classification process on the displayed output image.

Claims (61)

1. A method for inspecting features of an image using an image inspection controller that includes a processor communicatively coupled to a memory, said method comprising:

receiving, at the processor, an input image;

performing, on the input image, one of a semantic segmentation process and an object classification process to generate an output image;

wherein the semantic segmentation process comprises:

annotating, by the processor, at least one segment of the input image to produce a semantic segmentation annotated image;

generating, by the processor, a semantic segmentation model based on a semantic segmentation training vector derived from the semantic segmentation annotated image;

applying, by the processor, the semantic segmentation model to each pixel of the input image to generate the output image; and

displaying the output image; and

wherein the object classification process comprises:

annotating, by the processor, at least one object in an object mask to produce an object classification annotated image;

generating, by the processor, an object classification model based on an object classification training vector derived from the object classification annotated image;

applying, by the processor, the object classification model to the object mask to generate the output image, and

displaying the output image; and

prompting a user to select between approving and rejecting the displayed output image, and, in response to the user rejecting the displayed output image, performing at least one of i) an additional semantic segmentation process on the displayed output image, and ii) an additional object classification process on the displayed output image.

2. The method in accordance with claim 1 , further comprising receiving user input indicating a selection of performing the additional semantic segmentation process on the displayed output image.

3. The method in accordance with claim 2 , further comprising performing the additional semantic segmentation process on the displayed output image, the additional semantic segmentation process comprising generating and displaying an updated output image.

4. The method in accordance with claim 3 , further comprising prompting the user to select between approving the displayed updated output image, and at least one of i) performing a further semantic segmentation process on the displayed updated output image, and ii) performing a further object classification process on the displayed updated output image.

5. The method in accordance with claim 1 , further comprising receiving user input indicating a selection of performing the additional object classification process on the displayed output image.

6. The method in accordance with claim 5 , further comprising performing the additional object classification process on the displayed output image, the additional object classification process comprising generating and displaying an updated output image.

7. The method in accordance with claim 6 , further comprising prompting the user to select between approving the displayed updated output image, and at least one of i) performing a further semantic segmentation process on the displayed updated output image, and ii) performing a further object classification process on the displayed updated output image.

8. The method in accordance with claim 1 , wherein annotating at least one segment of the input image or annotating at least one object in the object mask comprises annotating the input image or the object mask, based on user input, on a graphical user interface displayed on a display device communicatively coupled to the processor.

9. An image inspection computing device comprising:

a memory device; and

at least one processor communicatively coupled to said memory device, wherein said at least one processor is configured to:

receive an input image;

perform, on the input image, one of a semantic segmentation process and an object classification process to generate an output image;

wherein the semantic segmentation process comprises:

annotating, by the processor, at least one segment of the input image to produce a semantic segmentation annotated image;

generating, by the processor, a semantic segmentation model based on a semantic segmentation training vector derived from the semantic segmentation annotated image;

applying, by the processor, the semantic segmentation model to each pixel of the input image to generate the output image; and

displaying the output image; and

wherein the object classification process comprises:

annotating, by the processor, at least one object in an object mask to produce an object classification annotated image;

generating, by the processor, an object classification model based on an object classification training vector derived from the object classification annotated image;

applying, by the processor, the object classification model to the object mask to generate the output image; and

displaying the output image; and

prompt a user to select between approving and rejecting the displayed output image, and, in response to the user rejecting the displayed output image, performing at least one of i) an additional semantic segmentation process on the displayed output image, and ii) an additional object classification process on the displayed output image.

10. The image inspection computing device in accordance with claim 9 , wherein said processor is further configured to receive user input indicating a selection of performing the additional semantic segmentation process on the displayed output image.

11. The image inspection computing device in accordance with claim 10 , wherein said processor is further configured to perform the additional semantic segmentation process on the displayed output image, the additional semantic segmentation process including generating and displaying an updated output image.

12. The image inspection computing device in accordance with claim 11 , wherein said processor is further configured to prompt the user to select between approving the displayed updated output image, and at least one of i) performing a further semantic segmentation process on the displayed updated output image, and ii) performing a further object classification process on the displayed updated output image.

13. The image inspection computing device in accordance with claim 9 , wherein said processor is further configured to receive user input indicating a selection of performing the additional object classification process on the displayed output image.

14. The image inspection computing device in accordance with claim 13 , wherein said processor is further configured to perform the additional object classification process on the displayed output image, the additional object classification process including generating and displaying an updated revised output image.

15. The image inspection computing device in accordance with claim 14 , wherein said processor is further configured to prompt the user to select between approving the displayed updated output image, and at least one of i) performing a further semantic segmentation process on the displayed updated output image, and ii) performing a further object classification process on the displayed updated output image.

16. The image inspection computing device in accordance with claim 9 , wherein to annotate at least one segment of the input image or to annotate at least one object in the object mask, said processor is configured to annotate the input image or the object mask, based on user input, on a graphical user interface displayed on a display device communicatively coupled to said processor.

17. A non-transitory computer-readable storage media having computer-executable instructions embodied thereon, wherein when executed by a computing device comprising at least one processor in communication with a memory, the computer-executable instructions cause the computing device to:

receive, at the processor, an input image;

perform, on the input image, one of a semantic segmentation process and an object classification process to generate an output image;

wherein the semantic segmentation process comprises:

annotating, by the processor, at least one segment of the input image to produce a semantic segmentation annotated image;

generating, by the processor, a semantic segmentation model based on a semantic segmentation training vector derived from the semantic segmentation annotated image;

applying, by the processor, the semantic segmentation model to each pixel of the input image to generate the output image; and

displaying the output image; and

wherein the object classification process comprises:

annotating, by the processor, at least one object in an object mask to produce an object classification annotated image;

generating, by the processor, an object classification model based on an object classification training vector derived from the object classification annotated image;

applying, by the processor, the object classification model to the object mask to generate the output image; and

displaying the output image; and

prompt a user to select between approving and rejecting the displayed output image, and, in response to the user rejecting the displayed output image, performing at least one of i) an additional semantic segmentation process on the displayed output image, and ii) an additional object classification process on the displayed output image.

18. The non-transitory computer-readable storage media in accordance with claim 17 , wherein to annotate at least one segment of the input image, the computer-executable instructions cause the computing device to receive user input indicating a selection of performing the additional semantic segmentation process on the displayed output image.

19. The non-transitory computer-readable storage media in accordance with claim 17 , wherein the computer-executable instructions cause the computing device to convert the object mask to a semantic segmentation mask.

20. The non-transitory computer-readable storage media in accordance with claim 17 , wherein the computer-executable instructions cause the computing device to convert a semantic segmentation mask to the object mask.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 21, 2021
From: GLOBAL LIFE SCIENCES SOLUTIONS USA LLC
To: MOLECULAR DEVICES, LLC
Reel/Frame 056596/0481 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 30, 2021
From: AL-KOFAHI, YOUSEF; MACDONALD, MICHAEL; SINGANAMALLI, ASHA; YOUSEFHUSSEIN, MOHAMMED; MARSHALL, WILL
To: GLOBAL LIFE SCIENCES SOLUTIONS USA LLC
Reel/Frame 055761/0237 →
Continuity (2)
Provisional Application 63016075 · Apr 27, 2020
Related Publication 20210334607A1 · Oct 28, 2021
Cited By (1)
US 12,205,362