IP Library Granted Patent US 10,789,695
Granted Patent B2
US 10,789,695 · App. 16/576,732 · Granted Sep 29, 2020

Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging

Inventors: Matthew C. Putman (Brooklyn, NY); John B. Putman (Celebration, FL); Vadim Pinskiy (Wayne, NJ); Joseph Succar (Brooklyn, NY)
Assignee: Nanotronics Imaging, Inc.
G06T5/50G06K9/00134G06K9/036G06K9/627G06K9/6269G06T3/4053
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,789,695
App. No.
16/576,732
Granted
Sep 29, 2020
Kind
B2
Abstract

Systems, methods, and computer-readable media for feedback on and improving the accuracy of super-resolution imaging. In some embodiments, a low resolution image of a specimen can be obtained using a low resolution objective of a microscopy inspection system. A super-resolution image of at least a portion of the specimen can be generated from the low resolution image of the specimen using a super-resolution image simulation. Subsequently, an accuracy assessment of the super-resolution image can be identified based on one or more degrees of equivalence between the super-resolution image and one or more actually scanned high resolution images of at least a portion of one or more related specimens identified using a simulated image classifier. Based on the accuracy assessment of the super-resolution image, it can be determined whether to further process the super-resolution image. The super-resolution image can be further processed if it is determined to further process the super-resolution image.

Claims (56)

1. A method for generating a super-resolution image for a specimen through a super-resolution system based on a low resolution image of the specimen comprising:

obtaining the low resolution image of the specimen using a low resolution objective of a microscopy inspection system;

detecting one or more artifacts in the low resolution image;

identifying a suitability of the one or more artifacts for generation of the super-resolution image of at least a portion of the specimen from the low resolution image, wherein the suitability of the one or more artifacts is identified by a suitability classifier trained from a first group of known artifacts suitable for super-resolution imaging and a second group of known artifacts unsuitable for the super-resolution imaging;

generating the super-resolution image of the at least the portion of the specimen from the low resolution image of the specimen using a super-resolution image simulation;

identifying an accuracy assessment of the super-resolution image based on one or more degrees of equivalence between the super-resolution image and one or more actually scanned high resolution images of at least a portion of one or more related specimens identified using a simulated image classifier;

determining whether to further process the super-resolution image based on the accuracy assessment of the super-resolution image and the suitability of the one or more artifacts; and

further processing the super-resolution image if it is determined to further process the super-resolution image.

2. The method of claim 1 , further comprising:

obtaining one or more high resolution images of the at least the portion of the specimen using a high resolution objective of the microscopy inspection system, if it is determined to further process the super-resolution image based on the accuracy assessment of the super-resolution image; and

assembling the super-resolution image and the one or more high resolution images of the at least the portion of the specimen to form a single coherent image of the at least the portion of the specimen as part of further processing the super-resolution image.

3. The method of claim 2 , wherein the one or more high resolution images of the at least the portion of the specimen obtained using the high resolution objective are high resolution images of the one or more artifacts in the low resolution image.

4. The method of claim 1 , wherein the one or more artifacts in the low resolution image are identified as unsuitable for generating the super-resolution image.

5. The method of claim 4 , further comprising:

obtaining one or more high resolution images of the one or more artifacts using a high resolution objective of the microscopy inspection system, if it is determined to further process the super-resolution image based on the suitability of the one or more artifacts for generation of the super-resolution image; and

assembling the super-resolution image and the one or more high resolution images of the one or more artifacts to form a single coherent image of the at least the portion of the specimen as part of further processing the super-resolution image.

6. The method of claim 1 , wherein the one or more artifacts in the low resolution image are identified as suitable for generating the super-resolution image, the method further comprising generating the super-resolution image of the at least the portion of the specimen from the low resolution image using the super-resolution image simulation in response to an identification that the one or more artifacts are suitable for generating the super-resolution image.

7. The method of claim 6 , wherein the super-resolution image is generated from at least the one or more artifacts in the low resolution image identified as suitable for generating the super-resolution image.

8. The method of claim 1 , wherein the simulated image classifier is trained using a plurality of known super-resolution images at two or more different image confidence determinations.

9. A super-resolution system comprising:

a microscopy inspection system for inspecting a specimen comprising:

a low resolution objective;

a high resolution objective;

one or more processors; and

at least one computer-readable storage medium having stored therein instructions which, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

obtaining a low resolution image of the specimen using the low resolution objective of a microscopy inspection system;

detecting one or more artifacts in the low resolution image;

identifying a suitability of the one or more artifacts for generation of a super-resolution image of at least a portion of the specimen from the low resolution image, wherein the suitability of the one or more artifacts is identified by a suitability classifier trained from a first group of known artifacts suitable for super-resolution imaging and a second group of known artifacts unsuitable for super resolution imaging;

generating a super-resolution image of the at least the portion of the specimen from the low resolution image of the specimen using a super-resolution image simulation;

identifying an accuracy assessment of the super-resolution image based on one or more degrees of equivalence between the super-resolution image and one or more actually scanned high resolution images of at least a portion of one or more related specimens;

determining whether to further process the super-resolution image based on the accuracy assessment of the super-resolution image and the suitability of the one or more artifacts; and

further processing the super-resolution image if it is determined to further process the super-resolution image.

10. The system of claim 9 , wherein the instructions which, when executed by the one or more processors, further cause the one or more processors to perform operations comprising:

obtaining one or more high resolution images of the at least the portion of the specimen using the high resolution objective of the microscopy inspection system, if it is determined to further process the super-resolution image based on the accuracy assessment of the super-resolution image; and

assembling the super-resolution image and the one or more high resolution images of the at least the portion of the specimen to form a single coherent image of the at least the portion of the specimen as part of further processing the super-resolution image.

11. The system of claim 10 , wherein the one or more high resolution images of the at least the portion of the specimen obtained using the high resolution objective are high resolution images of the one or more artifacts in the low resolution image.

12. The system of claim 9 , wherein the one or more artifacts are identified as unsuitable for generating the super-resolution image.

13. The system of claim 12 , wherein the instructions which, when executed by the one or more processors, further cause the one or more processors to perform operations comprising:

obtaining one or more high resolution images of the one or more artifacts using the high resolution objective of the microscopy inspection system, if it is determined to further process the super-resolution image based on the suitability of the one or more artifacts for generation of the super-resolution image; and

assembling the super-resolution image and the one or more high resolution images of the one or more artifacts to form a single coherent image of the at least the portion of the specimen as part of further processing the super-resolution image.

14. The system of claim 9 , wherein the one or more artifacts in the low resolution image are identified as suitable for generating the super-resolution image and the instructions which, when executed by the one or more processors, further cause the one or more processors to perform operations comprising generating the super-resolution image of the at least the portion of the specimen from the low resolution image using the super-resolution image simulation in response to an identification that the one or more artifacts are suitable for generating the super-resolution image.

15. The system of claim 9 , wherein the accuracy assessment of the super-resolution image is identified by a simulated image classifier trained using a plurality of known super-resolution images at two or more different image confidence determinations.

16. A non-transitory computer-readable storage medium having stored therein instructions which, when executed by one or more processors, cause the one or more processors to perform operations for generating a super-resolution image for a specimen based on a low resolution image of the specimen comprising:

receiving the low resolution image of the specimen captured by a low resolution objective of a microscopy inspection system;

detecting one or more artifacts in the low resolution image;

identifying a suitability of the one or more artifacts for generation of the super-resolution image of at least a portion of the specimen from the low resolution image, wherein the suitability of the one or more artifacts is identified by a suitability classifier trained from a first group of known artifacts suitable for super-resolution imaging and a second group of known artifacts unsuitable for super resolution imaging;

generating the super-resolution image of the at least the portion of the specimen from the low resolution image of the specimen using a super-resolution image simulation;

identifying an accuracy assessment of the super-resolution image based on one or more degrees of equivalence between the super-resolution image and one or more actually scanned high resolution images of at least a portion of one or more related specimens identified using a simulated image classifier;

determining whether to further process the super-resolution image based on the accuracy assessment of the super-resolution image and the suitability of the one or more artifacts; and

further processing the super-resolution image if it is determined to further process the super-resolution image.

17. The non-transitory computer-readable storage medium of claim 16 , wherein the operations for generating the super-resolution image for the specimen based on the low resolution image of the specimen further comprise:

obtaining one or more high resolution images of the at least the portion of the specimen using a high resolution objective of the microscopy inspection system, if it is determined to further process the super-resolution image based on the accuracy assessment of the super-resolution image; and

assembling the super-resolution image and the one or more high resolution images of the at least the portion of the specimen to form a single coherent image of the at least the portion of the specimen as part of further processing the super-resolution image.

18. The non-transitory computer-readable storage medium of claim 17 , wherein the one or more high resolution images of the at least the portion of the specimen obtained using the high resolution objective are high resolution images of the one or more artifacts in the low resolution image.

19. The non-transitory computer-readable storage medium of claim 16 , wherein the one or more artifacts in the low resolution image are identified as unsuitable for generating the super-resolution image.

20. The non-transitory computer-readable storage medium of claim 16 , wherein the accuracy assessment of the super-resolution image is identified by a simulated image classifier trained using a plurality of known super-resolution images at two or more different image confidence determinations.

Assignments (2)
SECURITY INTEREST Recorded Nov 30, 2023
From: NANOTRONICS IMAGING, INC.; NANOTRONICS HEALTH LLC; CUBEFABS INC.
To: ORBIMED ROYALTY & CREDIT OPPORTUNITIES IV, LP
Reel/Frame 065726/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 14, 2019
From: PUTMAN, MATTHEW C.; PUTMAN, JOHN B.; PINSKIY, VADIM; SUCCAR, JOSEPH R.
To: NANOTRONICS IMAGING, INC.
Reel/Frame 051007/0423 →
Continuity (3)
Continuation 16233258 · Dec 27, 2018
Continuation 16027056 · Jul 3, 2018
Related Publication 20200013155A1 · Jan 9, 2020
Cited By (1)
US 12,682,441