Few-shot learning for processing microscopy images
Disclosed herein are scientific instrument support systems, as well as related methods, computing devices, and computer-readable media. For example, in some embodiments, a scientific instrument support apparatus may include: first logic to receive, from a charged particle microscope, a microscopy image of a sample; second logic to generate a first processed image by processing the microscopy image through a general machine-learning model trained using a plurality of previously processed microscopy images; third logic to retrain the general machine-learning model with a related microscopy image, wherein the related microscopy image includes a label of an object related to the sample and the related microscopy image is not included in the plurality of previously processed microscopy images; and fourth logic to generate a second processed image, different from the first processed image, by processing the microscopy image through the retrained general machine-learning model.
1 . A scientific instrument support apparatus, comprising:
first logic to receive, from a charged particle microscope, a microscopy image of a sample;
second logic to generate a first processed image by applying image processing to the microscopy image through a general machine-learning model trained using a plurality of previously processed microscopy images, wherein the first processed image is an output of the general machine-learning model;
third logic to, based on the first processed image being unsatisfactory, select a related microscopy image based on a metric quantitating the similarity of the first processed image to an expected processing results of the microscopy image and to retrain the general machine-learning model with new training data including the related microscopy image, wherein the related microscopy image includes the first processed image supplied with a label of an object related to the sample and the related microscopy image is not included in the plurality of previously processed microscopy images; and
fourth logic to generate a second processed image, different from the first processed image, by processing the microscopy image through the retrained general machine-learning model.
2 . The scientific instrument support apparatus of claim 1 , fifth logic to provide the second processed image for display or further processing.
3 . The scientific instrument support apparatus of claim 1 , wherein the third logic is configured to determine that the first processed image is unsatisfactory based on the Jaccard index.
4 . The scientific instrument support apparatus of claim 1 , wherein the label includes corrections or annotations provided by a user.
5 . The scientific instrument support apparatus of claim 1 , wherein the third logic is further configured to select the related microscopy image based on input provided by a user via an interface.
6 . The scientific instrument support apparatus of claim 1 , wherein the microscopy image is added to the previously processed microscopy images to form a set of updated microscopy images, wherein a second general machine-learning model is trained by a second scientific instrument support apparatus using the set of updated microscopy images, and wherein the second scientific instrument support apparatus employs the second general machine-learning model to generate processed images.
7 . The scientific instrument support apparatus of claim 1 , wherein the general machine-learning model is trained through supervised learning using data augmentation with previously annotated data.
8 . The scientific instrument support apparatus of claim 1 , wherein the sample comprises a semiconductor device, and wherein each of the processed microscopy images are of the semiconductor device.
9 . The scientific instrument support apparatus of claim 1 , wherein the microscopy image comprises a transmission electron microscopy (TEM) image or scanning electron microscopy (SEM) image.
10 . The scientific instrument support apparatus of claim 1 , wherein the general machine-learning model comprises a convolutional neural network.
11 . The scientific instrument support apparatus of claim 1 , wherein the scientific instrument support apparatus is deployed to the charged particle microscope.
12 . A method for scientific instrument support executed by an electronic processor, the method comprising:
receiving, from a charged particle microscope, a microscopy image of a sample;
generating a first processed image by applying image processing to the microscopy image through a general machine-learning model trained using a plurality of previously processed microscopy images, wherein the first processed image is an output of the general machine-learning model;
determining an indication that the first processed image is unsatisfactory;
selecting a related microscopy image based on a metric quantitating the similarity of the first processed image to an expected processing result of the microscopy image;
retraining the general machine-learning model with new training data including the related microscopy image in response to the determination that the first processed image is unsatisfactory, wherein the related microscopy image is the first processed image supplied with a label of an object related to the sample; and
generating a second processed image, different from the first processed image, by processing the microscopy image through the retrained general machine-learning model.
13 . The method of claim 12 , wherein the related microscopy image is not included in the plurality of previously processed microscopy images.
14 . The method of claim 12 , wherein the indication that the first processed image is unsatisfactory is received from a user interface.
15 . The method of claim 12 , wherein the plurality of previously processed microscopy images is fine-tuned based on a type of samples processed or specific requirements of a project or a device comprising the electronic processor.
16 . A scientific instrument support system, comprising:
a charged particle microscope; and
an electronic processor configured to:
receive, from the charged particle microscope, a microscopy image of a sample;
generate a first processed image by applying image processing to the microscopy image through a general machine-learning model trained using a plurality of previously processed microscopy images, wherein the first processed image is an output of the general machine-learning model;
select a related microscopy image based on a metric quantitating the similarity of the first processed image to an expected processing results of the microscopy image;
retrain the general machine-learning model with new training data including the related microscopy image, the related image including the first processed image supplied with a label of an object related to the sample; and
generate a second processed image, different from the first processed image, by processing the microscopy image through the retrained general machine-learning model.
17 . The scientific instrument support system of claim 16 , wherein the related microscopy image is not included in the plurality of previously processed microscopy images.