Image processing system and image processing method
To implement image synthesis without loss of surface information of SE images and shadow information of BSE images, the present disclosure proposes image processing techniques include applying data of a first quality image (low quality image) to a trained model, estimating a structural feature and a material feature of a second quality image (high quality image) corresponding to the first quality image, calculating at least one shadow datum based on the structural feature and a synthesis parameter and calculating at least one gradation datum based on the material feature and a synthesis parameter, generating a synthesized image from the at least one shadow datum and the at least one gradation datum, and outputting the synthesized image as a prediction result of the second quality image (see FIG. 8 ).
1 . An image processing system that trains a machine learning model using a plurality of images acquired with respect to a target sample under different imaging conditions, to generate a trained model for converting a first quality image into a second quality image having a higher image quality than the first quality image, the image processing system comprising:
a storage device that stores a training process program for generating the trained model; and
a computer that reads the training process program from the storage device and executes a training process for generating the trained model,
wherein the computer is configured to execute:
a process of receiving at least one machine learning model, a plurality of first quality images, and a plurality of second quality images having a higher image quality than the first quality images;
a process of applying data of the plurality of first quality images to the at least one machine learning model and estimating a structural feature and a material feature of the second quality image corresponding to the plurality of first quality images;
a process of calculating at least one first shadow datum based on the structural feature;
a process of calculating at least one second shadow datum from the plurality of second quality images;
a first comparing process of comparing the at least one first shadow datum and the at least one second shadow datum;
a process of calculating at least one first gradation datum based on the material feature;
a process of calculating at least one second gradation datum from the plurality of second quality images;
a second comparing process of comparing the at least one first gradation datum and the at least one second gradation datum; and
a process of updating parameters of the at least one machine learning model based on a first comparison result obtained by the first comparing process and a second comparison result obtained by the second comparing process.
2 . The image processing system according to claim 1 ,
wherein the plurality of first quality images and the plurality of second quality images each include at least one secondary electron image and at least one backscattered electron image acquired by irradiating the target sample with a charged particle beam using a charged particle beam apparatus.
3 . The image processing system according to claim 2 ,
wherein the computer applies the at least one backscattered electron image of the first quality image to the at least one machine learning model and estimates a structural feature of the second quality image.
4 . The image processing system according to claim 2 ,
wherein the computer applies the at least one secondary electron image of the first quality image to the at least one machine learning model and estimates a material feature of the second quality image.
5 . The image processing system according to claim 1 ,
wherein the plurality of first quality images and the plurality of second quality images each include at least one shaded image.
6 . The image processing system according to claim 1 ,
wherein the structural feature includes at least one of a normal map, a bump map, a height map, or a displacement map, representing layer structures or surface irregularities of the target sample.
7 . The image processing system according to claim 1 ,
wherein the material feature includes a brightness map representing differences in acquired signals caused by a material of the target sample as gradation.
8 . The image processing system according to claim 1 ,
wherein the computer further executes a process of completing training of the at least one machine learning model based on whether the first comparison result and the second comparison result are less than or equal to a predetermined threshold, and obtaining a model as of the completion as the trained model.
9 . An image processing system that applies a first quality image of a target sample to a trained model to predict and output a second quality image having a higher image quality than the first quality image, the image processing system comprising:
a storage device that stores an image synthesis program for synthesizing the second quality image using the trained model; and
a computer that reads the image synthesis program from the storage device and executes an image synthesis process for synthesizing the second quality image,
wherein the computer is configured to execute:
a process of receiving a plurality of first quality images and a synthesis parameter;
a process of applying data of the first quality image to the trained model and estimating a structural feature and a material feature of the second quality image corresponding to the first quality image;
a process of calculating at least one shadow datum based on the structural feature and the synthesis parameter;
a process of calculating at least one gradation datum based on the material feature and the synthesis parameter; and
a process of generating a synthesized image from the at least one shadow datum and the at least one gradation datum, and outputting the synthesized image as a prediction result of the second quality image.
10 . The image processing system according to claim 9 ,
wherein the plurality of first quality images each include at least one secondary electron image and at least one backscattered electron image acquired by irradiating the target sample with a charged particle beam using a charged particle beam apparatus.
11 . The image processing system according to claim 10 ,
wherein the synthesis parameter includes a parameter specifying a shadow of the synthesized image and a parameter specifying a brightness of the synthesized image.
12 . The image processing system according to claim 11 ,
wherein the parameter specifying a shadow is a three-dimensional unit vector representing a direction of a detector that acquires a signal of the backscattered electron image, and
the parameter specifying a brightness is a mixing ratio between the at least one shadow datum and the at least one gradation datum.
13 . The image processing system according to claim 9 ,
wherein the computer applies the at least one backscattered electron image of the first quality image to the trained model and estimates a structural feature of the second quality image, and applies the at least one secondary electron image of the first quality image to the trained model and estimates a material feature of the second quality image.
14 . The image processing system according to claim 9 ,
wherein the computer receives a result of user confirmation of the synthesized image, and determines whether to execute again a synthesized image generation process.
15 . The image processing system according to claim 14 ,
wherein the computer receives again a synthesis parameter including a parameter value different from the synthesis parameter used in the synthesized image generation process previously performed, and executes again the synthesized image generation process using the synthesis parameter that the computer has received again.
16 . An image processing method performed by a computer that trains a machine learning model using a plurality of images acquired with respect to a target sample under different imaging conditions, to generate a trained model for converting a first quality image into a second quality image having a higher image quality than the first quality image, the image processing method comprising:
receiving at least one machine learning model, a plurality of first quality images, and a plurality of second quality images having a higher image quality than the first quality images;
applying data of the plurality of first quality images to the at least one machine learning model and estimating a structural feature and a material feature of the second quality image corresponding to the plurality of first quality images;
calculating at least one first shadow datum based on the structural feature;
calculating at least one second shadow datum from the plurality of second quality images;
comparing the at least one first shadow datum and the at least one second shadow datum and acquiring a first comparison result;
calculating at least one first gradation datum based on the material feature;
calculating at least one second gradation datum from the plurality of second quality images;
comparing the at least one first gradation datum and the at least one second gradation datum and acquiring a second comparison result; and
updating parameters of the at least one machine learning model based on the first comparison result and the second comparison result.
17 . An image processing method performed by a computer that applies a first quality image of a target sample to a trained model, to predict and output a second quality image having a higher image quality than the first quality image, the image processing method comprising:
receiving a plurality of first quality images and a synthesis parameter;
applying data of the first quality image to the trained model and estimating a structural feature and a material feature of the second quality image corresponding to the first quality image;
calculating at least one shadow datum based on the structural feature and the synthesis parameter;
calculating at least one gradation datum based on the material feature and the synthesis parameter; and
generating a synthesized image from the at least one shadow datum and the at least one gradation datum, and outputting the synthesized image as a prediction result of the second quality image.