Scanning electron microscope image processing using neural network
A neural network device includes: (1) a pre-processor configured to select target images from scanning electron microscope (SEM) images, based on frequencies respectively corresponding to the SEM images, and crop each of the target images into a plurality of cropped images; (2) a neural network processor configured to generate a crop detection image by inferring a target object from each of the plurality of cropped images by using a segmentation model trained to detect the target object in each of the plurality of cropped images; and (3) a post-processor configured to merge crop detection images with each other in a same size as the SEM images, based on position information of the plurality of cropped images.
1 . A neural network device comprising:
a pre-processor configured to;
convert time domain information of each scanning electron microscope (SEM) image of SEM images into frequency domain information,
generate, from the frequency domain information, a frequency image corresponding to each of the SEM images,
select target images from the SEM images based on a frequency value of at least a portion of each of the frequency images corresponding to the SEM images,
set at least another portion of each of the frequency images as a zero-padding area by substituting zero for frequency values respectively corresponding to pixels of the at least another portion of the frequency image, wherein the zero-padding area is set based on a center of the frequency image, and
crop each of the target images into a plurality of cropped images;
a neural network processor configured to generate crop detection images by inferring a target object from each of the plurality of cropped images by using a segmentation model trained to detect the target object in each of the plurality of cropped images; and
a post-processor configured to merge the crop detection images with each other to generate a merged image having the same size as the SEM images, based on position information of the plurality of cropped images.
2 . The neural network device of claim 1 , wherein the pre-processor is further configured to generate a calculation value based on frequency values respectively corresponding to pixels of an area other than the zero-padding area in each of the frequency images and select, as a target image of the target images, an SEM image corresponding to the frequency image for which the calculation value is greater than or equal to a preset threshold value.
3 . The neural network device of claim 1 , wherein the pre-processor is further configured to set a plurality of zero-padding areas having different sizes and generate a calculation value corresponding to each of the plurality of zero-padding areas, based on frequency values respectively corresponding to pixels of an area other than each of the plurality of zero-padding areas in each of the frequency images.
4 . The neural network device of claim 3 , wherein the pre-processor is further configured to calculate an average calculation value corresponding to an average of calculation values respectively corresponding to the plurality of zero-padding areas and select, as one of the target images, an SEM image corresponding to a frequency image for which the average calculation value is greater than or equal to a preset threshold value.
5 . The neural network device of claim 1 , wherein:
the plurality of cropped images include a first cropped image and a second cropped image adjacent to the first cropped image, wherein a portion of the first cropped image overlaps with a portion of the second cropped image.
6 . The neural network device of claim 5 , wherein the post-processor is further configured to remove the portion of the first cropped image from the first cropped image, the portion of the first cropped image overlapping with the portion of the second cropped image, and merge the first cropped image with the second cropped image.
7 . The neural network device of claim 1 , wherein:
the neural network processor is further configured to train the segmentation model based on a training SEM image, an augmented training image, a labeled image corresponding to the training SEM image, and a labeled image corresponding to the augmented training image, and
the augmented training image is generated by an augmentation technique of rotating, flipping, resizing, changing luminance of, adding noise to, and cropping at least one training SEM image.
8 . The neural network device of claim 1 , wherein the neural network processor is further configured to train the segmentation model by using a skip connection.
9 . The neural network device of claim 1 , wherein the neural network processor is further configured to train the segmentation model to prevent overfitting.
10 . The neural network device of claim 1 , wherein the neural network processor is further configured to perform transfer learning based on at least one of the plurality of cropped images and update the segmentation model that has been trained.
11 . A system comprising:
at least one processor; and
a non-transitory storage medium storing instructions configured to cause the at least one processor to perform image processing when the instructions are executed by the at least one processor, wherein the image processing includes:
obtaining N scanning electron microscope (SEM) images from an SEM with respect to an analysis target;
selecting M target images from the N SEM images, based on frequencies respectively corresponding to the N SEM images, wherein the selecting of the M target images includes:
generating N frequency images by converting time domain information of each of the N SEM images into frequency domain information,
setting a zero-padding area in each of the N frequency images and substituting zero for frequency values respectively corresponding to pixels of the zero-padding area,
generating a calculation value by calculating an average of frequency values respectively corresponding to pixels of an area other than the zero-padding area in each of the N frequency images, and
determining whether the calculation value is greater than or equal to a preset threshold value and selecting, as one of the M target images, an SEM image based on the calculation value;
cropping each of the M target images into K cropped images;
generating M×K crop detection images by inferring a target object, to which each of pixels of M×K cropped images belongs, by using a segmentation model trained to detect the target object; and
generating M predicted images by merging the M×K crop detection images with each other based on position information of the M×K cropped images, wherein each of the M predicted images has the same size as the N SEM images.
12 . The system of claim 11 , wherein the zero-padding area is the same among the N frequency images.
13 . The system of claim 11 , wherein each of the K cropped images at least partially overlaps with an adjacent one among the K cropped images.
14 . An operating method of a neural network device, the operating method comprising:
obtaining scanning electron microscope (SEM) images from an SEM with respect to an analysis target;
selecting target images from the SEM images, based on frequencies respectively corresponding to the SEM images, wherein the selecting of the target images includes:
generating frequency images by converting time domain information of each of the SEM images into frequency domain information,
setting a plurality of zero-padding areas of different sizes in each of the frequency images and generating calculation values respectively corresponding to the plurality of zero-padding areas,
calculating an average calculation value for each of the frequency images by calculating an average of the calculation values respectively corresponding to the plurality of zero-padding areas, and
selecting, as one of the target images, an SEM image corresponding to a frequency image for which the average calculation value is greater than or equal to a preset threshold value;
cropping each of the target images into a plurality of cropped images;
generating crop detection images by inferring a target object from each of the plurality of cropped images by using a segmentation model trained to detect the target object in each of the plurality of cropped images; and
merging the crop detection images with each other to generate a merged image having the same size as the SEM images, based on position information of the plurality of cropped images.