Fusion model training using distance metrics
A method and a system are presented for controlling a performance of a fusion model. The method includes obtaining a first set and a second set of candidate models for a first and second neural networks, respectively. Each of the first and second set of candidate models is pre-trained with a first source and a second source, respectively. For each possible pairing of one candidate model from the first neural network and one candidate model from the second neural network, a model distance D m is determined. A subset of possible pairings of one first candidate model and one second candidate model is selected based on the model distance D m between them. Using the subset of possible parings, the first neural network and the second neural network are combined to generate two branches for a fusion model neural network.
1 . A computer-implemented method comprising:
determining model distances between each of pairings of first candidate models of a first neural network and second candidate models of a second neural network, the determining based on the calculated distances between first latent feature spaces of the first candidate models and second latent feature spaces of the second candidate models in response to data samples of an input image;
selecting a subset of pairings from among the pairings of the first candidate models and the second candidate models based on respective model distances between them, the selected subset of pairings having the model distances shorter than other pairings from among the pairings of the first candidate models and the second candidate models;
combining the first neural network and the second neural network to join two branch neural networks for a fusion model neural network based on the selected subset of pairings;
training the fusion model neural network based on the selected subset of pairings to improve performance of the fusion model neural network; and
detecting a defect in an image based on the fusion model neural network having improved performance.
2 . The computer-implemented method of claim 1 , wherein:
each of the first candidate models is pre-trained with a first source;
each of the second candidate models is pre-trained with a second source; and
each of the model distances and the improved performance of the fusion model neural network is based on mutual information.
3 . The computer-implemented method of claim 2 , wherein the distance based on mutual information is one of a Wasserstein distance with mutual information loss as a measure, Euclidean distance, Mahalanobix distance, and Maximum mean discrepancy.
4 . The computer-implemented method of claim 2 , wherein selecting the subset of candidate models for the first neural network and the second neural network comprises selecting a pair that produces a shortest model distance, wherein the shortest model distance is a Wasserstein distance with mutual information loss as a measure.
5 . The computer-implemented method of claim 2 , wherein the first source is a form of the input image and the data samples.
6 . The computer-implemented method of claim 5 , wherein the second source is a form of the input image and the data samples, and the first source and the second source are different.
7 . The computer-implemented method of claim 2 , wherein each of the first source and the second source is one of a spectroscopy image and a microscopy image.
8 . The computer-implemented method of claim 2 , wherein the first candidate models are models trained from the first source using different model structures and hyper-parameters.
9 . The computer-implemented method of claim 2 , wherein each of the first source and the second source is one of Transmission Electron Microscopy (TEM) image and High Angle Annular Dark Field (HAADF) image.
10 . The computer-implemented method of claim 1 , wherein the selected subset of pairings from among the pairings of the first candidate models and the second candidate models have a shortest model distance from among the model distances between the pairings of the first candidate models and the second candidate models.
11 . A system for image-based predictions, comprising:
a first neural network;
a second neural network;
a processor determining model distances between each of pairings of first candidate models of the first neural network and second candidate models of the second neural network, the determining based on the calculated distances between first latent feature spaces of the first candidate models and second latent feature spaces of the second candidate models in response to data samples of an input image; and
a fusion model resulting from a selected pairing of the first neural network and the second neural network, the selected pairing of the first neural network and the second neural network having a first model distance that is shorter than the model distances of other pairings of the first candidate models of the first neural network and the second candidate models of the second neural network, the processor detecting a defect in an image for the image-based predictions based on the fusion model neural network.
12 . The system of claim 11 , wherein:
the first neural network is pre-trained with a first set of candidate models comprising the first candidate models using a first source as input data;
the second neural network is pre-trained with a second set of candidate models comprising the second candidate models using a second source as input data;
the fusion model is trained based on the selected pairing of the first neural network and the second neural network to improve performance of the fusion model;
the first model distance is between a first candidate model from among the first candidate models of the first neural network and a second candidate model from among the second candidate models of the second neural network; and
each of the model distances and the improved performance of the fusion model neural network is based on mutual information.
13 . The system of claim 12 , wherein the distance based on mutual information is one of a Wasserstein distance with mutual information loss as a measure, Euclidean distance, Mahalanobis distance, and Maximum mean discrepancy.
14 . The system of claim 12 , wherein the processor selects a pair of candidate models based on the model distance, wherein the model distance is a Wasserstein distance with mutual information loss as a measure, and wherein the pair of candidate models comprises a first candidate model of the first candidate models and a second candidate model of the second candidate models.
15 . The system of claim 12 , wherein the first source comprises a form of the input image and the data samples from a first data acquisition system.
16 . The system of claim 15 , wherein the second source comprises a form of the input image and the data samples from a second data acquisition system that is different from the first data acquisition system.
17 . The system of claim 12 , wherein each of the first source and the second source is one of a spectroscopy image and a microscopy image.
18 . The system of claim 12 , wherein the processor selects a first number of pairings of the first set of candidate models and the second set of candidate models for training the fusion model, wherein the first number of pairings have shortest model distance.
19 . The system of claim 12 , wherein the first set of candidate models are models trained from the first source using different model structures and hyper-parameters.
20 . The system of claim 12 , wherein each of the first source and the second source is one of Transmission Electron Microscopy (TEM) image and High Angle Annular Dark Field (HAADF) image.