Method and an apparatus for evaluating generative machine learning model
The invention relates to a method comprising receiving a set of input samples, said set of input images comprising real images and generated images; extracting a set of feature maps from multiple layers of a pre-trained neural network for both the real images and the generated images; determining statistics for each feature map of the set of feature maps; comparing statistics of the feature maps for the real images to statistics of the feature maps for the generated images by using a distance function to obtain a vector of distances; and averaging the distances of the vector of distances to have a value indicating a diversity of the generated images. The invention also relates to technical equipment for implementing the method.
1. A method, comprising:
receiving a set of input samples, said set of input samples comprising real images and generated images;
extracting a set of feature maps from multiple layers of a pre-trained neural network for both the real images and the generated images;
determining respective statistics for feature maps of the set of feature maps;
comparing respective statistics of the feature maps for the real images with corresponding respective statistics of the feature maps for the generated images, wherein the comparing of the statistics comprises using a distance function to obtain a vector of distances; and
averaging distances of the vector of distances to have a value providing information about a level of diversity of the generated images.
2. The method according to claim 1 , wherein the pre-trained neural network is one of an autoencoder or a ladder autoencoder trained via unsupervised or semi-supervised training.
3. The method according to claim 1 , wherein the statistics are determined per spatial or temporal location in addition to determining them per feature map.
4. The method according to claim 1 , wherein the statistics comprises one or more of the following: a feature map mean, a variance, covariance, cross-correlation between the feature maps or spatial and/or temporal locations.
5. The method according to claim 1 , wherein the averaging of the distances of the vector of distances produces a final distance, wherein an inverse of the final distance is the value for providing information about the level of diversity of the generated images.
6. The method according to claim 1 , further comprising:
determining a value indicating semantic stability of the generated images, wherein the determining of the value indicating semantic stability comprises inputting a random vector to the pre-trained neural network; and
varying the input random vector to determine a change in a semantic space.
7. An apparatus comprising at least one processor, non-transitory memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following:
receive a set of input samples, said set of input samples comprising real images and generated images;
extract a set of feature maps from multiple layers of a pre-trained neural network for both the real images and the generated images;
determine respective statistics for feature maps of the set of feature maps;
compare respective statistics of the feature maps for the real images with corresponding respective statistics of the feature maps for the generated images, wherein the comparing of the statistics comprises using a distance function to obtain a vector of distances; and
average distances of the vector of distances to have a value providing information about a level of diversity of the generated images.
8. The apparatus according to claim 7 , wherein the pre-trained neural network is one of an autoencoder or a ladder autoencoder trained via unsupervised or semi-supervised training.
9. The apparatus according to claim 7 , wherein the statistics are determined per spatial or temporal location in addition to determining them per feature map.
10. The apparatus according to claim 7 , wherein the statistics comprises one or more of the following: a feature map mean, a variance, covariance, cross-correlation between the feature maps or spatial and/or temporal locations.
11. The apparatus according to claim 7 , further comprising computer program code configured to cause the apparatus to produce a final distance, wherein an inverse of the final distance is the value for providing the information about the level of diversity of the generated images.
12. The apparatus according to claim 7 , further comprising computer program code configured to cause the apparatus to determine a value indicating semantic stability of the generated images, wherein the determining of the value indicating semantic stability comprises inputting a random vector to the pre-trained neural network and varying the input random vector to determine a change in a semantic space.
13. A computer program product embodied on a non-transitory computer readable medium, comprising computer program code configured to, when executed on at least one processor, cause an apparatus or a system to:
receive a set of input samples, said set of input samples comprising real images and generated images;
extract a set of feature maps from multiple layers of a pre-trained neural network for both the real images and the generated images;
determine respective statistics for feature maps of the set of feature maps;
compare respective statistics of the feature maps for the real images with corresponding respective statistics of the feature maps for the generated images, wherein the comparing of the statistics comprises using a distance function to obtain a vector of distances; and
average distances of the vector of distances to have a value providing information about a level of diversity of the generated images.
14. An apparatus comprising at least:
circuitry configured to receive a set of input samples, said set of input samples comprising real images and generated images;
circuitry configured to extract a set of feature maps from multiple layers of a pre-trained neural network for both the real images and the generated images;
circuitry configured to determine respective statistics for feature maps of the set of feature maps;
circuitry configured to compare respective statistics of the feature maps for the real images with corresponding respective statistics of the feature maps for the generated images, wherein the comparing of the statistics comprises using a distance function to obtain a vector of distances; and
circuitry configured to average distances of the vector of distances to have a value providing information about a level of diversity of the generated images.
15. The apparatus according to claim 14 , wherein the pre-trained neural network is one of an autoencoder or a ladder autoencoder trained via unsupervised or semi-supervised training.
16. The apparatus according to claim 14 , wherein the statistics are determined per spatial or temporal location in addition to determining them per feature map.
17. The apparatus according to claim 14 , wherein the statistics comprises one or more of the following: a feature map mean, a variance, covariance, cross-correlation between the feature maps or spatial and/or temporal locations.
18. The apparatus according to claim 14 , further comprising circuitry configured to produce a final distance, wherein an inverse of the final distance is the value for providing the information about the level of diversity of the generated images.
19. The apparatus according to claim 14 , further comprising circuitry configured to determine a value indicating semantic stability of the generated images, wherein the determining of the value indicating semantic stability further comprises inputting a random vector to the pre-trained neural network and varying the input random vector to determine a change in a semantic space.