Method and apparatus for detecting and explaining anomalies
Embodiments described herein relate to a method and apparatus for detecting and explaining anomalies in data obtained from an environment using an encoder-decoder machine learning model. A state of the environment is represented by a plurality of features, and the machine learning model is trained with a first set of data samples. Each data sample in the first set of data samples comprises values for each of the plurality of features. The method comprises determining a respective first threshold for each of the plurality of features based on respective maximum reconstruction errors for each feature found during training of the encoder-decoder machine learning model; obtaining an anomalous data sample; determining respective reconstruction errors for each feature in the anomalous data sample using the trained encoder-decoder machine learning model; and determining one or more features in the anomalous data sample that are responsible for the anomalous data sample being anomalous responsive to the reconstruction errors associated with the one or more features being greater than or equal to the respective first thresholds.
1. A method of detecting and explaining anomalies in data obtained from an environment using an encoder-decoder machine learning model, wherein a state of the environment is represented by a plurality of features, and wherein the encoder-decoder machine learning model is trained with a first set of data samples, and each data sample in the first set of data samples comprises values for each of the plurality of features, the method comprising:
determining a respective first threshold for each of the plurality of features based on respective maximum reconstruction errors for each feature found during training of the encoder-decoder machine learning model;
obtaining an anomalous data sample;
determining respective reconstruction errors for each feature in the anomalous data sample using the trained encoder-decoder machine learning model; and
determining one or more features in the anomalous data sample that are responsible for the anomalous data sample being anomalous responsive to the reconstruction errors associated with the one or more features being greater than or equal to the respective first thresholds.
2. The method of claim 1 , wherein the first set of data samples is assumed to contain no anomalies.
3. The method of claim 1 , further comprising training the encoder-decoder machine learning model with the first set of data samples.
4. The method of claim 1 , wherein the respective first threshold for each of the plurality of features is equal to the maximum reconstruction error for the respective feature.
5. The method of claim 1 , wherein the respective first threshold for each of the plurality of features is greater than 95th percentile of the maximum reconstruction error for the respective feature.
6. The method of claim 1 , wherein the respective first threshold for each of the plurality of features is set to the Nth percentile of the maximum reconstruction error for the respective feature responsive to the first set of data samples comprising (100−N) % anomalies, where N is an numerical value.
7. The method of claim 1 , further comprising training a sensitivity analysis model, wherein the sensitivity analysis model comprises a mapping function, by:
a. adding respective known noise signals to one or more samples in the first set of data samples to generate a noisy set of data samples;
b. for each sample in the noisy set of data samples:
i. inputting the sample into the trained encoder-decoder machine learning model to determine reconstruction errors for each feature in the sample; and
ii. determining which one or more features in the sample comprise noise signals by comparing the respective reconstruction errors for each feature to the respective first threshold; and
c. determining the mapping function such that the mapping function maps the reconstruction errors for features comprising a noise signal to the respective known noise signals added in step a.
8. The method of claim 7 , wherein the step of determining the mapping function comprises using multiple linear regression to determine a correlation between the reconstruction errors for features comprising a noise signal and two variables, wherein the two variables comprise the known noise signals and a number of noisy features in each data sample in the noisy set of data samples.
9. The method of claim 8 , further comprising:
for each of the one or more features that are responsible for the anomalous data sample being anomalous:
determining a noise level associated with the feature by using the mapping function to map a reconstruction error for the feature to a noise level.
10. The method of claim 1 , further comprising:
determining a second threshold based on a maximum average sample reconstruction error for the first set of data samples found during the training step, wherein an average sample reconstruction error for a sample is the average of the reconstruction errors for the features in said sample.
11. The method of claim 10 , wherein the step of obtaining an anomalous data sample comprises:
obtaining a first data sample in a second set of data samples;
inputting the first data sample into the encoder-decoder machine learning model to determine a first average sample reconstruction error for the first data sample;
comparing the first average sample reconstruction error to the second threshold; and
responsive to the first average sample reconstruction error being greater than the second threshold setting the first data sample as the anomalous data sample.
12. The method of claim 11 , further comprising:
responsive to the first average sample reconstruction error being less than or equal to the second threshold setting the first data sample as a normal data sample.
13. The method of claim 11 , further comprising:
calculating a first score value as the difference between the first average sample reconstruction error and the second threshold.
14. The method of claim 11 , wherein
the step of obtaining an anomalous data sample comprises:
determining a reconstruction error for each feature in the first data sample based on an output of the encoder-decoder machine learning model;
comparing each reconstruction error to the respective first thresholds for each feature; and
responsive to at least one of the reconstruction errors being greater than the respective first threshold, setting the first data sample as the anomalous data sample.
15. The method of claim 14 , further comprising:
responsive to all of the reconstruction errors being less than or equal to the respective first thresholds, setting the first data sample as a normal data sample.
16. The method of claim 14 , further comprising:
calculating a second score values for each feature as the difference between the reconstruction errors and the respective first thresholds;
sorting the one or more features in the anomalous data sample that are responsible for the anomalous data sample being anomalous based on the second score values associated with each of the one or more features.
17. The method of claim 1 , wherein the first set of data samples and the anomalous sample have been normalized.
18. The method of claim 1 , wherein the first set of data samples is calculated based on a rate of change of an original set of data samples.
19. An apparatus for detecting and explaining anomalies in data obtained from an environment using an encoder-decoder machine learning model, wherein a state of the environment is represented by a plurality of features, and wherein the machine learning model is trained with a first set of data samples, and each data sample in the first set of data samples comprises values for each of the plurality of features, the apparatus comprising processing circuitry configured to cause the apparatus to perform the method of claim 1 .
20. A non-transitory computer readable storage medium storing a computer program comprising instructions which, when executed on at least one processor, cause the at least one processor to carry out the method of claim 1 .