Optimizing detection of abnormal data points in time series data
In an approach for optimizing abnormal point detection, a processor receives a set of data, wherein the set of data is partially labeled time series data; determines a data block size for the set of data; splits the set of data into data blocks based on the data block size; computes trait measurements for traits for each data block; assigns a tag to each data block, wherein the tag is selected from the group consisting of a normal tag, an abnormality tag, and an unknown tag; uses the respective data blocks with either the normal tag or the abnormality tag as training data; updates the training data with artificial abnormalities; trains a detection model with the updated training data; and utilizes the trained detection model to predict whether the respective data blocks with the unknown tag have an abnormality or no abnormality.
1 . A computer-implemented method comprising:
receiving, by one or more processors, a set of data, wherein the set of data is partially labeled time series data;
determining, by the one or more processors, a data block size for the set of data;
splitting, by the one or more processors, the set of data into data blocks based on the data block size;
computing, by the one or more processors, trait measurements for traits for each data block;
assigning, by the one or more processors, a tag to each data block, wherein the tag is selected from the group consisting of a normal tag, an abnormality tag, and an unknown tag;
using, by the one or more processors, the respective data blocks with either the normal tag or the abnormality tag as training data;
updating, by the one or more processors, the training data with artificial abnormalities;
training, by the one or more processors, a detection model with the updated training data; and
utilizing, by the one or more processors, the trained detection model to predict whether the respective data blocks with the unknown tag have an abnormality or no abnormality.
2 . The computer-implemented method of claim 1 , wherein determining the data block size is done using spectral analysis methods to observe patterns and identify peaks on a power spectral density plot to identify cutoff points dictating the data block size.
3 . The computer-implemented method of claim 1 , further comprising:
storing, by the one or more processors, the trait measurements for each data block as records in a table with each row representing one data block of the data blocks and each column representing one trait of the traits.
4 . The computer-implemented method of claim 3 , further comprising:
adding, by the one or more processors, an additional column to the table with the assigned tag for each data block.
5 . The computer-implemented method of claim 1 , wherein the tag is assigned based on whether data points within a respective data block included a label or no label, and wherein the respective label indicates an abnormality or no abnormality.
6 . The computer-implemented method of claim 1 , wherein the abnormality tag indicates an abnormality in the respective data block, the normal tag indicates no abnormality in the respective data block, and the unknown tag indicates it is unknown whether there is an abnormality in the respective data block.
7 . The computer-implemented method of claim 1 , wherein updating the training data with artificial abnormalities comprises:
updating, by the one or more processors, one or more of the data blocks with the normal tag to the abnormality tag so the updated training data has a more equal number of respective data blocks with the normal tag and respective data blocks with the abnormality tag.
8 . A computer program product comprising:
one or more computer readable storage media and program instructions stored on the one or more computer readable storage media, the program instructions comprising:
program instructions to receive a set of data, wherein the set of data is partially labeled time series data;
program instructions to determine a data block size for the set of data;
program instructions to split the set of data into data blocks based on the data block size;
program instructions to compute trait measurements for traits for each data block;
program instructions to assign a tag to each data block, wherein the tag is selected from the group consisting of a normal tag, an abnormality tag, and an unknown tag;
program instructions to use the respective data blocks with either the normal tag or the abnormality tag as training data;
program instructions to update the training data with artificial abnormalities;
program instructions to train a detection model with the updated training data; and
program instructions to utilize the trained detection model to predict whether the respective data blocks with the unknown tag have an abnormality or no abnormality.
9 . The computer program product of claim 8 , wherein the program instructions to determine the data block size is done using spectral analysis methods to observe patterns and identify peaks on a power spectral density plot to identify cutoff points dictating the data block size.
10 . The computer program product of claim 8 , further comprising:
program instructions to store the trait measurements for each data block as records in a table with each row representing one data block of the data blocks and each column representing one trait of the traits.
11 . The computer program product of claim 10 , further comprising:
program instructions to add an additional column to the table with the assigned tag for each data block.
12 . The computer program product of claim 8 , wherein the tag is assigned based on whether data points within a respective data block included a label or no label, and wherein the respective label indicates an abnormality or no abnormality.
13 . The computer program product of claim 8 , wherein the abnormality tag indicates an abnormality in the respective data block, the normal tag indicates no abnormality in the respective data block, and the unknown tag indicates it is unknown whether there is an abnormality in the respective data block.
14 . The computer program product of claim 8 , wherein the program instructions to update the training data with artificial abnormalities comprise:
program instructions to update one or more of the data blocks with the normal tag to the abnormality tag so the updated training data has a more equal number of respective data blocks with the normal tag and respective data blocks with the abnormality tag.
15 . A computer system comprising:
one or more computer processors;
one or more computer readable storage media;
program instructions collectively stored on the one or more computer readable storage media for execution by at least one of the one or more computer processors, the stored program instructions comprising:
program instructions to receive a set of data, wherein the set of data is partially labeled time series data;
program instructions to determine a data block size for the set of data;
program instructions to split the set of data into data blocks based on the data block size;
program instructions to compute trait measurements for traits for each data block;
program instructions to assign a tag to each data block, wherein the tag is selected from the group consisting of a normal tag, an abnormality tag, and an unknown tag;
program instructions to use the respective data blocks with either the normal tag or the abnormality tag as training data;
program instructions to update the training data with artificial abnormalities;
program instructions to train a detection model with the updated training data; and
program instructions to utilize the trained detection model to predict whether the respective data blocks with the unknown tag have an abnormality or no abnormality.
16 . The computer system of claim 15 , wherein the program instructions to determine the data block size is done using spectral analysis methods to observe patterns and identify peaks on a power spectral density plot to identify cutoff points dictating the data block size.
17 . The computer system of claim 15 , further comprising:
program instructions to store the trait measurements for each data block as records in a table with each row representing one data block of the data blocks and each column representing one trait of the traits.
18 . The computer system of claim 17 , further comprising:
program instructions to add an additional column to the table with the assigned tag for each data block.
19 . The computer system of claim 15 , wherein the tag is assigned based on whether data points within a respective data block included a label or no label, and wherein the respective label indicates an abnormality or no abnormality.
20 . The computer system of claim 15 , wherein the abnormality tag indicates an abnormality in the respective data block, the normal tag indicates no abnormality in the respective data block, and the unknown tag indicates it is unknown whether there is an abnormality in the respective data block.