Method and device for determining data anomaly
Computer-implemented methods, non-transitory, computer-readable media, and computer-implemented systems for determination of anomalous data are provided. In a computer-implemented method, a plurality of data packets is received within a predetermined time period, the plurality of data packets comprising a data structure. A historical distribution of historical data including the data structure as the data packets is determined. The plurality of data packets is compared to the historical distribution to generate a comparison result. If it is determined that data anomaly exists in the plurality of data packets according to the comparison result, an alert indicating the data anomaly is generated.
1. A computer-implemented method for determination of anomalous data, the computer-implemented method comprising:
processing, by one or more processors, data packets comprising a data structure to determine a historical distribution of historical data based on the data structure and a current distribution corresponding to a data distribution state of the data packets;
determining, by the one or more processors, a first distribution state parameter of a randomly selected data packet in the current distribution;
determining, by the one or more processors, a second distribution state parameter of the randomly selected data packet in the historical distribution;
comparing, by the one or more processors, the current distribution of the data packets with the historical distribution by determining a difference value between the first distribution state parameter and the second distribution state parameter to generate a comparison result;
determining, by the one or more processors, that the difference value exceeds a predetermined difference threshold;
in response to determining that the difference value exceeds the predetermined difference threshold determining, by the one or more processors, that a data anomaly exists in the data packets; and
in response to determining that the data anomaly exists in the data packets, generating, by the one or more processors, an alert indicating the data anomaly.
2. The computer-implemented method of claim 1 , wherein comparing the data packets with the historical distribution comprises:
generating a plurality of distribution state parameters by substituting the data packets into the historical distribution; and
comparing the plurality of distribution state parameters with a predetermined threshold related to a distribution state to determine a number of data packets having a distribution state parameter exceeding the predetermined threshold.
3. The computer-implemented method of claim 1 , wherein comparing the data packets with the historical distribution comprises:
determining a data distribution state of the data packets as a current distribution; and
comparing the current distribution with the historical distribution.
4. The computer-implemented method of claim 3 , wherein comparing the current distribution with the historical distribution comprises:
determining a distribution center of the current distribution;
determining a distribution center of the historical distribution;
determining an offset between the distribution center of the current distribution and the distribution center of the historical distribution;
determining that the offset between the distribution center of the current distribution and the distribution center of the historical distribution exceeds a predetermined offset threshold; and
in response to determining that the offset exceeds the predetermined offset threshold, determining that the data anomaly exists in the data packets.
5. The computer-implemented method of claim 3 , wherein comparing the current distribution with the historical distribution comprises:
determining a first distribution state parameter of a randomly selected data packet in the current distribution;
determining a second distribution state parameter of the randomly selected data packet in the historical distribution;
determining a difference value between the first distribution state parameter and the second distribution state parameter;
determining that the difference value exceeds a predetermined difference threshold; and
in response to determining that the difference value exceeds the predetermined difference threshold, determining that the data anomaly exists in the data packets.
6. The computer-implemented method of claim 1 , wherein the historical distribution comprises a historical probability distribution determined by processing the historical data with a hybrid Gaussian model.
7. The computer-implemented method of claim 1 , wherein the historical distribution comprises a historical clustering distribution determined by processing the historical data with a clustering algorithm.
8. A non-transitory, computer-readable medium storing one or more instructions executable by a computer system to perform one or more operations for determination of anomalous data, comprising:
processing data packets comprising a data structure to determine a historical distribution of historical data based on the data structure and a current distribution corresponding to a data distribution state of the data packets;
determining a first distribution state parameter of a randomly selected data packet in the current distribution;
determining a second distribution state parameter of the randomly selected data packet in the historical distribution;
comparing the current distribution of the data packets with the historical distribution by determining a difference value between the first distribution state parameter and the second distribution state parameter to generate a comparison result;
determining that the difference value exceeds a predetermined difference threshold;
in response to determining that the difference value exceeds the predetermined difference threshold determining that a data anomaly exists in the data packets; and
in response to determining that the data anomaly exists in the data packets, generating an alert indicating the data anomaly.
9. The non-transitory, computer-readable medium of claim 8 , wherein comparing the data packets with the historical distribution comprises:
generating a plurality of distribution state parameters by substituting the data packets into the historical distribution; and
comparing the plurality of distribution state parameters with a predetermined threshold related to a distribution state to determine a number of data packets having a distribution state parameter exceeding the predetermined threshold.
10. The non-transitory, computer-readable medium of claim 8 , wherein comparing the data packets with the historical distribution comprises:
determining a data distribution state of the data packets as a current distribution; and
comparing the current distribution with the historical distribution.
11. The non-transitory, computer-readable medium of claim 10 , wherein comparing the current distribution with the historical distribution comprises:
determining a distribution center of the current distribution;
determining a distribution center of the historical distribution;
determining an offset between the distribution center of the current distribution and the distribution center of the historical distribution;
determining that the offset between the distribution center of the current distribution and the distribution center of the historical distribution exceeds a predetermined offset threshold; and
in response to determining that the offset exceeds the predetermined offset threshold, determining that the data anomaly exists in the data packets.
12. The non-transitory, computer-readable medium of claim 10 , wherein comparing the current distribution with the historical distribution comprises:
determining a first distribution state parameter of a randomly selected data packet in the current distribution;
determining a second distribution state parameter of the randomly selected data packet in the historical distribution;
determining a difference value between the first distribution state parameter and the second distribution state parameter;
determining that the difference value exceeds a predetermined difference threshold; and
in response to determining that the difference value exceeds the predetermined difference threshold, determining that the data anomaly exists in the data packets.
13. The non-transitory, computer-readable medium of claim 8 , wherein the historical distribution comprises a historical probability distribution determined by processing the historical data with a hybrid Gaussian model.
14. The non-transitory, computer-readable medium of claim 8 , wherein the historical distribution comprises a historical clustering distribution determined by processing the historical data with a clustering algorithm.
15. A computer-implemented system for determination of anomalous data, comprising:
one or more computers; and
one or more computer memory devices interoperably coupled with the one or more computers and having tangible, non-transitory, machine-readable media storing one or more instructions that, when executed by the one or more computers, perform one or more operations, comprising:
processing data packets comprising a data structure to determine a historical distribution of historical data based on the data structure and a current distribution corresponding to a data distribution state of the data packets;
determining a first distribution state parameter of a randomly selected data packet in the current distribution;
determining a second distribution state parameter of the randomly selected data packet in the historical distribution;
comparing the current distribution of the data packets with the historical distribution by determining a difference value between the first distribution state parameter and the second distribution state parameter to generate a comparison result;
determining that the difference value exceeds a predetermined difference threshold;
in response to determining that the difference value exceeds the predetermined difference threshold determining that a data anomaly exists in the data packets; and
in response to determining that the data anomaly exists in the data packets, generating an alert indicating the data anomaly.
16. The computer-implemented system of claim 15 , wherein comparing the data packets with the historical distribution comprises:
generating a plurality of distribution state parameters by substituting the data packets into the historical distribution; and
comparing the plurality of distribution state parameters with a predetermined threshold related to a distribution state to determine a number of data packets having a distribution state parameter exceeding the predetermined threshold.
17. The computer-implemented system of claim 15 , wherein comparing the data packets with the historical distribution comprises:
determining a data distribution state of the data packets as a current distribution; and
comparing the current distribution with the historical distribution.
18. The computer-implemented system of claim 17 , wherein comparing the current distribution with the historical distribution comprises:
determining a distribution center of the current distribution;
determining a distribution center of the historical distribution;
determining an offset between the distribution center of the current distribution and the distribution center of the historical distribution;
determining that the offset between the distribution center of the current distribution and the distribution center of the historical distribution exceeds a predetermined offset threshold; and
in response to determining that the offset exceeds the predetermined offset threshold, determining that the data anomaly exists in the data packets.
19. The computer-implemented system of claim 17 , wherein comparing the current distribution with the historical distribution comprises:
determining a first distribution state parameter of a randomly selected data packet in the current distribution;
determining a second distribution state parameter of the randomly selected data packet in the historical distribution;
determining a difference value between the first distribution state parameter and the second distribution state parameter;
determining that the difference value exceeds a predetermined difference threshold; and
in response to determining that the difference value exceeds the predetermined difference threshold, determining that the data anomaly exists in the data packets.
20. The computer-implemented system of claim 15 , wherein the historical distribution comprises a historical probability distribution determined by processing the historical data with a hybrid Gaussian model or a clustering algorithm.