Machine learning anomaly detection mechanism
Techniques and structures to facilitate anomaly detection within a networking system, including receiving a plurality of performance metric messages at a database system, extracting a plurality of anomaly detection messages included in the performance metric messages, storing the plurality of anomaly detection messages in an in-memory database and executing a machine learning model to process the plurality of anomaly detection messages in the in-memory database to detect whether anomalous usage of the networking system has been detected.
1. A method, comprising:
obtaining a first plurality of performance metric messages at a database system;
extracting a first plurality of anomaly detection data from the first plurality of performance metric messages;
obtaining a second plurality of performance metric messages at the database system;
extracting a second plurality of anomaly detection data from the second plurality of performance metric messages;
performing maintenance of a monitoring component;
storing the first plurality of anomaly detection data and the second plurality of anomaly detection data by a queueing system in memory separate from a first execution engine and a second execution engine;
distributing, by a load balancer, the first plurality of anomaly detection data to the first execution engine from the memory;
distributing, by the load balancer, the second plurality of anomaly detection data to the second execution engine from the memory;
determining, by the first execution engine, whether one or more data points in the first plurality of anomaly detection data is anomalous by applying a first machine learning model to the first plurality of anomaly detection data;
determining, by the second execution engine, whether one or more data points in the second plurality of anomaly detection data is anomalous by applying a second machine learning model to the second plurality of anomaly detection data, wherein the determining of the one or more data points in the first plurality of anomaly detection data and the second plurality of anomaly detection data is performed while the monitoring module undergoes maintenance without the loss of the first plurality of anomaly detection data and the second plurality of anomaly detection data at the first and second execution engines; and
generating, by a monitoring component, one or more alerts according to a result of the determining whether one or more datapoints in the first and second plurality of anomaly detection data are anomalous, the monitoring component operating independently from the first and second execution engines.
2. The method of claim 1 , further comprising:
storing the first plurality of anomaly detection data and second plurality of anomaly detection data.
3. The method of claim 1 , further comprising:
generating a first request to process the first plurality of anomaly detection data;
transmitting the first request to the first execution engine;
generating a second request to process the second plurality of anomaly detection data; and
transmitting the second request to the second execution engine.
4. The method of claim 1 , wherein the first plurality of anomaly detection data and the second plurality of anomaly detection data are processed in parallel by the first execution engine and second execution engine.
5. The method of claim 1 , further comprising:
generating an incident alert upon detection of anomalous usage within the first plurality of anomaly detection data or second plurality of anomaly detection data.
6. The method of claim 1 , wherein a load balancer is configured to perform distribution of a plurality of sets of anomaly detection data.
7. The method of claim 1 , further comprising:
storing the first plurality of anomaly detection data and the second plurality of anomaly detection data by a queueing system.
8. The method of claim 1 , the first plurality of anomaly detection data and the second plurality of anomaly detection data comprising one or more of: request latency metrics or resource utilization percentage metrics.
9. The method of claim 1 , the first plurality of anomaly detection data and second plurality of anomaly detection data comprising metrics pertaining to requests received from a plurality of client devices, the client devices being associated with a plurality of tenants of a multi-tenant database system.
10. The method of claim 1 , the first plurality of anomaly detection data being associated with a first Point of Development (POD) of the database system and the second plurality of anomaly detection data being associated with a second POD of the database system.
11. A system, comprising:
a database system implemented using a server system including at least one processor, the database system configurable to cause:
obtaining a first plurality of performance metric messages at a database system;
extracting a first plurality of anomaly detection data from the first plurality of performance metric messages;
obtaining a second plurality of performance metric messages at the database system;
extracting a second plurality of anomaly detection data from the second plurality of performance metric messages;
performing maintenance of a monitoring component;
storing the first plurality of anomaly detection data and the second plurality of anomaly detection data by a queueing system in memory separate from a first execution engine and a second execution engine;
distributing, by a load balancer, the first plurality of anomaly detection data to the first execution engine from the memory;
distributing, by the load balancer, the second plurality of anomaly detection data to the second execution engine from the memory;
determining, by the first execution engine, whether one or more data points in the first plurality of anomaly detection data is anomalous by applying a first machine learning model to the first plurality of anomaly detection data;
determining, by the second execution engine, whether one or more data points in the second plurality of anomaly detection data is anomalous by applying a second machine learning model to the second plurality of anomaly detection data, wherein the determining of the one or more data points in the first plurality of anomaly detection data and the second plurality of anomaly detection data is performed while the monitoring module undergoes maintenance without the loss of the first plurality of anomaly detection data and the second plurality of anomaly detection data at the first and second execution engines; and
generating, by a monitoring component, one or more alerts according to a result of the determining whether one or more datapoints in the first and second plurality of anomaly detection data are anomalous, the monitoring component operating independently from the first and second execution engines.
12. The system of claim 11 , the database system further configurable to cause:
storing the first plurality of anomaly detection data and second plurality of anomaly detection data.
13. The system of claim 11 , the database system further configurable to cause:
generating a first request to process the first plurality of anomaly detection data;
transmitting the first request to the first execution engine;
generating a second request to process the second plurality of anomaly detection data; and
transmitting the second request to the second execution engine.
14. The system of claim 11 , wherein the first plurality of anomaly detection data and the second plurality of anomaly detection data are processed in parallel by the first execution engine and second execution engine.
15. The system of claim 11 , the first plurality of anomaly detection data and the second plurality of anomaly detection data comprising one or more of: request latency metrics or resource utilization percentage metrics.
16. A non-transitory computer-readable medium comprising computer-readable instructions stored thereon that, when executed by one or more processors, are configurable to cause the one or more processors to:
obtain a first plurality of performance metric messages at a database system;
extract a first plurality of anomaly detection data from the first plurality of performance metric messages;
obtain a second plurality of performance metric messages at the database system;
extract a second plurality of anomaly detection data from the second plurality of performance metric messages;
perform maintenance of a monitoring component;
store the first plurality of anomaly detection data and the second plurality of anomaly detection data by a queueing system in memory separate from a first execution engine and a second execution engine;
distribute, by a load balancer, the first plurality of anomaly detection data to the first execution engine from the memory;
distribute, by the load balancer, the second plurality of anomaly detection data to the second execution engine from the memory;
determine, by the first execution engine, whether one or more data points in the first plurality of anomaly detection data is anomalous by applying a first machine learning model to the first plurality of anomaly detection data;
determine, by the second execution engine, whether one or more data points in the second plurality of anomaly detection data is anomalous by applying a second machine learning model to the second plurality of anomaly detection data, wherein the determining of the one or more data points in the first plurality of anomaly detection data and the second plurality of anomaly detection data is performed while the monitoring module undergoes maintenance without the loss of the first plurality of anomaly detection data and the second plurality of anomaly detection data at the first and second execution engines; and
generate, by a monitoring component, one or more alerts according to a result of the determining whether one or more datapoints in the first and second plurality of anomaly detection data are anomalous, the monitoring component operating independently from the first and second execution engines.
17. The non-transitory computer-readable medium of claim 16 , storing the first plurality of anomaly detection data and second plurality of anomaly detection data.
18. The non-transitory computer-readable medium of claim 17 , further comprising stored thereon instructions that, when executed by one or more processors, are configurable to further cause the one or more processors to:
generate a first request to process the first plurality of anomaly detection data;
transmit the first request to the first execution engine;
generating a second request to process the second plurality of anomaly detection data; and
transmitting the second request to the second execution engine.
19. The non-transitory computer-readable medium of claim 16 , wherein the first plurality of anomaly detection data and the second plurality of anomaly detection data are processed in parallel by the first execution engine and second execution engine.