Systems and methods for real-time data processing, monitoring, and alerting
Large quantities of stored data from a plurality of sources, as well as large quantities of incoming real-time data that may also be coming from a plurality of sources, can be compared by pre-modeling, indexing, and aggregating data prior to conducting the comparison. A system can include a monitoring and alerting agent, an interactive visualization agent, and a machine learning agent to provide up-to-date transaction-level, store-level, and customer-level alerting and attribution.
1. A system for monitoring large quantities of real-time data to produce a customizable visualization indicative of detected anomalous transactions, the system comprising:
a front-end application configured for defining a filter;
a database containing a plurality of static data;
a plurality of sources of real-time data;
an ingestion module configured to apply the filter to the static data to generate filtered static data;
an integration module configured to integrate data from the plurality of sources of real-time data and enrich the integrated real-time data such that the integrated real-time data has a common format; and
an analytic enrichment engine configured to combine the filtered static data with the integrated real-time data to generate the custom visualization of the real-time data at the front-end application, such that a user at the front-end application can generate and monitor metrics relating to individual data.
2. The system of claim 1 , further comprising a monitoring agent configured to generate an alert when the integration module detects a variation between the data from the plurality of real-time data sources and an expected range based on the filtered static data.
3. The system of claim 2 , wherein the expected range is generated by the monitoring agent based upon the filtered static data corresponding to non-anomalous transactions.
4. The system of claim 2 , wherein the monitoring agent is configured to detect misuse of gift cards.
5. The system of claim 2 , further comprising an analytical engine configured to generate the alert.
6. The system of claim 5 , wherein the analytical engine produces alerts based upon a behavior analytics model.
7. The system of claim 6 , wherein the analytical engine is configured to apply the behavior analytics model to all of the data from the plurality of real-time data sources simultaneously.
8. The system of claim 6 , wherein the behavior analytics model is a gravitational model.
9. The system of claim 2 , wherein the alert is based upon an indication of an anomalous type of transaction by a particular customer.
10. The system of claim 2 , wherein the alert is based upon an indication of an anomalous type of transaction relating to a particular item.
11. A method for monitoring a plurality of real-time data sources simultaneously and generating a custom visualization indicative of detected anomalies, the method comprising:
defining a filter at a front-end application;
storing a plurality of static data at a database;
ingesting the static data at an ingestion module configured to apply the filter to the static data;
integrating data from a plurality of sources of real-time data at an integration module;
enriching the integrated real-time data from the plurality of sources of real-time data such that the integrated real-time data all has a common format; and
combining the filtered static data and the enriched real-time data to generate the custom visualization of the real-time data at the front-end application, such that a user at the front-end application can generate and monitor metrics relating to individual data.
12. The method of claim 11 , further comprising detecting a variation between the data from the plurality of real-time data sources and an expected range based on the filtered static data.
13. The method of claim 12 , further comprising generating an alert based upon the detected variation.
14. The method of claim 12 , wherein monitoring metrics relating to individual data comprises determining the expected range based upon the filtered static data corresponding to non-anomalous transactions.
15. The method of claim 12 , wherein monitoring metrics relating to individual data comprises detecting misuse of gift cards.
16. The method of claim 11 , further comprising producing an alert at an analytics module based upon a behavior analytics model.
17. The method of claim 16 , wherein producing the alert based upon the behavior analytics model comprises applying the behavior analytics model to all of the data from the plurality of real-time data sources simultaneously.
18. The method of claim 16 , wherein the behavior analytics model is a gravitational model.
19. The system of claim 13 , wherein the alert is based upon an indication of a type of anomalous transaction by a particular customer.
20. The system of claim 13 , wherein the alert is based upon an indication of a type of anomalous transaction relating to a particular item.