Data pipeline monitoring
View Patent ↗A method and system for data pipeline monitoring receives an event data object and a current status data object from one or more subsystems of a pipeline. The system analyzes the event data object and the current status data object to determine a first and second validation value. The system, in response to determining that either the first or second validation value is not valid, sends a notification.
1. A method comprising:
receiving an event data object that indicates a historical status of one or more subsystems of a pipeline;
receiving a current status data object that indicates a current status of one or more subsystems of a pipeline;
analyzing the event data object to determine a first validation value of the event data object by applying a first validation criteria to the event data object;
analyzing the current status data object to determine a second validation value of the current status data object by applying a second validation criteria to the current status data object;
in response to determining that the first validation value is not valid, sending a first notification regarding the event data object;
in response to determining that the second validation value is not valid, sending a second notification regarding the current status data object; and
wherein the method is performed using one or more processors.
2. The method of claim 1 , wherein the first validation criteria comprises criteria specified in a script.
3. The method of claim 1 , wherein a source of the event data object is an event log.
4. The method of claim 1 , wherein the current status data object comprises information regarding one or more of: disk space, memory usage, and central processing unit (CPU) usage.
5. The method of claim 1 , wherein:
receiving an event data object that indicates a historical status of one or more subsystems of a pipeline is scheduled based on a first preconfigured schedule; and
receiving a current status data object that indicates a current status of one or more subsystems of a pipeline is scheduled based on the first preconfigured schedule.
6. The method of claim 1 , further comprising:
customizing a format of the first notification based on the first validation value; and
customizing a format of the second notification based on the second validation value.
7. The method of claim 1 , further comprising:
customizing a first recipient of the first notification based on the first validation value; and
customizing a second recipient of the second notification based on the second validation value.
8. The method of claim 1 , further comprising displaying a user interface based on the event data object, the current status data object, the first validation value, the second validation value, the first notification and the second notification.
9. The method of claim 8 , wherein the user interface comprises a summary card summarizing the first validation value and the second validation value for a particular subsystem of the one or more subsystems.
10. A system, comprising:
a plurality of computers;
one or more non-transitory data storage media in the plurality of computers storing logic comprising one or more sequences of instructions which when executed cause the computers to perform:
receiving an event data object that indicates a historical status of one or more subsystems of a pipeline;
receiving a current status data object that indicates a current status of one or more subsystems of a pipeline;
analyzing the event data object to determine a first validation value of the event data object by applying a first validation criteria to the event data object;
analyzing the current status data object to determine a second validation value of the current status data object by applying a second validation value to the current status data object;
in response to determining that the first validation value is not valid, sending a first notification regarding the event data object; and
in response to determining that the second validation value is not valid, sending a second notification regarding the current status data object.
11. The system of claim 10 , wherein the first validation criteria comprises criteria specified in a script.
12. The system of claim 10 , wherein a source of the event data object is an event log.
13. The system of claim 10 , wherein the current status data object comprises information regarding one or more of: disk space, memory usage, and central processing unit (CPU) usage.
14. The system of claim 10 , wherein:
the instructions for receiving an event data object that indicates a historical status of one or more subsystems of a pipeline is scheduled based on a first preconfigured schedule; and
the instructions receiving a current status data object that indicates a current status of one or more subsystems of a pipeline is scheduled based on the first preconfigured schedule.
15. The system of claim 10 , further comprising instructions for:
customizing a format of the first notification based on the first validation value; and
customizing a format of the second notification based on the second validation value.
16. The system of claim 10 , further comprising instructions for:
customizing a first recipient of the first notification based on the first validation value; and
customizing a second recipient of the second notification based on the second validation value.
17. The system of claim 10 , further comprising instructions for displaying a user interface based on the event data object, the current status data object, the first validation value, the second validation value, the first notification and the second notification.
18. The system of claim 17 , wherein the user interface comprises a summary card summarizing the first validation value and the second validation value for a particular subsystem of the one or more subsystems.