Data pipeline monitoring
A method and system for data pipeline monitoring receives an event data object and a current status data object from one or more subsystems of a pipeline. The system analyzes the event data object and the current status data object to determine a first and second validation value. The system, in response to determining that either the first or second validation value is not valid, sends a notification.
1. A method providing an improvement in monitoring pipelined data transformation processes in distributed computer systems, the method comprising:
receiving an event data object that indicates a historical status of one or more subsystems of a pipeline;
receiving a current status data object that indicates a current status of one or more subsystems of a pipeline;
analyzing the event data object to determine a first validation value of the event data object by applying a first validation criteria to the event data object;
analyzing the current status data object to determine a second validation value of the current status data object by applying a second validation criteria to the current status data object;
displaying the first validation value in a graphical user interface; and
displaying the second validation value in the graphical user interface;
wherein the method is performed using one or more processors.
2. The method of claim 1 , further comprising causing displaying in the graphical user interface a summary card for the one or more subsystems that summarizes the first validation value and the second validation value.
3. The method of claim 2 , the summary card for the one or more subsystems further comprising a plurality of numerical values, each numerical value indicating a number of validation values associated with a particular status type for the one or more subsystems.
4. The method of claim 2 , further comprising causing to be displayed in the summary card a graph of one or more prior validation results for the one or more subsystems.
5. The method of claim 4 , the graph comprising a line graph.
6. The method of claim 4 , the graph comprising a histogram.
7. The method of claim 1 , a source of the event data object being an event log.
8. The method of claim 1 , the current status data object comprising information regarding one or more of: disk space, memory usage, and central processing unit (CPU) usage.
9. A computer system providing an improvement in monitoring pipelined data transformation processes in distributed computer systems, the computer system comprising:
one or more processors;
one or more non-transitory computer-readable storage media storing sequences of instructions which, when executed by the one or more processors, cause the one or more processors to perform:
receiving an event data object that indicates a historical status of one or more subsystems of a pipeline;
receiving a current status data object that indicates a current status of one or more subsystems of a pipeline;
analyzing the event data object to determine a first validation value of the event data object by applying a first validation criteria to the event data object;
analyzing the current status data object to determine a second validation value of the current status data object by applying a second validation criteria to the current status data object;
displaying the first validation value in a graphical user interface; and
displaying the second validation value in the graphical user interface.
10. The system of claim 9 , further comprising sequences of instructions which, when executed by the one or more processors, cause the one or more processors to perform causing to be displayed in the graphical user interface a summary card for the one or more subsystems that summarizes the first validation value and the second validation value.
11. The system of claim 10 , the summary card for the one or more subsystems comprising a plurality of numerical values, each numerical value indicating a number of validation values associated with a particular status type for the one or more subsystems.
12. The system of claim 10 , further comprising sequences of instructions which, when executed by the one or more processors, cause the one or more processors to perform causing to be displayed in the summary card a graph of one or more prior validation results for the one or more subsystems.
13. The system of claim 12 , the graph comprising a line graph.
14. The system of claim 12 , the graph comprising a histogram.
15. The system of claim 9 , a source of the event data object being an event log.
16. The system of claim 9 , the current status data object comprising information regarding one or more of: disk space, memory usage, and central processing unit (CPU) usage.
17. One or more non-transitory computer-readable media storing one or more sequences of instructions, providing an improvement in monitoring pipelined data transformation processes in distributed computer systems and which, when executed by one or more processors, cause:
receiving an event data object that indicates a historical status of one or more subsystems of a pipeline;
receiving a current status data object that indicates a current status of one or more subsystems of a pipeline;
analyzing the event data object to determine a first validation value of the event data object by applying a first validation criteria to the event data object;
analyzing the current status data object to determine a second validation value of the current status data object by applying a second validation criteria to the current status data object;
displaying the first validation value in a graphical user interface; and
displaying the second validation value in the graphical user interface.
18. The one or more non-transitory computer-readable media of claim 17 , further comprising instructions, which, when executed by one or more processors, cause displaying in the graphical user interface a summary card for the one or more subsystems that summarizes the first validation value and the second validation value.
19. The one or more non-transitory computer-readable media of claim 18 , the summary card for the one or more subsystems comprising a plurality of numerical values, each numerical value indicating a number of validation values associated with a particular status type for the one or more subsystems.
20. The one or more non-transitory computer-readable media of claim 18 , further comprising instructions, which, when executed by one or more processors, cause displaying in the summary card a graph of one or more prior validation results for the one or more subsystems.