Data pipeline monitoring
A method and system for data pipeline monitoring receives an event data object and a current status data object from one or more subsystems of a pipeline. The system analyzes the event data object and the current status data object to determine a first and second validation value. The system, in response to determining that either the first or second validation value is not valid, sends a notification.
1. A method providing an improvement in monitoring pipelined data transformation processes in distributed computer systems, the method comprising:
receiving an event data object that indicates a historical status of one or more subsystems of a pipeline;
receiving a current status data object that indicates a current status of one or more subsystems of a pipeline;
analyzing the event data object to determine a first validation value of the event data object by applying a first validation criteria to the event data object;
analyzing the current status data object to determine a second validation value of the current status data object by applying a second validation criteria to the current status data object;
each of the first validation criteria and the second validation criteria comprising any of a programming instruction, script, query, configuration file, or business logic that validates the event data object or the current status data object to determine values that represents characteristics of a health state of the pipeline;
based on determining that the first validation value is not valid, causing to be displayed in a graphical user interface the first validation value;
based on determining that the second validation value is not valid, causing to be displayed in the graphical user interface the second validation value;
wherein the method is performed using one or more processors.
2. The method of claim 1 , further comprising:
causing to be displayed in the graphical user interface a summary card for the one or more subsystems that summarizes the first validation value and the second validation value.
3. The method of claim 2 , wherein the summary card for the one or more subsystems further comprises a plurality of numerical values, wherein each numerical value indicates a number of validation values associated with a particular status type for the one or more subsystems.
4. The method of claim 2 , further comprising:
causing to be displayed in the summary card a graph of one or more prior validation results for the one or more subsystems.
5. The method of claim 4 , wherein the graph comprises a line graph.
6. The method of claim 4 , wherein the graph comprises a histogram.
7. The method of claim 1 , wherein a source of the event data object is an event log.
8. The method of claim 1 , wherein the current status data object comprises information regarding one or more of: disk space, memory usage, and central processing unit (CPU) usage.
9. A system providing an improvement in monitoring pipelined data transformation processes in distributed computer systems, the system comprising:
one or more processors;
one or more non-transitory computer-readable storage media storing sequences of instructions which, when executed by the one or more processors, cause the one or more processors to perform:
receiving an event data object that indicates a historical status of one or more subsystems of a pipeline;
receiving a current status data object that indicates a current status of one or more subsystems of a pipeline;
analyzing the event data object to determine a first validation value of the event data object by applying a first validation criteria to the event data object;
analyzing the current status data object to determine a second validation value of the current status data object by applying a second validation criteria to the current status data object;
each of the first validation criteria and the second validation criteria comprising any of a programming instruction, script, query, configuration file, or business logic that validates the event data object or the current status data object to determine values that represents characteristics of a health state of the pipeline;
in response to determining that the first validation value is not valid, causing to be displayed in a graphical user interface the first validation value; and
in response to determining that the second validation value is not valid, causing to be displayed in the graphical user interface the second validation value.
10. The system of claim 9 , further comprising sequences of instructions which, when executed by the one or more processors, cause the one or more processors to perform:
causing to be displayed in the graphical user interface a summary card for the one or more subsystems that summarizes the first validation value and the second validation value.
11. The system of claim 10 , wherein the summary card for the one or more subsystems further comprises a plurality of numerical values, wherein each numerical value indicates a number of validation values associated with a particular status type for the one or more subsystems.
12. The system of claim 10 , further comprising sequences of instructions which, when executed by the one or more processors, cause the one or more processors to perform:
causing to be displayed in the summary card a graph of one or more prior validation results for the one or more subsystems.
13. The system of claim 12 , wherein the graph comprises a line graph.
14. The system of claim 12 , wherein the graph comprises a histogram.
15. The system of claim 9 , wherein a source of the event data object is an event log.
16. The system of claim 9 , wherein the current status data object comprises information regarding one or more of: disk space, memory usage, and central processing unit (CPU) usage.
17. One or more non-transitory computer-readable media storing one or more sequences of instructions, providing an improvement in monitoring pipelined data transformation processes in distributed computer systems and which, when executed by one or more processors, cause:
receiving an event data object that indicates a historical status of one or more subsystems of a pipeline;
receiving a current status data object that indicates a current status of one or more subsystems of a pipeline;
analyzing the event data object to determine a first validation value of the event data object by applying a first validation criteria to the event data object;
analyzing the current status data object to determine a second validation value of the current status data object by applying a second validation criteria to the current status data object;
each of the first validation criteria and the second validation criteria comprising any of a programming instruction, script, query, configuration file, or business logic that validates the event data object or the current status data object to determine values that represents characteristics of a health state of the pipeline;
in response to determining that the first validation value is not valid, causing to be displayed in a graphical user interface the first validation value; and
in response to determining that the second validation value is not valid, causing to be displayed in the graphical user interface the second validation value.
18. The one or more non-transitory computer-readable media of claim 17 , further comprising instructions, which, when executed by one or more processors, cause:
causing to be displayed in the graphical user interface a summary card for the one or more subsystems that summarizes the first validation value and the second validation value.
19. The one or more non-transitory computer-readable media of claim 18 , wherein the summary card for the one or more subsystems further comprises a plurality of numerical values, wherein each numerical value indicates a number of validation values associated with a particular status type for the one or more subsystems.
20. The one or more non-transitory computer-readable media of claim 18 , further comprising instructions, which, when executed by one or more processors, cause:
causing to be displayed in the summary card a graph of one or more prior validation results for the one or more subsystems.