Monitoring operation of a data pipeline to obtain representations of operation quality
Methods and systems for managing operation of a data pipeline are disclosed. To manage the operation, a system may include one or more data sources, a data manager, and one or more downstream consumers. Interruptions to the operation may impact provision of data processing services by the data pipeline and may cause the data processing services to no longer align with operation quality goals for the data pipeline. To maintain compliance with the operation quality goals, the operation may be monitored over time. Operation data may be obtained for the data pipeline and may be used to determine representations of operation quality of the data pipeline. The representations of operation quality of the data pipeline may be compared to the operation quality goals and actions may be performed to remediate differences between the representations of operation quality of the data pipeline and the operation quality goals.
1. A method of managing a data pipeline, the method comprising:
monitoring the data pipeline based on monitoring criteria to obtain operation data of the data pipeline that is indicative of a performance of the data pipeline, the operation data being based at least on operation data of an application programming interface (API) that initiated transfer of data through the data pipeline in response to a data retrieval requests obtained by the API, the operation data of the API being indicative of a performance of the API;
obtaining representations of operation quality of the data pipeline by comparing the operation data with one or more evaluation metrics, the one or more evaluation metrics comprising at least a predefined uptime duration threshold for the data pipeline;
evaluating the representations of the operation quality of the data pipeline for compliance with operation quality goals for the data pipeline to obtain a degree of alignment with a use of the data pipeline; and
performing an action set based on the degree of alignment with the use of the data pipeline to improve a quality of data processing services provided by the data pipeline.
2. The method of claim 1 , wherein the monitoring criteria specifies at least the following as monitored quantities:
an actual uptime of the data pipeline;
occurrences of misalignments of the data pipeline, the occurrences of the misalignments of the data pipeline being based on the operation data of the API; and
portions of data managed by the data pipeline that are impacted by the occurrences of the misalignments.
3. The method of claim 2 , wherein the monitoring criteria further specifies at least the following as additional monitored quantities:
re-alignments of the misalignments of the data pipeline; and
durations of time between the misalignments of the data pipeline and the re-alignments of the misalignments.
4. The method of claim 3 , wherein the one or more evaluation metrics further comprise:
a threshold number of occurrences for the misalignments of the data pipeline; and
a value ascribed by a downstream consumer to the portions of the data.
5. The method of claim 4 , wherein the one or more evaluation metrics further specify at least the following additional metrics:
a value ascribed by the downstream consumer for the durations of time.
6. The method of claim 2 , wherein the misalignments of the data pipeline occur when the API outputs, in response to the data retrieval request obtained by the API, data different from requested data specified in the data retrieval request.
7. The method of claim 1 , wherein performing the action set comprises:
obtaining a report based on the degree of alignment with the use of the data pipeline;
distributing the report to one or more persons to obtain user feedback; and
selecting one or more actions to be performed based on the user feedback.
8. The method of claim 7 , wherein the one or more actions, when performed, modify operation of the data pipeline to obtain an updated data pipeline.
9. The method of claim 1 , wherein the data pipeline is established in response to the API attempting to retrieve data specified in the data retrieval request from a data source in which the data is stored.
10. The method of claim 1 , wherein the operation data of the API comprises a data retrieval result of the data retrieval request being processed by the API.
11. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations for managing a data pipeline, the operations comprising:
monitoring the data pipeline based on monitoring criteria to obtain operation data of the data pipeline that is indicative of a performance of the data pipeline, the operation data being based at least on operation data of an application programming interface (API) that initiated transfer of data through the data pipeline in response to a data retrieval requests obtained by the API, the operation data of the API being indicative of a performance of the API;
obtaining representations of operation quality of the data pipeline by comparing the operation data with one or more evaluation metrics, the one or more evaluation metrics comprising at least a predefined uptime duration threshold for the data pipeline;
evaluating the representations of the operation quality of the data pipeline for compliance with operation quality goals for the data pipeline to obtain a degree of alignment with a use of the data pipeline; and
performing an action set based on the degree of alignment with the use of the data pipeline to improve a quality of data processing services provided by the data pipeline.
12. The non-transitory machine-readable medium of claim 11 , wherein the monitoring criteria specifies at least the following as monitored quantities:
an actual uptime of the data pipeline;
occurrences of misalignments of the data pipeline, the occurrences of the misalignments of the data pipeline being based on the operation data of the API; and
portions of data managed by the data pipeline that are impacted by the occurrences of the misalignments.
13. The non-transitory machine-readable medium of claim 12 , wherein the monitoring criteria further specifies at least the following as additional monitored quantities:
re-alignments of the misalignments of the data pipeline; and
durations of time between the misalignments of the data pipeline and the re-alignments of the misalignments.
14. The non-transitory machine-readable medium of claim 13 , wherein the one or more evaluation metrics further comprise:
a threshold number of occurrences for the misalignments of the data pipeline; and
a value ascribed by a downstream consumer to the portions of the data.
15. The non-transitory machine-readable medium of claim 14 , wherein the one or more evaluation metrics further specify at least the following additional metrics:
a value ascribed by the downstream consumer for the durations of time.
16. A data processing system, comprising:
a processor; and
a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations for managing a data pipeline, the operations comprising:
monitoring the data pipeline based on monitoring criteria to obtain operation data of the data pipeline that is indicative of a performance of the data pipeline, the operation data being based at least on operation data of an application programming interface (API) that initiated transfer of data through the data pipeline in response to a data retrieval requests obtained by the API, the operation data of the API being indicative of a performance of the API;
obtaining representations of operation quality of the data pipeline by comparing the operation data with one or more evaluation metrics, the one or more evaluation metrics comprising at least a predefined uptime duration threshold for the data pipeline;
evaluating the representations of the operation quality of the data pipeline for compliance with operation quality goals for the data pipeline to obtain a degree of alignment with a use of the data pipeline; and
performing an action set based on the degree of alignment with the use of the data pipeline to improve a quality of data processing services provided by the data pipeline.
17. The data processing system of claim 16 , wherein the monitoring criteria specifies at least the following as monitored quantities:
an actual uptime of the data pipeline;
occurrences of misalignments of the data pipeline, the occurrences of the misalignments of the data pipeline being based on the operation data of the API; and
portions of data managed by the data pipeline that are impacted by the occurrences of the misalignments.
18. The data processing system of claim 17 , wherein the monitoring criteria further specifies at least the following as additional monitored quantities:
re-alignments of the misalignments of the data pipeline; and
durations of time between the misalignments of the data pipeline and the re-alignments of the misalignments.
19. The data processing system of claim 18 , wherein the one or more evaluation metrics further comprise:
a threshold number of occurrences for the misalignments of the data pipeline; and
a value ascribed by a downstream consumer to the portions of the data.
20. The data processing system of claim 19 , wherein the one or more evaluation metrics further specify at least the following additional metrics:
a value ascribed by the downstream consumer for the durations of time.