System and method for managing a data pipeline using a digital twin
Methods and systems for managing operation of a data pipeline are disclosed. To manage the data pipeline, a system may include one or more data sources, a data repository, and one or more downstream consumers. Updates to the data pipeline may cause the data pipeline to become misaligned. To avoid misalignment and, therefore, failure of the data pipeline, any updates intended for implementation in the data pipeline may first be implemented in a digital twin of the data pipeline. The digital twin and the data pipeline may operate concurrently using substantially identical data. The performance of the digital twin following implementation of the potential update may be compared to the performance of the data pipeline without implementation of the potential update to obtain a performance delta. If the performance delta meets a threshold, the potential update may cause failure of the data pipeline and may be remediated prior to implementation.
1. A method of managing a data pipeline, the method comprising:
obtaining a potential update intended to modify operation of at least a portion of the data pipeline;
simulating the data pipeline using a digital twin of the data pipeline, the potential update, and input data to obtain a first simulated performance report, the input data being data obtained from one or more data sources associated with the data pipeline;
making a determination, based on the first simulated performance report and a first live performance report, regarding whether the data pipeline will become misaligned by implementation of the potential update, the first live performance report being based on operation of the data pipeline using the input data;
in a first instance of the determination in which the data pipeline will not become misaligned by implementation of the potential update:
initiating an update process for the data pipeline based on the potential update to obtain an updated data pipeline;
in a second instance of the determination in which the data pipeline will become misaligned by implementation of the potential update:
denying the potential update for implementation in the data pipeline; and
performing an action set to remediate the potential update.
2. The method of claim 1 , wherein the digital twin duplicates the operation of the data pipeline and the digital twin operates concurrently with the data pipeline.
3. The method of claim 2 , wherein simulating the data pipeline comprises:
modifying operation of at least a portion of the digital twin using the potential update to obtain an updated digital twin.
4. The method of claim 3 , wherein simulating the data pipeline further comprises:
initiating, in response to obtaining the updated digital twin, generation of:
the first simulated performance report by the updated digital twin; and
the first live performance report by the data pipeline.
5. The method of claim 4 , wherein the first simulated performance report comprises:
a listing of records associated with the input data, each record of the listing of the records comprising:
a first format of a data package prior to encountering an application programming interface associated with the digital twin; and
a second format of a data package after encountering the application programming interface.
6. The method of claim 5 , wherein making the determination comprises:
identifying at least one performance delta between operation of the updated digital twin and the operation of the data pipeline;
comparing the at least one performance delta to a threshold, the threshold indicating that the data pipeline has become misaligned when met.
7. The method of claim 6 , wherein identifying the at least one performance delta comprises:
obtaining a difference based on the second format of the data package from the first simulated performance report and a corresponding second format of the data package from the first live performance report.
8. The method of claim 7 , wherein the difference comprises at least one selected from a list consisting of:
an indicator of a failure of the application programming interface; and
a quantification of a formatting change between the second format of the data package from the first simulated performance report and the corresponding second format of the data package from the first live performance report.
9. The method of claim 8 , wherein the failure of the application programming interface causes the at least one performance delta to meet the threshold and the data pipeline becoming misaligned causes failure of the data pipeline.
10. The method of claim 9 , wherein initiating the update process comprises:
monitoring operation of the updated data pipeline for a duration of time.
11. The method of claim 10 , wherein monitoring the operation of the updated data pipeline for the duration of time comprises:
obtaining a second live performance report for the data pipeline for the duration of time;
comparing the second live performance report to a second simulated performance report to obtain a second performance delta, the second simulated performance report being based on continued operation of the updated digital twin for the duration of time; and
comparing the second performance delta to the threshold to obtain an operational status for the updated digital twin, the operational status indicating a degree to which the updated digital twin accurately duplicates the operation of the updated data pipeline.
12. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations for managing a data pipeline, the operations comprising:
obtaining a potential update intended to modify operation of at least a portion of the data pipeline;
simulating the data pipeline using a digital twin of the data pipeline, the potential update, and input data to obtain a first simulated performance report, the input data being data obtained from one or more data sources associated with the data pipeline;
making a determination, based on the first simulated performance report and a first live performance report, regarding whether the data pipeline will become misaligned by implementation of the potential update, the first live performance report being based on operation of the data pipeline using the input data;
in a first instance of the determination in which the data pipeline will not become misaligned by implementation of the potential update:
initiating an update process for the data pipeline based on the potential update to obtain an updated data pipeline;
in a second instance of the determination in which the data pipeline will become misaligned by implementation of the potential update:
denying the potential update for implementation in the data pipeline; and
performing an action set to remediate the potential update.
13. The non-transitory machine-readable medium of claim 12 , wherein the digital twin duplicates the operation of the data pipeline and the digital twin operates concurrently with the data pipeline.
14. The non-transitory machine-readable medium of claim 13 , wherein simulating the data pipeline comprises:
modifying operation of at least a portion of the digital twin using the potential update to obtain an updated digital twin.
15. The non-transitory machine-readable medium of claim 14 , wherein simulating the data pipeline further comprises:
initiating, in response to obtaining the updated digital twin, generation of:
the first simulated performance report by the updated digital twin; and
the first live performance report by the data pipeline.
16. The non-transitory machine-readable medium of claim 15 , wherein the first simulated performance report comprises:
a listing of records associated with the input data, each record of the listing of the records comprising:
a first format of a data package prior to encountering an application programming interface associated with the digital twin; and
a second format of a data package after encountering the application programming interface.
17. A data processing system, comprising:
a processor; and
a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations for managing a data pipeline, the operations comprising:
obtaining a potential update intended to modify operation of at least a portion of the data pipeline;
simulating the data pipeline using a digital twin of the data pipeline, the potential update, and input data to obtain a first simulated performance report, the input data being data obtained from one or more data sources associated with the data pipeline;
making a determination, based on the first simulated performance report and a first live performance report, regarding whether the data pipeline will become misaligned by implementation of the potential update, the first live performance report being based on operation of the data pipeline using the input data;
in a first instance of the determination in which the data pipeline will not become misaligned by implementation of the potential update:
initiating an update process for the data pipeline based on the potential update to obtain an updated data pipeline;
in a second instance of the determination in which the data pipeline will become misaligned by implementation of the potential update:
denying the potential update for implementation in the data pipeline; and
performing an action set to remediate the potential update.
18. The data processing system of claim 17 , wherein the digital twin duplicates the operation of the data pipeline and the digital twin operates concurrently with the data pipeline.
19. The data processing system of claim 18 , wherein simulating the data pipeline comprises:
modifying operation of at least a portion of the digital twin using the potential update to obtain an updated digital twin.
20. The data processing system of claim 19 , wherein simulating the data pipeline further comprises:
initiating, in response to obtaining the updated digital twin, generation of:
the first simulated performance report by the updated digital twin; and
the first live performance report by the data pipeline.