Tracking errors in data set lineage
Tracking errors in data set lineage including receiving, from a user, a change to a model data set, wherein the model data set is a reusable modeling layer comprising at least a portion of a data source retrieved from a data warehouse; accessing a list of dependent worksheets utilizing the model data set as a data source wherein each dependent worksheet is configured to perform analysis on the portion of the first data source within the model data set without changing the model data set; generating an error report for the dependent worksheets utilizing the model data set; and providing, to the user, the error report for the dependent worksheets utilizing the model data set.
1. A method of tracking errors in data set lineage, the method comprising:
receiving, from a client computing system, a change to a model data set, wherein the model data set is a reusable modeling layer comprising at least a portion of a first data source retrieved from a cloud-based data warehouse, wherein the model data set comprises a model data set worksheet;
accessing, from a worksheet repository, a list of dependent worksheets utilizing the model data set worksheet as a second data source wherein each dependent worksheet is configured to perform analysis on the portion of the first data source within the model data set worksheet without changing the model data set worksheet;
generating, based on the change to the model data set, an error report for the dependent worksheets utilizing the model data set including, for each dependent worksheet in the list of dependent worksheets utilizing the model data set:
rebuilding the dependent worksheet with the change to the model data set, including retrieving the first data source and the second data source and populating the dependent worksheet with the first data source and the second data source; and
determining whether the rebuilt dependent worksheet causes at least one error, including evaluating the rebuilt dependent worksheet for unrenderable results caused by referral to a removed element from within the model data set; and
providing, to the client computing system, the error report for the dependent worksheets utilizing the model data set.
2. The method of claim 1 , wherein rebuilding the dependent worksheet with the change to the model data set comprises rebuilding the dependent worksheet without retrieving the data source from the data warehouse.
3. The method of claim 1 , wherein rebuilding the dependent worksheet with the change to the model data set comprises rebuilding the dependent worksheet using a local data source.
4. The method of claim 1 , wherein generating the error report for the dependent worksheets utilizing the model data set further comprises organizing the error report to group together similar errors from different dependent worksheets.
5. The method of claim 1 , wherein providing, to the client computing system, the error report for the dependent worksheets utilizing the model data set comprises generating a suggested error resolution action.
6. The method of claim 1 , wherein determining whether the rebuilt dependent worksheet causes the at least one error comprises determining whether a function in the rebuilt dependent worksheet refers to a removed column within the model data set.
7. The method of claim 1 , wherein the change to the model data set is a change to a column of the model data set.
8. An apparatus for tracking errors in data set lineage, the apparatus comprising a computer processor, a computer memory operatively coupled to the computer processor, the computer memory having disposed within it computer program instructions that, when executed by the computer processor, cause the apparatus to carry out the steps of:
receiving, from a client computing system, a change to a model data set, wherein the model data set is a reusable modeling layer comprising at least a portion of a first data source retrieved from a cloud-based data warehouse, wherein the model data set comprises a model data set worksheet;
accessing, from a worksheet repository, a list of dependent worksheets utilizing the model data set worksheet as a second data source wherein each dependent worksheet is configured to perform analysis on the portion of the first data source within the model data set worksheet without changing the model data set worksheet;
generating, based on the change to the model data set, an error report for the dependent worksheets utilizing the model data set including, for each dependent worksheet in the list of dependent worksheets utilizing the model data set:
rebuilding the dependent worksheet with the change to the model data set, including retrieving the first data source and the second data source and populating the dependent worksheet with the first data source and the second data source; and
determining whether the rebuilt dependent worksheet causes at least one error, including evaluating the rebuilt dependent worksheet for unrenderable results caused by referral to a removed element from within the model data set; and
providing, to the client computing system, the error report for the dependent worksheets utilizing the model data set.
9. The apparatus of claim 8 , wherein rebuilding the dependent worksheet with the change to the model data set comprises rebuilding the dependent worksheet without retrieving the data source from the data warehouse.
10. The apparatus of claim 8 , wherein rebuilding the dependent worksheet with the change to the model data set comprises rebuilding the dependent worksheet using a local data source.
11. The apparatus of claim 8 , wherein generating the error report for the dependent worksheets utilizing the model data set further comprises organizing the error report to group together similar errors from different dependent worksheets.
12. The apparatus of claim 8 , wherein providing, to the client computing system, the error report for the dependent worksheets utilizing the model data set comprises generating a suggested error resolution action.
13. The apparatus of claim 8 , wherein determining whether the rebuilt dependent worksheet causes the at least one error comprises determining whether a function in the rebuilt dependent worksheet refers to a removed column within the model data set.
14. The apparatus of claim 8 , wherein the change to the model data set is a change to a column of the model data set.
15. A non-transitory computer readable medium for tracking errors in data set lineage, the non-transitory computer readable medium storing computer program instructions that, when executed, cause a computer to carry out the steps of:
receiving, from a client computing system, a change to a model data set, wherein the model data set is a reusable modeling layer comprising at least a portion of a first data source retrieved from a cloud-based data warehouse, wherein the model data set comprises a model data set worksheet;
accessing, from a worksheet repository, a list of dependent worksheets utilizing the model data set worksheet as a second data source wherein each dependent worksheet is configured to perform analysis on the portion of the first data source within the model data set worksheet without changing the model data set worksheet;
generating, based on the change to the model data set, an error report for the dependent worksheets utilizing the model data set including, for each dependent worksheet in the list of dependent worksheets utilizing the model data set:
rebuilding the dependent worksheet with the change to the model data set, including retrieving the first data source and the second data source and populating the dependent worksheet with the first data source and the second data source; and
determining whether the rebuilt dependent worksheet causes at least one error, including evaluating the rebuilt dependent worksheet for unrenderable results caused by referral to a removed element from within the model data set; and
providing, to the client computing system, the error report for the dependent worksheets utilizing the model data set.
16. The non-transitory computer readable medium of claim 15 , wherein rebuilding the dependent worksheet with the change to the model data set comprises rebuilding the dependent worksheet without retrieving the data source from the data warehouse.
17. The non-transitory computer readable medium of claim 15 , wherein rebuilding the dependent worksheet with the change to the model data set comprises rebuilding the dependent worksheet using a local data source.
18. The non-transitory computer readable medium of claim 15 , wherein generating the error report for the dependent worksheets utilizing the model data set further comprises organizing the error report to group together similar errors from different dependent worksheets.
19. The non-transitory computer readable medium of claim 15 , wherein providing, to the client computing system, the error report for the dependent worksheets utilizing the model data set comprises generating a suggested error resolution action.
20. The non-transitory computer readable medium of claim 15 , wherein determining whether the rebuilt dependent worksheet causes the at least one error comprises determining whether a function in the rebuilt dependent worksheet refers to a removed column within the model data set.