Systems and methods for updating data pipelines

1. A computer-implemented method comprising:

identifying at least one file that defines a series of transformations for data that passes through a production version of a data pipeline for an application, wherein the data pipeline reads input data from a plurality of input tables and writes output data to at least one output table;

designating at least one additional file as a development version of the data pipeline, wherein the at least one additional file comprises a modified version of the at least one file and defines a modified series of transformations for the data;

creating an additional version of the output table;

detecting an execution of the additional file that is designated as the development version of the data pipeline; and

in response to both detecting the execution of the additional file and determining that the additional file is designated as the development version of the data pipeline:

directing the execution of the additional file to process at least one input table in the plurality of input tables and process an additional version of each remaining input table in the plurality of input tables such that the production version of the data pipeline and the development version of the data pipeline both use the at least one input table as a common input; and

directing output data written by the additional file to the additional version of the output table instead of the output table, thereby enabling a developer to test changes to the data pipeline via the development version of the data pipeline instead of the production version of the data pipeline.

2. The computer-implemented method of claim 1 , wherein:

designating the additional file as a development version of the data pipeline comprises detecting the additional file in a directory designated as a development directory; and

determining that the additional file is designated as the development version of the data pipeline comprises determining that the additional file is stored in the development directory.

3. The computer-implemented method of claim 2 , wherein determining that the additional file is stored in the development directory comprises parsing a name of the development directory with a regular expression.

4. The computer-implemented method of claim 1 , further comprising:

creating an additional version of at least one input table in the plurality of input tables at least in part by copying data from the at least one input table; and

directing the additional file to read input data from the additional version of the at least one input table instead of the input table in response to both detecting the execution of the additional file and determining that the additional file is designated as the development version of the data pipeline.

5. The computer-implemented method of claim 4 , further comprising:

determining that the additional version of the at least one input table is no longer up to date; and

updating the additional version of the at least one input table by copying up-to-date data from the at least one input table.

6. The computer-implemented method of claim 1 , wherein:

the output table is accessible to end users of the application; and

the additional version of the output table is not accessible to end users of the application.

7. The computer-implemented method of claim 1 :

further comprising creating a list of tables not to copy; and

wherein creating the additional versions of the remaining input tables comprises:

identifying a set of tables relevant to the application; and

for each table within the set of tables that is not in the list of tables not to copy, creating an additional version of the table.

8. The computer-implemented method of claim 1 , further comprising replacing data within the output table with the output data written by the development version of the data pipeline to the additional version of the output table.

9. The computer-implemented method of claim 1 , further comprising modifying the file by copying content from the additional file to the file.

10. The computer-implemented method of claim 1 , further comprising creating, at regular intervals, at least one new file that comprises a new modified version of the file and that is designated as the development version of the data pipeline.

11. A system comprising:

an identification module, stored in memory, that identifies at least one file that defines a series of transformations for data that passes through a production version of a data pipeline for an application, wherein the data pipeline reads input data from a plurality of input tables and writes output data to at least one output table;

a designation module, stored in memory, that designates at least one additional file as a development version of the data pipeline, wherein the at least one additional file comprises a modified version of the at least one file and defines a modified series of transformations for the data;

a creation module, stored in memory, that creates an additional version of the output table;

a detection module, stored in memory, that detects an execution of the additional file that is designated as the development version of the data pipeline;

a direction module, stored in memory, that, in response to both detecting the execution of the additional file and determining that the additional file is designated as the development version of the data pipeline:

directs the execution of the additional file to process at least one input table in the plurality of input tables and process an additional version of each remaining input table in the plurality of input tables such that the production version of the data pipeline and the development version of the data pipeline both use the at least one input table as a common input; and

directs output data written by the additional file to the additional version of the output table instead of the output table, thereby enabling a developer to test changes to the data pipeline via the development version of the data pipeline instead of the production version of the data pipeline; and

at least one physical processor configured to execute the identification module, the designation module, the creation module, the detection module, and the direction module.

12. The system of claim 11 , wherein:

the designation module designates the additional file as a development version of the data pipeline by detecting the additional file in a directory designated as a development directory; and

the detection module determines that the additional file is designated as the development version of the data pipeline by determining that the additional file is stored in the development directory.

13. The system of claim 12 , wherein the detection module determines that the additional file is stored in the development directory by parsing a name of the development directory with a regular expression.

14. The system of claim 11 , wherein:

the creation module creates an additional version of the at least one input table in the plurality of input tables at least in part by copying data from the at least one input table; and

the direction module directs the additional file to read input data from the additional version of the input table instead of the input table in response to both detecting the execution of the additional file and determining that the additional file is designated as the development version of the data pipeline.

15. The system of claim 14 , wherein the creation module:

determines that the additional version of the at least one input table is no longer up to date; and

updates the additional version of the at least one input table by copying up-to-date data from the at least one input table.

16. The system of claim 11 , wherein:

the output table is accessible to end users of the application; and

the additional version of the output table is not accessible to end users of the application.

17. The system of claim 11 , wherein:

the creation module creates a list of tables not to copy;

the creation module creates the additional versions of the remaining input tables by:

identifying a set of tables relevant to the application; and

for each table within the set of tables that is not in the list of tables not to copy, creating an additional version of the table.

18. The system of claim 11 , wherein the creation module replaces data within the output table with the output data written by the development version of the data pipeline to the additional version of the output table.

19. The system of claim 11 , wherein the creation module modifies the file by copying content from the additional file to the file.

20. A non-transitory computer-readable medium comprising one or more computer-readable instructions that, when executed by at least one processor of a computing device, cause the computing device to:

identify at least one file that defines a series of transformations for data that passes through a production version of a data pipeline for an application, wherein the data pipeline reads input data from a plurality of input tables and writes output data to at least one output table;

designate at least one additional file as a development version of the data pipeline, wherein the at least one additional file comprises a modified version of the at least one file and defines a modified series of transformations for the data;

create an additional version of the output table;

detect an execution of the additional file that is designated as the development version of the data pipeline; and

in response to both detecting the execution of the additional file and determining that the additional file is designated as the development version of the data pipeline:

direct the execution of the additional file to process at least one input table in the plurality of input tables and process an additional version of each remaining input table in the plurality of input tables such that the production version of the data pipeline and the development version of the data pipeline both use the at least one input table as a common input; and

direct output data written by the additional file to the additional version of the output table instead of the output table, thereby enabling a developer to test changes to the data pipeline via the development version of the data pipeline instead of the production version of the data pipeline.