PIPELINE MANAGEMENT TOOL
Systems, methods, and non-transitory computer readable media are provided for managing pipelines of operations on data. A system may access data and provide a set of functions for the data. The system may receive a user's selection of one or more functions from the set of functions. The system may generate a pipeline of operations for the data based on the user's selection. The pipeline of operations may include the function(s) selected by the user.
1 . A system comprising:
one or more processors;
memory storing instructions that, when executed by the one or more processors, cause the system to perform:
providing access to a set of functions, each function configured to perform one or more operations on data, wherein the set of functions comprise a first function and a second function that references or calls the first function;
receiving a modification of the first function;
providing options to confirm the modification of the first function, reverse the modification of the first function, or to store the modification as a new function;
receiving an indication of whether to confirm or reverse the modification of the first function, or to store the modification as a new function; and
selectively modify the first function and the second function according to the indication.
2 . The system of claim 1 , wherein the selectively modifying of the first function and the second function comprises:
in response to receiving an indication to store the modification as a new function, refraining from modifying the second function.
3 . The system of claim 1 , wherein the modification of the first function comprises a change in code corresponding to the first function.
4 . The system of claim 1 , wherein the modification of the first function comprises a change in a restriction of a data type of an argument for the first function.
5 . The system of claim 4 , wherein the instructions further cause the system to perform:
providing a listing of portions of the data as potential arguments based on the change in the restriction of the data type; and
ranking the potential arguments.
6 . The system of claim 1 , wherein the modification of the first function comprises a change in a dataframe of the first function.
7 . The system of claim 1 , wherein the instructions that, when executed by the one or more processors, cause the system to perform:
communicating an impact of the modification of the first function on the second function, wherein the impact comprises a resulting change in data operations of the second function.
8 . The system of claim 1 , wherein the providing access to the set of functions comprises suggesting the set of functions based on the data or a historical usage of the set of functions and the suggesting the set of functions comprises ranking the set of functions according to a historical likelihood of use for a data type of the data or a frequency of prior use of the set of functions.
9 . The system of claim 8 , wherein suggesting the set of functions comprises suggesting one or more parameters for at least one function.
10 . The system of claim 1 , wherein the data is first data, and wherein execution of the instructions further cause the system to perform:
determining that second data is historically used with the first data by the first function at least a threshold number of times; and
suggesting, via the pipeline creation interface, that the second data be used as an argument of the first function.
11 . A method, comprising:
providing access to a set of functions, each function configured to perform one or more operations on data, wherein the set of functions comprise a first function and a second function that references or calls the first function;
receiving a modification of the first function;
providing options to confirm the modification of the first function, reverse the modification of the first function, or to store the modification as a new function;
receiving an indication of whether to confirm or reverse the modification of the first function, or to store the modification as a new function; and
selectively modify the first function and the second function according to the indication.
12 . The method of claim 11 , wherein the selectively modifying of the first function and the second function comprises:
in response to receiving an indication to store the modification as a new function, refraining from modifying the second function.
13 . The method of claim 11 , wherein the modification of the first function comprises a change in code corresponding to the first function.
14 . The method of claim 11 , wherein the modification of the first function comprises a change in a restriction of a data type of an argument for the first function.
15 . The method of claim 14 , further comprising:
providing a listing of portions of the data as potential arguments based on the change in the restriction of the data type; and
ranking the potential arguments.
16 . The method of claim 11 , wherein the modification of the first function comprises a change in a dataframe of the first function.
17 . The method of claim 11 , further comprising:
communicating an impact of the modification of the first function on the second function, wherein the impact comprises a resulting change in data operations of the second function.
18 . The method of claim 11 , wherein the providing access to the set of functions comprises suggesting the set of functions based on the data or a historical usage of the set of functions and the suggesting the set of functions comprises ranking the set of functions according to a historical likelihood of use for a data type of the data or a frequency of prior use of the set of functions.
19 . The method of claim 18 , wherein suggesting the set of functions comprises suggesting one or more parameters for at least one function.
20 . A non-transitory computer readable medium comprising instructions that, when executed, cause one or more processors to perform:
providing access to a set of functions, each function configured to perform one or more operations on data, wherein the set of functions comprise a first function and a second function that references or calls the first function;
receiving a modification of the first function;
providing options to confirm the modification of the first function, reverse the modification of the first function, or to store the modification as a new function;
receiving an indication of whether to confirm or reverse the modification of the first function, or to store the modification as a new function; and
selectively modify the first function and the second function according to the indication.