Customizable pipeline for integrating data
Systems, methods, and non-transitory computer readable media are provided for customizing pipelines for integrating data. A file to be ingested into a data analysis platform may be determined. The file type of the file may be detected. The file may be transformed based on the file type. The transformation may include applying a set of operations to the file. The set of operations may correspond to a pipeline of operations associated with the file type.
1. A system comprising: one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the system to perform:
determining a file is to be ingested into a data analysis platform in response to a user selecting the file;
detecting a file type of the file based on structure information, pattern information, and actual information within the file, the detecting comprising:
continuously passing the file to different detectors until one of the detectors recognizes the file, wherein the one of the detectors generates metadata including information relating to the file type and information relating to one or more transformations to be performed based on the file type; and
in response to none of the different detectors recognizing the file, marking the file as a generic file and creating a new detector to recognize the generic file; and
streaming the file to a data transformation platform to transform the file based on the file type and based on the one of the detectors that recognizes the file, wherein the transformation includes applying a set of operations to the file, the set of operations comprising identifying data to be integrated into the file using a pipeline operation associated with the file type, wherein the pipeline operation enables a dynamic selection of options for one or more operations that include a linear pipeline and a branching pipeline of operations.
2. The system of claim 1 , wherein the one or more operations include one or more serial operations, one or more parallel operations, and one or more join operations, the one or more join operations merging two or more results of the parallel operations.
3. The system of claim 1 , wherein the one or more operations include a normalizing operation.
4. The system of claim 1 , wherein the file type of the file is detected based on an arrangement of information within the file.
5. The system of claim 1 , wherein the instructions further cause the system to perform:
changing the pipeline operation; and
removing one or more effects of a prior pipeline of operations.
6. The system of claim 1 , wherein the transformation of the file is case dependent, upload dependent, or user dependent.
7. A method implemented by a computing system including one or more processors and storage media storing machine-readable instructions, wherein the method is performed using the one or more processors, the method comprising:
determining a file is to be ingested into a data analysis platform in response to a user selecting the file;
detecting a file type of the file based on structure information, pattern information, and actual information within the file, the detecting comprising:
continuously passing the file to different detectors until one of the detectors recognizes the file, wherein the one of the detectors generates metadata including information relating to the file type and information relating to one or more transformations to be performed based on the file type; and
in response to none of the different detectors recognizing the file, marking the file as a generic file and creating a new detector to recognize the generic file; and
streaming the file to a data transformation platform to transform the file based on the file type and based on the one of the detectors that recognizes the file, wherein the transformation includes applying a set of operations to the file, the set of operations comprising identifying data to be integrated into the file using a pipeline operation associated with the file type, wherein the pipeline operation enables a dynamic selection of options for one or more operations that include a linear pipeline and a branching pipeline of operations.
8. The method of claim 7 , wherein the one or more operations include one or more serial operations, one or more parallel operations, and one or more join operations, the one or more join operations merging two or more results of the parallel operations.
9. The method of claim 7 , wherein the one or more operations include a normalizing operation.
10. The method of claim 7 , wherein the file type of the file is detected based on an arrangement of information within the file.
11. The method of claim 7 , further comprising:
changing the pipeline operation; and
removing one or more effects of a prior pipeline of operations.
12. The method of claim 7 , wherein the transformation of the file is case dependent, upload dependent, or user dependent.
13. A non-transitory computer readable medium comprising instructions that, when executed, cause one or more processors to perform:
determining a file is to be ingested into a data analysis platform in response to a user selecting the file;
detecting a file type of the file based on structure information, pattern information, and actual information within the file, the detecting comprising:
continuously passing the file to different detectors until one of the detectors recognizes the file, wherein the one of the detectors generates metadata including information relating to the file type and information relating to one or more transformations to be performed based on the file type; and
in response to none of the different detectors recognizing the file, marking the file as a generic file and creating a new detector to recognize the generic file; and
streaming the file to a data transformation platform to transform the file based on the file type and based on the one of the detectors that recognizes the file, wherein the transformation includes applying a set of operations to the file, the set of operations comprising identifying data to be integrated into the file using a pipeline operation associated with the file type, wherein the pipeline operation enables a dynamic selection of options for one or more operations that include a linear pipeline and a branching pipeline of operations.
14. The non-transitory computer readable medium of claim 13 , wherein the one or more operations include one or more serial operations, one or more parallel operations, and one or more join operations, the one or more join operations merging two or more results of the parallel operations.
15. The non-transitory computer readable medium of claim 13 , wherein the one or more operations include a normalizing operation.
16. The non-transitory computer readable medium of claim 13 , wherein the file type of the file is detected based on an arrangement of information within the file.
17. The non-transitory computer readable medium of claim 13 , wherein the instructions further cause the one or more processors to perform:
changing the pipeline operation; and
removing one or more effects of a prior pipeline of operations.