Systems and methods for importing data from electronic data files
Computer implemented systems and methods are disclosed for importing data from electronic data files. In accordance with some embodiments, source electronic data files are received at a data importation system and managed by the data importation system. The data importation system may load detector/transformer plugins and determined whether any of the load detector/transformer plugins apply to the received source electronic data files. The data importation system may also generate transformed data using a transformer corresponding to a detector that applies to the source electronic data files. The data importation system may import the transformed data into at least one of a data analysis system and a database.
1. A system comprising:
one or more memory devices that store a set of instructions; and
at least one processor that executes the set of instructions to perform operations for transforming an updated source electronic data file into an updated transformed electronic data file that is capable of being imported into a data analysis system, the operations comprising:
receiving a source electronic data file;
receiving an update to the source electronic data file as the updated source electronic data file;
selecting a detector/transformer plugin from a plurality of detector/transformer plugins stored in a database, the selecting including cycling through at least a portion of the plurality of detector/transformer plugins based on priorities associated with the plurality of detector/transformer plugins and determining which of the at least the portion of the plurality of detector/transformer plugins is capable of transforming the updated source electronic data file into the updated transformed electronic data file;
performing the transforming of the updated source electronic data file into the updated transformed electronic data file using the selected detector/transformer plugin; and
providing the updated transformed electronic data file for importing into the data analysis system.
2. The system of claim 1 , wherein the cycling through of the at least the portion of the detector/transformer plugins includes:
loading the at least the portion of the plurality of detector/transformer plugins;
determining whether a first detector/transformer of the at least the portion of the detector/transformer plugins applies; and
based on a determination that the first detector/transformer does not apply, moving onto a second detector/transformer of the at least the portion of the detector/transformer plugins.
3. The system of claim 2 , wherein the at least the portion of the detector/transformer plugins is selected based on a priority associated with each of the at least the portion of detector/transformer plugins exceeding a priority threshold.
4. The system of claim 2 , wherein the determination that the first detector/transformer does not apply is based on a determination that a transformer of the detector/transformer plugin is not capable of performing the transforming of the updated source electronic data file into the updated transformed electronic data file.
5. The system of claim 1 , the operations further comprising, based on a determination that the selected detector/transformer is associated with a data mapper, mapping the source electronic data file and storing the mapped version of the source electronic data file to support mapping of a format of the source electronic data file to a canonical format included in a transformation template.
6. The system of claim 1 , wherein the performing the transforming of the updated source electronic data file is based on a selection of a process option included in a graphical user interface.
7. The system 1 , wherein the source electronic data file is a spreadsheet and the selecting of the detector/transformer plugin is based on file types associated with a tab of a spreadsheet.
8. A method comprising:
performing operations for transforming an updated source electronic data file into an updated transformed electronic data file that is capable of being imported into a data analysis system, the operations comprising:
receiving a source electronic data file;
receiving an update to the source electronic data file as the updated source electronic data file;
selecting the detector/transformer plugin from a plurality of detector/transformer plugins stored in a database, the selecting including cycling through at least a portion of the plurality of detector/transformer plugins based on priorities associated with the plurality of detector/transformer plugins and determining which of the at least the portion of the plurality of detector/transformer plugins is capable of transforming the updated source electronic data file into the updated transformed electronic data file;
performing the transforming of the updated source electronic data file into the updated transformed electronic data file using the selected detector/transformer plugin; and
providing the updated transformed electronic data file for importing into the data analysis system.
9. The method of claim 8 , wherein the cycling through of the at least the portion of the detector/transformer plugins includes:
loading the at least the portion of the plurality of detector/transformer plugins;
determining whether a first detector/transformer of the at least the portion of the detector/transformer plugins applies; and
based on a determination that the first detector/transformer does not apply, moving onto a second detector/transformer of the at least the portion of the detector/transformer plugins.
10. The method of claim 9 , wherein the at least the portion of the detector/transformer plugins is selected based on a priority associated with each of the at least the portion of detector/transformer plugins exceeding a priority threshold.
11. The method of claim 9 , wherein the determination that the first detector/transformer does not apply is based on a determination that a transformer of the detector/transformer plugin is not capable of performing the transforming of the updated source electronic data file into the updated transformed electronic data file.
12. The method of claim 8 , the operations further comprising, based on a determination that the selected detector/transformer is associated with a data mapper, mapping the source electronic data file and storing the mapped version of the source electronic data file to support mapping of a format of the source electronic data file to a canonical format included in a transformation template.
13. The method of claim 8 , wherein the performing the transforming of the updated source electronic data file is based on a selection of a process option included in a graphical user interface.
14. The method of claim 8 , wherein the source electronic data file is a spreadsheet and the selecting of the detector/transformer plugin is based on file types associated with a tab of a spreadsheet.
15. A non-transitory computer-readable storage medium comprising instructions that, when executed by at least one processor of a machine, cause the machine to perform operations for transforming an updated source electronic data file into an updated transformed electronic data file that is capable of being imported into a data analysis system, the operations comprising:
receiving a source electronic data file;
receiving an update to the source electronic data file as the updated source electronic data file;
selecting the detector/transformer plugin from a plurality of detector/transformer plugins stored in a database, the selecting including cycling through at least a portion of the plurality of detector/transformer plugins based on priorities associated with the plurality of detector/transformer plugins and determining which of the at least the portion of the plurality of detector/transformer plugins is capable of transforming the updated source electronic data file into the updated transformed electronic data file;
performing the transforming of the updated source electronic data file into the updated transformed electronic data file using the selected detector/transformer plugin; and
providing the updated transformed electronic data file for importing into the data analysis system.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the cycling through of the at least the portion of the detector/transformer plugins includes:
loading the at least the portion of the plurality of detector/transformer plugins;
determining whether a first detector/transformer of the at least the portion of the detector/transformer plugins applies; and
based on a determination that the first detector/transformer does not apply, moving onto a second detector/transformer of the at least the portion of the detector/transformer plugins.
17. The non-transitory computer-readable storage medium of claim 16 , wherein the at least the portion of the detector/transformer plugins is selected based on a priority associated with each of the at least the portion of detector/transformer plugins exceeding a priority threshold.
18. The non-transitory computer-readable storage medium of claim 16 , wherein the determination that the first detector/transformer does not apply is based on a determination that a transformer of the detector/transformer plugin is not capable of performing the transforming of the updated source electronic data file into the updated transformed electronic data file.
19. The non-transitory computer-readable storage medium of claim 15 , the operations further comprising, based on a determination that the selected detector/transformer is associated with a data mapper, mapping the source electronic data file and storing the mapped version of the source electronic data file to support mapping of a format of the source electronic data file to a canonical format included in a transformation template.
20. The non-transitory computer-readable storage medium of claim 15 , wherein the performing the transforming of the updated source electronic data file is based on a selection of a process option included in a graphical user interface.