Systems and methods for importing data from electronic data files
Computer implemented systems and methods are disclosed for importing data from electronic data files. In accordance with some embodiments, a file format is assigned to a source electronic data files by a data importation system. The data importation system may further identify a file type identifier associated with the source electronic data file and map the source electronic data file to a transformation template. The data importation system may further store the file format, file type identifier, and an indication of the transformation template as a file type profile associated with the source electronic data file in a database.
1. A method for importing data, the method comprising:
receiving a source electronic data file;
determining a file type of the source electronic data file;
receiving information associated with transformed electronic data file;
selecting a detector/transformer plugin from a plurality of detector/transformer plugins by at least:
for each detector/transformer plugin in at least a portion of the plurality of detector/transformer plugins,
determining whether the each detector/transformer plugin is capable of transforming the source electronic data file into the transformed electronic data file based on the file type and the information associated with the transformed electronic data file; and
selecting the selected detector/transformer plugin based on one or more priorities associated with the at least a portion of the plurality of detector/transformer plugins;
transforming the source electronic data file into the transformed electronic data file using the selected detector/transformer plugin; and
providing the transformed electronic data file for importing into a data analysis system;
wherein at least a part of the method is performed using one or more processors.
2. The method of claim 1 , wherein the determining a file type of the source electronic data file includes extracting a text string from the source electronic data file and determining the file type based on the extracted text string.
3. The method of claim 1 , wherein the determining a file type of the source electronic data file includes extracting a text string at a predetermined file location in the source electronic data file and determining the file type based on the extracted text string.
4. The method of claim 1 , wherein the determining a file type of the source electronic data file includes:
generating instructions for displaying a visual indicator for indicating that the file type associated with the source electronic data file is to be determined; and
receiving a file type identifier associated with the source electronic data file.
5. The method of claim 1 , wherein the determining a file type of the source electronic data file includes determining the file type based on at least one selected from a group consisting of a file format, a file structure, a text string, a metadata, one or more file headers, an order of the one or more file headers, and a type of data.
6. The method of claim 1 , wherein the receiving a source electronic data file includes receiving a plurality of source electronic data files from a single entity.
7. The method of claim 6 , wherein the selecting a detector/transformer plugin from a plurality of detector/transformer plugins includes selecting the detector/transformer plugin by determining whether the each detector/transformer plugin is capable of transforming each source electronic data file of the plurality of source electronic data files into a corresponding transformed electronic data file.
8. The method of claim 1 , wherein the transforming the source electronic data file into the transformed electronic data file includes:
determining that the selected detector/transformer plugin is associated with a transformation template; and
mapping the source electronic data file to a canonical format included in the transformation template.
9. The method of claim 1 , further comprising:
receiving a request to delete the source electronic data file;
detecting that a transformed electronic data file is associated with the source electronic data file; and
deleting the source electronic data file and the transformed electronic data file.
10. A system for importing data, the system comprising:
one or more processors; and
one or more memories having instructions stored thereon, wherein the instructions, when executed by the one or more processors, cause the system to perform operations comprising:
receiving a source electronic data file;
determining a file type of the source electronic data file;
receiving information associated with transformed electronic data file;
selecting a detector/transformer plugin from a plurality of detector/transformer plugins by at least:
for each detector/transformer plugin in at least a portion of the plurality of detector/transformer plugins,
determining whether the each detector/transformer plugin is capable of transforming the source electronic data file into the transformed electronic data file based on the file type and the information associated with the transformed electronic data file; and
selecting the selected detector/transformer plugin based on one or more priorities associated with the at least a portion of the plurality of detector/transformer plugins;
transforming the source electronic data file into the transformed electronic data file using the selected detector/transformer plugin; and
providing the transformed electronic data file for importing into a data analysis system.
11. The system of claim 10 , wherein the determining a file type of the source electronic data file includes extracting a text string from the source electronic data file and determining the file type based on the extracted text string.
12. The system of claim 10 , wherein the determining a file type of the source electronic data file includes extracting a text string at a predetermined file location in the source electronic data file and determining the file type based on the extracted text string.
13. The system of claim 10 , wherein the determining a file type of the source electronic data file includes:
generating instructions for displaying a visual indicator for indicating that the file type associated with the source electronic data file is to be determined; and
receiving a file type identifier associated with the source electronic data file.
14. The system of claim 10 , wherein the determining a file type of the source electronic data file includes determining the file type based on at least one selected from a group consisting of a file format, a file structure, a text string, a metadata, one or more file headers, an order of the one or more file headers, and a type of data.
15. The system of claim 10 , wherein the receiving a source electronic data file includes receiving a plurality of source electronic data files from a single entity.
16. The system of claim 15 , wherein the selecting a detector/transformer plugin from a plurality of detector/transformer plugins includes selecting the detector/transformer plugin by determining whether the each detector/transformer plugin is capable of transforming each source electronic data file of the plurality of source electronic data files into a corresponding transformed electronic data file.
17. The system of claim 10 , wherein the transforming the source electronic data file into the transformed electronic data file includes:
determining that the selected detector/transformer plugin is associated with a transformation template; and
mapping the source electronic data file to a canonical format included in the transformation template.
18. The system of claim 10 , wherein the operations further comprise:
receiving a request to delete the source electronic data file;
detecting that a transformed electronic data file is associated with the source electronic data file; and
deleting the source electronic data file and the transformed electronic data file.
19. A method for importing data, the method comprising:
receiving a source electronic data file;
determining a file type of the source electronic data file;
selecting a detector/transformer plugin from a plurality of detector/transformer plugins by at least:
for each detector/transformer plugin in at least a portion of the plurality of detector/transformer plugins,
determining whether the each detector/transformer plugin is capable of transforming the source electronic data file into transformed electronic data file based on the file type; and
selecting the selected detector/transformer plugin based on one or more priorities associated with the at least a portion of the plurality of detector/transformer plugins;
transforming the source electronic data file into the transformed electronic data file using the selected detector/transformer plugin by at least:
determining that the selected detector/transformer plugin is associated with a transformation template; and
mapping the source electronic data file to a canonical format included in the transformation template; and
providing the transformed electronic data file for importing into a data analysis system;
wherein at least a part of the method is performed using one or more processors.
20. The method of claim 19 , wherein the determining a file type of the source electronic data file includes:
generating instructions for displaying a visual indicator for indicating that the file type associated with the source electronic data file is to be determined; and
receiving a file type identifier associated with the source electronic data file.