Vectorization for parsing of complexly structured files
Embodiments of the present disclosure are directed to methods and systems for processing of Electronic Data Interchange (EDI) files. More specifically, processing EDI files includes importing an EDI file into an initial data frame table. Each data segment of the EDI file can be imported into a different row of the initial data frame table and the rows and columns of the initial data frame table can comprise an array of fields. Vectorization processing can then be performed on the array of fields in the initial data frame table. A set of one or more target tables can then be populated with data from the imported EDI file based on performing the vectorization processing on the array of fields in the initial data frame table.
1. A method for processing Electronic Data Interchange (EDI) files, the method comprising:
importing, by a parser of a records management and processing system, an EDI file into an initial data table, wherein each data segment of the imported EDI file is imported into a different row of the initial data table and wherein the rows and columns of the initial data table comprise an array of fields;
performing, by the parser of the records management and processing system, vectorization processing on the array of fields in the initial data table, wherein performing the vectorization processing on the array of fields in the initial data table comprises performing a join operation on the initial data table with a file type specific data table to produce a joined data table, the file type specific data table selected from a plurality of file type specific data tables based on a file type for the imported EDI file; and
populating, by the parser of the records management and processing system, a set of one or more target tables with data from the imported EDI file based on performing the vectorization processing on the array of fields in the initial data table.
2. The method of claim 1 , wherein performing the vectorization processing on the array of fields in the initial data table and populating the set of one or more target tables with data from the imported EDI file comprise processing the entire EDI file and are performed in a single iteration.
3. The method of claim 1 , wherein performing the vectorization processing on the array of fields in the initial data table further comprises filtering the joined data table.
4. The method of claim 3 , wherein filtering the joined data table comprises removing redundant fields from the joined data table in a single iteration on the joined data table.
5. The method of claim 4 , wherein performing the vectorization processing on the array of fields in the initial data table further comprises splitting each row of the filtered data table into a split data table, wherein each row of the split data table comprises a single field of data from the imported EDI file.
6. The method of claim 5 , wherein populating the set of one or more target tables with data from the imported EDI comprises filtering the split data table and storing data from the split data table in the one or more target tables based on the filtering.
7. A system comprising:
a processor; and
a memory coupled with and readable by the processor and storing therein a set of instructions which, when executed by the processor, causes the processor to process Electronic Data Interchange (EDI) files by:
importing an EDI file into an initial data table, wherein each data segment of the imported EDI file is imported into a different row of the initial data table and wherein the rows and columns of the initial data table comprise an array of fields;
performing vectorization processing on the array of fields in the initial data table, wherein performing the vectorization processing on the array of fields in the initial data table comprises performing a join operation on the initial data table with a file type specific data table to produce a joined data frame table, the file type specific data table selected from a plurality of file type specific data tables based on a file type for the imported EDI file; and
populating a set of one or more target tables with data from the imported EDI file based on performing the vectorization processing on the array of fields in the initial data table.
8. The system of claim 7 , wherein performing the vectorization processing on the array of fields in the initial data table and populating the set of one or more target tables with data from the imported EDI file comprise processing the entire EDI file and are performed in a single iteration.
9. The system of claim 7 , wherein performing the vectorization processing on the array of fields in the initial data table further comprises filtering the joined data table.
10. The system of claim 9 , wherein filtering the joined data table comprises removing redundant fields from the joined data table in a single iteration on the joined data table.
11. The system of claim 10 , wherein performing the vectorization processing on the array of fields in the initial data table further comprises splitting each row of the filtered data table into a split data table, wherein each row of the split data table comprises a single field of data from the imported EDI file.
12. The system of claim 11 , wherein populating the set of one or more target tables with data from the imported EDI comprises filtering the split data table and storing data from the split data table in the one or more target tables based on the filtering.
13. A non-transitory, computer-readable medium comprising a set of instructions stored therein which, when executed by a processor, causes the processor to process Electronic Data Interchange (EDI) files by:
importing an EDI file into an initial data table, wherein each data segment of the imported EDI file is imported into a different row of the initial data table and wherein the rows and columns of the initial data table comprise an array of fields;
performing vectorization processing on the array of fields in the initial data table, wherein performing the vectorization processing on the array of fields in the initial data table comprises performing a join operation on the initial data table with a file type specific data table to produce a joined data table, the file type specific data table selected from a plurality of file type specific data tables based on a file type for the imported EDI file; and
populating a set of one or more target tables with data from the imported EDI file based on performing the vectorization processing on the array of fields in the initial data table.
14. The non-transitory, computer-readable medium of claim 13 , wherein performing the vectorization processing on the array of fields in the initial data table further comprises filtering the joined data table.
15. The non-transitory, computer-readable medium of claim 14 , wherein filtering the joined data table comprises removing redundant fields from the joined data table in a single iteration on the joined data table.
16. The non-transitory, computer-readable medium of claim 15 , wherein performing the vectorization processing on the array of fields in the initial data table further comprises splitting each row of the filtered data table into a split data table, wherein each row of the split data table comprises a single field of data from the imported EDI file.
17. The non-transitory, computer-readable medium of claim 16 , wherein populating the set of one or more target tables with data from the imported EDI comprises filtering the split data table and storing data from the split data table in the one or more target tables based on the filtering.