Bulk data extract hybrid job processing
Methods for hybrid job processing may include receiving raw data records stored within a plurality of tables from a plurality of systems of record at a raw data layer within a data exchange. Methods may include generating, based on a data model, a list of dependencies between the plurality of tables. Each table included in a second subset of the plurality of tables may be dependent on at least one table included in a first subset of the plurality of tables. Methods may include processing the first subset of the plurality of tables concurrently with one another. The processing includes modeling the raw data records and transmitting the modeled data records to the model data layer. Methods may include processing each table included in the second subset after completion of processing of the table included in the first subset from which the table in the second subset depends on.
1. A method for hybrid job processing, the hybrid job processing comprising modeling raw data from a raw data layer to modeled data in a model data layer, the method comprising:
receiving raw data records, said raw data records comprising a plurality of tables from a plurality of systems of record, at the raw data layer within a data exchange;
generating a data model from the raw data records, the data model comprising one or more dependencies within the plurality of tables, wherein each table in a second subset of the plurality of tables is dependent on at least one table in a first subset of the plurality of tables;
processing, at a computer processor:
the first subset of tables using parallel processing; and
the second subset of the tables using serial processing, wherein processing is initiated for each table in the second subset when a table with an associated dependency is complete;
said processing comprising:
modeling the raw data records stored in the first subset of the plurality of tables at the raw data layer to form modeled records; and
transmitting the modeled records in the first subset of tables to the model data layer;
wherein:
each of the tables in the plurality of tables is labeled based on a sequence of numbers; and
the first subset of tables comprises root tables labeled with an odd number and the second subset of tables comprises leaf tables labeled with an even number, each leaf table dependent on a root table.
2. The method of claim 1 , wherein the data exchange resides on an Oracle Exadata box.
3. The method of claim 1 , wherein the modeling the raw data records comprises removing duplicate records from the raw data records.
4. The method of claim 1 , wherein the modeling the raw data records comprises reformatting the raw data records from ASCII format to a format consumable by the data exchange.
5. The method of claim 1 , wherein the hybrid job processing reduces a processing time of more than half a billion records from over 27 hours to under 5 hours.
6. The method of claim 1 , wherein the hybrid job processing reduces a processing time of the raw data records by over 70%.
7. A method for hybrid job processing, the hybrid job processing comprising modeling raw data from a raw data layer to modeled data in a model data layer, the method comprising:
receiving raw data records, said raw data records stored within a plurality of tables from a plurality of systems of record, at the raw data layer within a data exchange;
generating, based on a data model from the raw data records, the data model comprising one or more dependencies within the plurality of tables, wherein each table in a second subset of the plurality of tables is dependent on at least one table in a first subset of the plurality of tables;
processing, at a processor:
the first subset of tables using parallel processing; and
the second subset of tables using serial processing, wherein processing is initiated for each table in the second subset when a table with an associated dependency is complete;
said processing comprising:
modeling the raw data records stored in the second subset of the plurality of tables at the raw data layer to form modeled records; and
transmitting the modeled data records in the first subset of the plurality of tables to the model data layer; and
wherein:
each of the tables in the plurality of tables is labeled based on a sequence of numbers; and
the first subset of tables comprises root tables labeled with an odd number and the second subset of tables comprises leaf tables labeled with an even number, each leaf table dependent on a root table.
8. The method of claim 7 , wherein the data exchange resides on an Oracle Exadata box.
9. The method of claim 7 , wherein the modeling the raw data records comprises removing duplicate records from the raw data records.
10. The method of claim 7 , wherein the modeling the raw data records comprises reformatting the raw data records from ASCII format to a format consumable by the data exchange.
11. The method of claim 7 , wherein the hybrid job processing reduces a processing time of more than half a billion records from over 27 hours to under 5 hours.
12. The method of claim 7 , wherein the hybrid job processing reduces a processing time of the raw data records by over 70%.