System and method for providing a consolidated data hub
The present disclosure is directed to a system for data consolidation. The system may include processors, servers, and/or storage devices. Processors in the system may be configurable to perform operations like importing data from, transforming the imported data into a plurality of tables, identifying tables comprising outlier attributes, and modifying the identified tables by normalizing or deleting corresponding attributes, Operations of the disclosed systems may also include performing a conformity check on the integration tables, generating two or more data structures arranging tables based on downstream modeling requirements, storing the two or more data structures in the single storage location, and provisioning the one or more data structures for downstream modeling.
1 . A system for data consolidation comprising:
one or more processors; and
one or more storage devices storing instructions that, when executed, configure the one or more processors to perform operations comprising:
importing data from a plurality of sources to a single storage location through at least one iterative import job;
tokenizing one or more elements in the imported data based on a sensitivity of the one or more elements;
transforming the imported data and the tokenized one or more elements into a plurality of integration tables, the plurality of integration tables having an indexing key and an attribute;
identifying integration tables comprising outlier attributes;
modifying the identified integration tables by normalizing or deleting corresponding attributes;
after modifying the identified integration tables, performing a conformity check on the plurality of integration tables by executing a conformity job, the conformity job comprising a script that compares the plurality of integration tables to a control table comprising control data to ensure data completeness and adjusts attributes in the plurality of integration tables based on values in the control table with a matching indexing key;
generating two or more data structures arranging at least a portion of the plurality of integration tables based on downstream modeling requirements,
wherein generating the two or more data structures includes maintaining a change log storing changes to the plurality of integration tables, and
wherein generating the two or more data structures comprises:
receiving the downstream modeling requirements from a downstream user, the downstream modeling requirements specifying a life cycle event that includes an application stage; and
after transforming the imported data and the tokenized one or more elements into the plurality of integration tables, filtering the plurality of integration tables based on the life cycle event to extract data corresponding to the application stage for arranging into at least one of the two or more data structures;
storing the two or more data structures in the single storage location;
provisioning the two or more data structures for downstream modeling;
receiving a downstream data model trained on at least one of the two or more data structures;
determining, by monitoring the change log, that at least one of the plurality of integration tables was modified; and
in response to determining at least one of the plurality of integration tables was modified, triggering a retraining of the data model using modified integration tables.
2 . The system of claim 1 , wherein generating the two or more data structures further includes exposing the change log to an application programming interface accessible to users for retrieving the two or more data structures.
3 . The system of claim 1 , wherein:
the two or more data structures are distinct from each other and are tailored for specific downstream requirements; and
the conformity job comprises: determining irregularities in the attributes including using the control table to identify data outside a range, implementing a code change, reloading data to impacted attributes, and manipulating the control table to enforce data quality standards.
4 . The system of claim 3 , wherein the data in the two or more data structures is the same.
5 . The system of claim 1 , wherein the operations further comprise exposing the single storage location to an application programing interface accessible to users for retrieving the two or more data structures, wherein tokenizing the one or more elements in the imported data based on a sensitivity of the one or more elements includes hiding or deleting data.
6 . The system of claim 1 , wherein the operations further comprise:
generating a data dashboard configured to display results of the conformity job, the data dashboard including filtering options for asset class domain and options for lifecycle domain; and
transmitting instructions to display the data dashboard to a user.
7 . The system of claim 6 , wherein the data dashboard includes a control button configured to trigger displays of live data in the data dashboard.
8 . The system of claim 1 , wherein
transforming the imported data comprises creating an incremental dataset by comparing sources with target dates to eliminate outdated sources;
the integration tables comprise profiling tables and conformity tables, wherein the profiling tables store asset class attributes, and wherein conformity tables store data type attributes; and
provisioning the one or more data structures for downstream modeling comprises generating persistent tables and exposing them to application programming interfaces accessible to downstream users.
9 . A computer-implemented method for data consolidation comprising:
importing data from a plurality of sources to a single storage location through at least one iterative import job;
tokenizing one or more elements in the imported data based on a sensitivity of the one or more elements;
transforming the imported data and the tokenized one or more elements into a plurality of integration tables, the plurality of integration tables having an indexing key and an attribute;
identifying integration tables comprising outlier attributes;
modifying the identified integration tables by normalizing or deleting corresponding attributes;
after modifying the identified integration tables, performing a conformity check on the plurality of integration tables by executing a conformity job, the conformity job comprising a script that compares the plurality of integration tables to a control table comprising control data to ensure data completeness and adjusts attributes in the plurality of integration tables based on values in the control table with a matching indexing key;
generating two or more data structures arranging at least a portion of the plurality of integration tables based on downstream modeling requirements,
wherein generating the two or more data structures includes maintaining a change log storing changes to the plurality of integration tables, and
wherein generating the two or more data structures comprises:
receiving the downstream modeling requirements from a downstream user, the downstream modeling requirements specifying a life cycle event that includes an application stage; and
after transforming the imported data and the tokenized one or more elements into the plurality of integration tables, filtering the plurality of integration tables based on the life cycle event to extract data corresponding to the application stage for arranging into at least one of the two or more data structures;
storing the two or more data structures in the single storage location;
provisioning the two or more data structures for downstream modeling;
receiving a downstream data model trained on at least one of the two or more data structures;
determining, by monitoring the change log, that at least one of the plurality of integration tables was modified; and
in response to determining at least one of the plurality of integration tables was modified, triggering a retraining of the data model using modified integration tables.
10 . The method of claim 9 , wherein:
generating the two or more data structures further includes exposing the change log to an application programming interface accessible to users for retrieving the two or more data structures.
11 . The method of claim 9 , wherein:
the two or more data structures are distinct from each other and are tailored for specific downstream requirements; and
the conformity job comprises: determining irregularities in the attributes including using the control table to identify data outside a range, implementing a code change, reloading data to impacted attributes, and manipulating the control table to enforce data quality standards.
12 . The method of claim 11 , wherein the data in the two or more data structures is the same.
13 . The method of claim 9 , further comprising exposing the single storage location to an application programing interface accessible to users for retrieving the two or more data structures, wherein tokenizing the one or more elements in the imported data based on a sensitivity of the one or more elements includes hiding or deleting data.
14 . The method of claim 9 , further comprising:
generating a data dashboard configured to display results of the conformity job, the data dashboard including filtering options for asset class domain and options for lifecycle domain; and
transmitting instructions to display the data dashboard to a user.
15 . The method of claim 14 , wherein the data dashboard includes a control button configured to trigger displays of live data in the data dashboard.
16 . The method of claim 9 , wherein
transforming the imported data comprises creating an incremental dataset by comparing sources with target dates to eliminate outdated sources;
the integration tables comprise profiling tables and conformity tables, the profiling tables storing asset class attributes and conformity tables storing data type attributes; and
provisioning the one or more data structures for downstream modeling comprises generating persistent tables and exposing them to application programming interfaces accessible to downstream users.
17 . A server comprising:
at least one processor;
a storage location connected to the at least one processor; and
a remote access card connected to the at least one processor and the storage location,
wherein the at least one processor is configured to:
import data from a plurality of sources to the storage location by connecting to the plurality of data sources through the remote access card and implementing a plurality of import jobs;
tokenize one or more elements in the imported data based on a sensitivity of the one or more elements;
transform the imported data and the tokenized one or more elements into a plurality of tables, the plurality of tables having an indexing key and an attribute;
identify tables comprising outlier attributes;
modify the identified tables by normalizing or deleting corresponding attributes;
perform a conformity check on the tables by executing a conformity job, the conformity job comprising a script that compares the plurality of tables to a control table comprising control data to ensure data completeness and adjusts attributes in the plurality of integration tables based on values in the control table with matching indexing key;
generate two or more data structures arranging at least a portion of the plurality of tables based on downstream modeling requirements,
wherein generating the two or more data structures includes maintaining a change log storing changes to the plurality of integration tables, and
wherein generating the two or more data structures comprises:
receiving the downstream modeling requirements from a downstream user, the downstream modeling requirements specifying a life cycle event that includes an application stage; and
after transforming the imported data and the tokenized one or more elements into the plurality of integration tables, filtering the plurality of integration tables based on the life cycle event to extract data corresponding to the application stage for arranging into at least one of the two or more data structures;
store the two or more data structures in the storage location;
expose the one or more data structures for downstream modeling;
receiving a downstream data model trained on at least one of the two or more data structures;
determining, by monitoring the change log, that at least one of the plurality of integration tables was modified; and
in response to determining at least one of the plurality of integration tables was modified, triggering a retraining of the data model using modified integration tables.