Systems and methods for data storage and processing
Systems and methods for processing data are provided. The system may include at least a processor and a non-transient data memory storage, the data memory storage containing machine-readable instructions for execution by the processor, the machine-readable instructions configured to, when executed by the processor, provide an information delivery platform configured to: extract raw data from a plurality of source systems; load and store the raw data at a non-transient data store; receive a request to generate data for consumption for a specific purpose; In response to the request, select a set of data from the raw data based on a data map; transform the selected set of data into a curated set of data based on the data map; and transmit the curated set of data to a channel for consumption.
1. A computer-implemented method comprising:
receiving, by a processor, a request from a second processor for retrieval of data, the request comprising a requested data format;
in response to the request,
executing, by the processor using raw data stored within a database, a data model configured to identify a portion of the raw data corresponding to the request and further identify code to transform the portion of the raw data;
transforming, by the processor, the portion of raw data by applying the code to the portion of the raw data identified via the data model to obtain transformed data comprising the portion of raw data in the requested data format; and
transmitting, by the processor, the transformed data to the second processor.
2. The method of claim 1 , wherein the requested data format corresponds to a group of users accessing an application associated with the request.
3. The method of claim 1 , further comprising:
extracting, by the processor, the raw data from a plurality of source systems; and
loading and storing, by the processor, the raw data in the database.
4. The method of claim 1 , wherein the raw data are stored at the database in a data format that is identical to a source data format of the raw data in a plurality of source systems.
5. The method of claim 1 , wherein the processor generates, using the data model, a data map comprising a graph linking one or more data columns of the raw data to one or more data fields of the transformed data.
6. The method of claim 1 , wherein the data model is a machine learning model.
7. The method of claim 1 , wherein the database is distributed across a network of different nodes.
8. The method of claim 1 , further comprising:
aggregating, by the processor, the portion of the raw data in accordance with the data model.
9. A system comprising at least a processor and a non-transient data memory storage, the non-transient data memory storage containing machine-readable instructions for execution by the processor, the machine-readable instructions configured to cause the processor to:
receive a request from a second processor for retrieval of data, the request comprising a requested data format;
in response to the request,
execute using raw data stored within a database, a data model configured to identify a portion of the raw data corresponding to the request and further identify code to transform the portion of the raw data;
transform the portion of raw data by applying the code to the portion of the raw data identified via the data model to obtain transformed data comprising the portion of raw data in the requested data format; and
transmit the transformed data to the second processor.
10. The system of claim 9 , wherein the requested data format corresponds to a group of users accessing an application associated with the request.
11. The system of claim 9 , wherein the instructions further cause the processor to:
extract the raw data from a plurality of source systems; and
load and store the raw data in the database.
12. The system of claim 9 , wherein the raw data are stored at the database in a data format that is identical to a source data format of the raw data in a plurality of source systems.
13. The system of claim 9 , wherein the processor generates, using the data model, a data map comprising a graph linking one or more data columns of the raw data to one or more data fields of the transformed data.
14. The system of claim 9 , wherein the data model is a machine learning model.
15. The system of claim 9 , wherein the database is distributed across a network of different nodes.
16. The system of claim 9 , wherein the instructions further cause the processor to:
aggregate the portion of the raw data in accordance with the data model.
17. A computer system comprising:
a database storing raw data;
a processor in communication with the database, the processor configured to:
receive a request from a second processor for retrieval of data, the request comprising a requested data format;
in response to the request,
execute using the raw data stored within the database, a data model configured to identify a portion of the raw data corresponding to the request and further identify code to transform the portion of the raw data;
transform the portion of raw data by applying the code to the portion of the raw data identified via the data model to obtain transformed data comprising the portion of raw data in the requested data format; and
transmit the transformed data to the second processor.
18. The system of claim 17 , wherein the requested data format corresponds to a group of users accessing an application associated with the request.
19. The system of claim 17 , wherein the raw data are stored at the database in a data format that is identical to a source data format of the raw data in a plurality of source systems.
20. The system of claim 17 , wherein the data model is a machine learning model.