Systems, methods, and apparatuses for combining, interpreting, and distributing non-uniform datasets into structured datasets

1 . A system for a distributed execution engine for combining, interpreting, and distributing non-uniform datasets into structured datasets, the system comprising:

a memory device with computer-readable program code stored thereon;

a communication device;

a processing device operatively coupled to the memory device and the communication device, wherein the processing device is configured to execute the computer-readable program code to:

identify at least one configuration file, the at least one configuration file comprising at least one location associated with one or more datasets or at least one script file comprising at least one instruction;

identify the at least one script file based on the at least one location associated with the at least one configuration file;

retrieve the one or more datasets based on the at least one location associated with the at least one configuration file, wherein the one or more datasets comprise one or more flexible schemas organized based on data available in the one or more datasets;

identify, by an execution engine, at least one instruction from the at least one script file, wherein the at least one instruction is written in a data centric language;

build, by the execution engine and using the at least one instruction, a tree of execution based on the one or more datasets and the data available in the one or more datasets, wherein the tree of execution comprises a plurality of nodes, and wherein the plurality of nodes comprises the at least one instruction corresponding to the data available and the one or more flexible schemas, and wherein each node comprises a key value pair corresponding to the data available in each node;

execute the at least one instruction at each node based on the key value pair on the one or more datasets in the tree of execution, wherein the key value pair indicates at least one present attribute and at least one present value from the data available at each node, wherein the execution is performed in parallel for at least two or more nodes in the tree of execution and using at least one computational cluster comprising at least one computing component at each node and wherein the two or more nodes work together to process data from the one or more datasets using sub-tasks from the at least one script file;

and generate an output comprising a structured dataset from the one or more datasets and based on the execution of the at least one instruction using the tree of execution, wherein the structured dataset comprises a standardized format of the one or more datasets based on the key value pair for each dataset of the one or more datasets.

2 . The system of claim 1 , wherein building the tree of execution further comprises:

generating at least one checkpoint, and wherein the at least one checkpoint comprises a periodic storage of at least one intermediate result before the output of the structured dataset of the execution of the at least one instruction for subsequent retrieval.

3 . The system of claim 1 , wherein the data centric language comprises at least one extensible language feature, wherein the at least one extensible language feature provides a conversion between the at least one instruction and building the tree of execution.

4 . The system of claim 1 , wherein the processing device is configured to execute the computer-readable program code to execute a linear series of operations to transform input, wherein the linear series of operations comprises:

identifying at least one element associated with one or more datasets, wherein the at least one element comprises at least one characteristic;

arranging the at least one element associated with the one or more datasets based on the at least one characteristic;

identifying at least one pattern in the one or more datasets based on the at least one characteristic;

deleting at least one element based on an inconsistency with the identified at least one pattern, wherein the inconsistency is based on a matching of the at least one element and the at least one pattern;

inserting at least one new element in the one or more datasets based on at least one missing element; and

merging data from one or more datasets based on a completeness or a relatedness of the one or more datasets wherein the one or more datasets share at least one element with at least one matching characteristic.

5 . The system of claim 1 , wherein the processing device is configured to execute the computer-readable program code to run a set of rules to ensure the validity of the scripts, wherein the running of the set of rules comprise one or more of the following steps:

verifying that data in the one or more datasets has a correct data type;

verifying that data in the one or more datasets is logically consistent;

ensuring that at least one field used in the one or more datasets conform to one or more external formatting rules; and

verifying that two or more datasets are unique from one another based on the at least one element associated with each dataset.

6 . The system of claim 1 , wherein retrieval of the at least one data source further comprises defining each data source using at least one coordinate that comprises a system name, a dataset name, or a version.

7 . The system of claim 1 , wherein the script file comprises at least one schema for organizing the output and wherein the at least one schema comprises at least one attribute.

8 . The system of claim 1 , wherein the at least one schema further comprises a consistent structure for the output.

9 . The system of claim 1 , wherein the tree of execution comprises a checkpoint corresponding to each sub-task at each node, and wherein the checkpoint comprises a current state for the data within each node.

10 . The system of claim 1 , wherein the two or more nodes process data in parallel and simultaneously.

11 . A computer program product for a distributed execution engine for combing, interpreting, and distributing non-uniform datasets into structured datasets, the computer program product comprising a non-transitory computer-readable medium having computer-readable program code portions embodied therein, the computer-readable program code portions comprising:

an executable portion configured for identifying at least one configuration file, the at least one configuration file comprising at least one location associated with one or more datasets or at least one script file comprising at least one instruction;

an executable portion configured for identifying the at least one script file based on the at least one location associated with the at least one configuration file;

an executable portion configured for retrieving the one or more datasets based on the at least one location associated with the at least one configuration file, wherein the one or more datasets comprise one or more flexible schemas organized based on data available in the one or more datasets;

an executable portion configured for identifying, by an execution engine, at least one instruction from the at least one script file, wherein the at least one instruction is written in a data centric language;

an executable portion configured for building, by the execution engine and using the at least one instruction, a tree of execution based on the one or more datasets and the data available in the one or more datasets, wherein the tree of execution comprises a plurality of nodes, and wherein the plurality of nodes comprises the at least one instruction corresponding to the data available and the one or more flexible schemas, and wherein each node comprises a key value pair corresponding to the data available in each node;

an executable portion configured for executing the at least one instruction at each node based on the key value pair on the one or more datasets in the tree of execution, wherein the key value pair indicates at least one present attribute and at least one present value from the data available at each node, wherein the execution is performed in parallel for at least two or more nodes in the tree of execution and using at least one computational cluster comprising at least one computing component at each node and wherein the two or more nodes work together to process data from the one or more datasets using sub-tasks from the at least one script file;

and an executable portion configured for generating an output comprising a structured dataset from the one or more datasets and based on the execution of the at least one instruction using the tree of execution, wherein the structured dataset comprises a standardized format of the one or more datasets based on the key value pair for each dataset of the one or more datasets associated with the at least one instruction of the script file.

12 . The computer program product of claim 11 , wherein building the tree of execution further comprises generating at least one checkpoint, and wherein the at least one checkpoint comprises periodic storage of at least one intermediate result before the output of the structured dataset of the execution of the at least one instruction script for retrieval at a later time.

13 . The computer program product of claim 11 , wherein the data centric language comprises at least one extensible language feature, wherein the at least one extensible language feature provides a conversion between the at least one instruction and building the tree of execution.

14 . The computer program product of claim 11 , wherein the processing device is configured to execute the computer-readable program code to execute a linear series of operations to transform input, wherein the linear series of operations comprise one or more of the following steps:

identifying at least one element associated with one or more datasets, wherein the at least one element comprises at least one characteristic;

arranging the at least one element associated with the one or more datasets based on the at least one characteristic;

identifying at least one pattern in the one or more datasets based on the at least one characteristic;

deleting at least one element based on an inconsistency with the identified at least one pattern, wherein the inconsistency is based on a matching of the at least one element and the at least one pattern;

inserting at least one new element in the one or more datasets based on at least one missing element; and

merging data from one or more datasets based on a completeness or a relatedness of the one or more datasets wherein the one or more datasets share at least one element with at least one matching characteristic.

15 . The computer program product of claim 11 , wherein the processing device is configured to execute the computer-readable program code to run a set of rules to ensure the validity of the scripts, wherein the running of the set of rules comprise one or more of the following steps:

verifying that data in the one or more datasets has a correct data type;

verifying that data in the one or more datasets is logically consistent;

ensuring that at least one field used in the one or more datasets conform to one or more external formatting rules; and

verifying that two or more datasets are unique from one another based on the at least one element associated with each dataset.

16 . The computer program product of claim 11 , wherein retrieval of the at least one data source further comprises defining each data source using at least one coordinate that comprises a system name, a dataset name, or a version.

17 . A computer-implemented method for a distributed execution engine for combining, interpreting, and distributing non-uniform datasets into structured datasets, the computer-implemented method comprising:

identifying at least one configuration file, the at least one configuration file comprising at least one location associated with one or more datasets or at least one script file comprising at least one instruction;

identifying the at least one script file based on the at least one location associated with the at least one configuration file;

retrieving the one or more datasets based on the at least one location associated with the at least one configuration file, wherein the one or more datasets comprise one or more flexible schemas organized based on data available in the one or more datasets;

identifying, by an execution engine, at least one instruction from the at least one script file, wherein the at least one instruction is written in a data centric language;

building, by the execution engine and using the at least one instruction, a tree of execution based on the one or more datasets and the data available in the one or more datasets, wherein the tree of execution comprises a plurality of nodes, and wherein the plurality of nodes comprises the at least one instruction corresponding to the data available and the one or more flexible schemas, and wherein each node comprises a key value pair corresponding to the data available in each node;

executing the at least one instruction at each node based on the key value pair on the one or more datasets in the tree of execution, wherein the key value pair indicates at least one present attribute and at least one present value from the data available at each node, wherein the execution is performed in parallel for at least two or more nodes in the tree of execution and using at least one computational cluster comprising at least one computing component at each node and wherein the two or more nodes work together to process data from the one or more datasets using sub-tasks from the at least one script file;

and generating an output comprising a structured dataset from the one or more datasets and based on the execution of the at least one instruction using the tree of execution, wherein the structured dataset comprises a standardized format of the one or more datasets based on the key value pair for each dataset of the one or more datasets.

18 . The computer implemented method of claim 17 , wherein building the tree of execution further comprises generating at least one checkpoint, and wherein the at least one checkpoint comprises periodic storage of at least one intermediate result before the output of the structured dataset of the execution of the at least one instruction for subsequent retrieval.

19 . The computer implemented method of claim 17 , wherein the data centric language comprises at least one extensible language feature, wherein the at least one extensible language feature provides a conversion between the at least one instruction and building the tree of execution.