Data engine integration and data refinement

1. A computer-implemented method comprising:

receiving, by a data refinement engine, a request for processed data;

determining, by the data refinement engine, a data source for unprocessed data and processing required to transform the unprocessed data into the processed data, the data source including a plurality of data nodes that are separate from the data refinement engine;

determining, by the data refinement engine, that both the plurality of data nodes and the data refinement engine are configured to perform processing required to transform the unprocessed data into the processed data;

determining, by the data refinement engine, (i) a portion of the processing for the plurality of data nodes to perform to transform a portion of the unprocessed data into a portion of the processed data and (ii) a remaining portion of the processing for the data refinement engine to perform to transform a remaining portion of the unprocessed data into a remaining portion of the processed data, wherein the respective portions of processing for the data refinement engine and the plurality of data nodes are determined to balance a processing load of the data refinement engine with a processing load of the plurality of data nodes;

based on determining (i) the portion of the processing for the plurality of data nodes to perform to transform the portion of the unprocessed data into the portion of the processed data and (ii) the remaining portion of the processing for the data refinement engine to perform to transform the remaining portion of the unprocessed data into the remaining portion of the processed data, transmitting, by the data refinement engine and to the plurality of data nodes, code including instructions (i) to perform the portion of the processing required to transform the portion of the unprocessed data into the portion of the processed data and (ii) to provide, to the data refinement engine, the remaining portion of the unprocessed data;

receiving, by the data refinement engine and from the plurality of data nodes, the portion of the processed data and the remaining portion of the unprocessed data;

transforming, by the data refinement engine, the remaining portion of the unprocessed data into the remaining portion of the processed data by performing the remaining portion of the processing on the remaining portion of the unprocessed data; and

transmitting, by the data refinement engine, the portion of the processed data received from the plurality of data nodes and the remaining portion of the processed data processed by the data refinement engine.

2. The method of claim 1 , wherein processing required to transform the unprocessed data into the processed data comprises:

filtering, aggregation, wrangling, searching, data mining, text analytics, on demand loading, incremental refreshing, streaming, data blending, complex ETL workflows, or multi-sourcing.

3. The method of claim 1 , wherein the data refinement engine receives the request for processed data request from a dashboard application.

4. The method of claim 1 , comprising:

generating a query tree that includes query tasks for processing; and

determining that the portion of the processing required to transform the portion of the unprocessed data into the portion of the processed data includes a first portion of the query tasks and the remaining portion of the processing required to transform the remaining portion of the unprocessed data into the remaining portion of the processed data includes a second portion of the query tasks,

wherein transmitting, by the data refinement engine and to the plurality of data nodes, code including instructions (i) to perform the portion of the processing required to transform the portion of the unprocessed data into the portion of the processed data and (ii) to provide, to the data refinement engine, the remaining portion of the unprocessed data is further based on determining that the portion of the processing required to transform the portion of the unprocessed data into the portion of the processed data includes the first portion of the query tasks and the remaining portion of the processing required to transform the remaining portion of the unprocessed data into the remaining portion of the processed data includes the second portion of the query tasks.

5. The method of claim 1 , comprising:

after transmitting, by the data refinement engine and to the plurality of data nodes, the code including the instructions (i) to perform the portion of the processing required to transform the portion of the unprocessed data into the portion of the processed data and (ii) to provide, to the data refinement engine, the remaining portion of the unprocessed data, receiving, by the data refinement engine and from the plurality of data nodes, heartbeats and execution status updates.

6. The method of claim 1 , wherein transmitting, by the data refinement engine and to the plurality of data nodes, the code including the instructions (i) to perform the portion of the processing required to transform the portion of the unprocessed data into the portion of the processed data and (ii) to provide, to the data refinement engine, the remaining portion of the unprocessed data comprises:

identifying a particular data node of the plurality of data nodes that stores a sub-portion of the portion of the unprocessed data; and

transmitting, to the particular data node of the plurality of data nodes, code to perform a sub-portion of processing required to transform the sub-portion of the portion of the unprocessed data into a sub-portion of the portion of processed data.

7. The method of claim 1 , comprising:

identifying, by the data refinement engine, a group of the plurality of nodes that stores copies of a same sub-portion of the portion of the unprocessed data;

determining, by the data refinement engine, a processing load of each node in the group of the plurality of nodes; and

for each node in the group of the plurality of nodes, determining, by the data refinement engine, a sub-portion of the portion of the processing required to transform the portion of the unprocessed data into the portion of the processed data to balance processing loads of each node in the group of the plurality of nodes;

wherein transmitting, by the data refinement engine and to the plurality of data nodes, code including instructions (i) to perform the portion of the processing required to transform the portion of the unprocessed data into the portion of the processed data and (ii) to provide, to the data refinement engine, the remaining portion of the unprocessed data comprises:

for each node in the group of the plurality of nodes, transmitting, by the data refinement engine and to a respective node in the group of the plurality of data nodes, code including instructions to perform a respective sub-portion of the portion of the processing required to transform the portion of the unprocessed data into the portion of the processed data.

8. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

receiving, by a data refinement engine, a request for processed data;

determining, by the data refinement engine, a data source for unprocessed data and processing required to transform the unprocessed data into the processed data, the data source including a plurality of data nodes that are separate from the data refinement engine;

determining, by the data refinement engine, that both the plurality of data nodes and the data refinement engine are configured to perform processing required to transform the unprocessed data into the processed data;

determining, by the data refinement engine, (i) a portion of the processing for the plurality of data nodes to perform to transform a portion of the unprocessed data into a portion of the processed data and (ii) a remaining portion of the processing for the data refinement engine to perform to transform a remaining portion of the unprocessed data into a remaining portion of the processed data, wherein the respective portions of processing for the data refinement engine and the plurality of data nodes are determined to balance a processing load of the data refinement engine with a processing load of the plurality of data nodes;

based on determining (i) the portion of the processing for the plurality of data nodes to perform to transform the portion of the unprocessed data into the portion of the processed data and (ii) the remaining portion of the processing for the data refinement engine to perform to transform the remaining portion of the unprocessed data into the remaining portion of the processed data, transmitting, by the data refinement engine and to the plurality of data nodes, code including instructions (i) to perform the portion of the processing required to transform the portion of the unprocessed data into the portion of the processed data and (ii) to provide, to the data refinement engine, the remaining portion of the unprocessed data;

receiving, by the data refinement engine and from the plurality of data nodes, the portion of the processed data and the remaining portion of the unprocessed data;

transforming, by the data refinement engine, the remaining portion of the unprocessed data into the remaining portion of the processed data by performing the remaining portion of the processing on the remaining portion of the unprocessed data; and

transmitting, by the data refinement engine, the portion of the processed data received from the plurality of data nodes and the remaining portion of the processed data processed by the data refinement engine.

9. The system of claim 8 , wherein processing required to transform the unprocessed data into the processed data comprises:

filtering, aggregation, wrangling, searching, data mining, text analytics, on demand loading, incremental refreshing, streaming, data blending, complex ETL workflows, or multi-sourcing.

10. The system of claim 8 , wherein the data refinement engine receives the request for processed data request from a dashboard application.

11. The system of claim 8 , wherein the operations further comprise:

generating a query tree that includes query tasks for processing; and

determining that the portion of the processing required to transform the portion of the unprocessed data into the portion of the processed data includes a first portion of the query tasks and the remaining portion of the processing required to transform the remaining portion of the unprocessed data into the remaining portion of the processed data includes a second portion of the query tasks,

wherein transmitting, by the data refinement engine and to the plurality of data nodes, code including instructions (i) to perform the portion of the processing required to transform the portion of the unprocessed data into the portion of the processed data and (ii) to provide, to the data refinement engine, the remaining portion of the unprocessed data is further based on determining that the portion of the processing required to transform the portion of the unprocessed data into the portion of the processed data includes the first portion of the query tasks and the remaining portion of the processing required to transform the remaining portion of the unprocessed data into the remaining portion of the processed data includes the second portion of the query tasks.

12. The system of claim 8 , wherein the operations further comprise:

after transmitting, by the data refinement engine and to the plurality of data nodes, the code including the instructions (i) to perform the portion of the processing required to transform the portion of the unprocessed data into the portion of the processed data and (ii) to provide, to the data refinement engine, the remaining portion of the unprocessed data, receiving, by the data refinement engine and from the plurality of data nodes, heartbeats and execution status updates.

13. The system of claim 8 , wherein transmitting, by the data refinement engine and to the plurality of data nodes, the code including the instructions (i) to perform the portion of the processing required to transform the portion of the unprocessed data into the portion of the processed data and (ii) to provide, to the data refinement engine, the remaining portion of the unprocessed data comprises:

identifying a particular data node of the plurality of data nodes that stores a sub-portion of the portion of the unprocessed data; and

transmitting, to the particular data node of the plurality of data nodes, code to perform a sub-portion of processing required to transform the sub-portion of the portion of the unprocessed data into a sub-portion of the portion of processed data.

14. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

receiving, by a data refinement engine, a request for processed data;

determining, by the data refinement engine, a data source for unprocessed data and processing required to transform the unprocessed data into the processed data, the data source including a plurality of data nodes that are separate from the data refinement engine;

determining, by the data refinement engine, that both the plurality of data nodes and the data refinement engine are configured to perform processing required to transform the unprocessed data into the processed data;

determining, by the data refinement engine, (i) a portion of the processing for the plurality of data nodes to perform to transform a portion of the unprocessed data into a portion of the processed data and (ii) a remaining portion of the processing for the data refinement engine to perform to transform a remaining portion of the unprocessed data into a remaining portion of the processed data, wherein the respective portions of processing for the data refinement engine and the plurality of data nodes are determined to balance a processing load of the data refinement engine with a processing load of the plurality of data nodes;

based on determining (i) the portion of the processing for the plurality of data nodes to perform to transform the portion of the unprocessed data into the portion of the processed data and (ii) the remaining portion of the processing for the data refinement engine to perform to transform the remaining portion of the unprocessed data into the remaining portion of the processed data, transmitting, by the data refinement engine and to the plurality of data nodes, code including instructions (i) to perform the portion of the processing required to transform the portion of the unprocessed data into the portion of the processed data and (ii) to provide, to the data refinement engine, the remaining portion of the unprocessed data;

receiving, by the data refinement engine and from the plurality of data nodes, the portion of the processed data and the remaining portion of the unprocessed data;

transforming, by the data refinement engine, the remaining portion of the unprocessed data into the remaining portion of the processed data by performing the remaining portion of the processing on the remaining portion of the unprocessed data; and

transmitting, by the data refinement engine, the portion of the processed data received from the plurality of data nodes and the remaining portion of the processed data processed by the data refinement engine.

15. The non-transitory computer-readable medium of claim 14 , wherein processing required to transform the unprocessed data into the processed data comprises:

filtering, aggregation, wrangling, searching, data mining, text analytics, on demand loading, incremental refreshing, streaming, data blending, complex ETL workflows, or multi-sourcing.

16. The non-transitory computer-readable medium of claim 14 , wherein the operations further comprise:

generating a query tree that includes query tasks for processing; and

determining that the portion of the processing required to transform the portion of the unprocessed data into the portion of the processed data includes a first portion of the query tasks and the remaining portion of the processing required to transform the remaining portion of the unprocessed data into the remaining portion of the processed data includes a second portion of the query tasks,

wherein transmitting, by the data refinement engine and to the plurality of data nodes, code including instructions (i) to perform the portion of the processing required to transform the portion of the unprocessed data into the portion of the processed data and (ii) to provide, to the data refinement engine, the remaining portion of the unprocessed data is further based on determining that the portion of the processing required to transform the portion of the unprocessed data into the portion of the processed data includes the first portion of the query tasks and the remaining portion of the processing required to transform the remaining portion of the unprocessed data into the remaining portion of the processed data includes the second portion of the query tasks.

17. The non-transitory computer-readable medium of claim 14 , wherein the operations further comprise:

after transmitting, by the data refinement engine and to the plurality of data nodes, the code including the instructions (i) to perform the portion of the processing required to transform the portion of the unprocessed data into the portion of the processed data and (ii) to provide, to the data refinement engine, the remaining portion of the unprocessed data, receiving, by the data refinement engine and from the plurality of data nodes, heartbeats and execution status updates.

18. The non-transitory computer-readable medium of claim 14 , wherein transmitting, by the data refinement engine and to the plurality of data nodes, the code including the instructions (i) to perform the portion of the processing required to transform the portion of the unprocessed data into the portion of the processed data and (ii) to provide, to the data refinement engine, the remaining portion of the unprocessed data comprises:

identifying a particular data node of the plurality of data nodes that stores a sub-portion of the portion of the unprocessed data; and

transmitting, to the particular data node of the plurality of data nodes, code to perform a sub-portion of processing required to transform the sub-portion of the portion of the unprocessed data into a sub-portion of the portion of processed data.