Data engine integration and data refinement
View Patent ↗Data engine integration and data refinement are described. The actions include receiving, by an in-memory engine, a request for data. The actions include transmitting, by the in-memory engine and to a metadata storage unit, data that is associated with the request. The actions include receiving metadata. The actions include determining a first amount of processing to be performed by the in-memory engine and a second amount of processing to be performed by a data engine. The actions include transmitting a request (i) for a first portion of unprocessed data and (ii) to perform a second amount of processing on a second portion of unprocessed data. The actions include receiving the first portion of unprocessed data and the second portion of processed data. The actions include processing the first portion of unprocessed data. The actions include, providing the first portion of processed data and the second portion of processed data.
1. A computer-implemented method comprising:
receiving, by an in-memory engine, a data request;
transmitting, by the in-memory engine and to a metadata storage unit, data that is associated with the request;
receiving, by the in-memory engine and from the metadata storage unit, metadata;
based on the metadata, generating a query tree that includes query tasks for processing;
determining, by the in-memory engine, a first portion of the data to be processed by the in-memory engine according to a first group of the query tasks and a second portion of the data to be processed by a data engine according to a second group of the query tasks to balance a processing load of the in-memory engine and a processing load of the data engine;
transmitting, by the in-memory engine and to the data engine, a request (i) for the first portion of the data and (ii) to process the second portion of the data according to the second group of the query tasks;
receiving, by the in-memory engine and from the data engine, the first portion of the data and the second portion of the data that is processed by the data engine;
processing, by the in-memory engine, the first portion of the data according to the first group of the query tasks; and
in response to the data request, providing, by the in-memory engine, the first portion of the data that is processed by the in-memory engine and the second portion of the data that is processed by the data engine.
2. The method of claim 1 , wherein processing performed by the in-memory engine and processing performed by the data engine comprises:
filtering, aggregation, wrangling, searching, data mining, text analytics, on demand loading, incremental refreshing, streaming, data blending, complex ETL workflows, or multi-sourcing.
3. The method of claim 1 , wherein the in-memory engine receives the data request from a dashboard application.
4. The method of claim 1 , comprising:
after transmitting, by the in-memory engine and to the data engine, a request (i) for the first portion of the data and (ii) to process the second portion of the data according to the second group of the query tasks, receiving, by the in-memory engine and from the data engine, heartbeats and execution status updates.
5. The method of claim 1 , wherein transmitting, by the in-memory engine and to the data engine, a request (i) for the first portion of the data and (ii) to process the second portion of the data according to the second group of the query tasks comprises:
identifying a processing node of the data engine that stores the second portion of the data; and
transmitting, to the processing node of the data engine, the request to process the second portion of the data according to the second groups of the query tasks.
6. A system comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
receiving, by an in-memory engine, a data request;
transmitting, by the in-memory engine and to a metadata storage unit, data that is associated with the request;
receiving, by the in-memory engine and from the metadata storage unit, metadata;
based on the metadata, generating a query tree that includes query tasks for processing;
determining, by the in-memory engine, a first portion of the data to be processed by the in-memory engine according to a first group of the query tasks and a second portion of the data to be processed by a data engine according to a second group of the query tasks to balance a processing load of the in-memory engine and a processing load of the data engine;
transmitting, by the in-memory engine and to the data engine, a request (i) for the first portion of the data and (ii) to process the second portion of the data according to the second group of the query tasks;
receiving, by the in-memory engine and from the data engine, the first portion of the data and the second portion of the data that is processed by the data engine;
processing, by the in-memory engine, the first portion of the data according to the first group of the query tasks; and
in response to the data request, providing, by the in-memory engine, the first portion of the data that is processed by the in-memory engine and the second portion of the data that is processed by the data engine.
7. The system of claim 6 , wherein processing performed by the in-memory engine and processing performed by the data engine comprises:
filtering, aggregation, wrangling, searching, data mining, text analytics, on demand loading, incremental refreshing, streaming, data blending, complex ETL workflows, or multi-sourcing.
8. The system of claim 6 , wherein the in-memory engine receives the data request from the dashboard a dashboard application.
9. The system of claim 6 , wherein the operations further comprise:
after transmitting, by the in-memory engine and to the data engine, a request (i) for the first portion of the data and (ii) to process the second portion of the data according to the second group of the query tasks, receiving, by the in-memory engine and from the data engine, heartbeats and execution status updates.
10. The system of claim 6 , wherein transmitting, by the in-memory engine and to the data engine, a request (i) for the first portion of the data and (ii) to process the second portion of the data according to the second group of the query tasks comprises:
identifying a processing node of the data engine that stores the second portion of the data; and
transmitting, to the processing node of the data engine, the request to process the second portion of the data according to the second groups of the query tasks.
11. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:
receiving, by an in-memory engine, a data request;
transmitting, by the in-memory engine and to a metadata storage unit, data that is associated with the request;
receiving, by the in-memory engine and from the metadata storage unit, metadata;
based on the metadata, generating a query tree that includes query tasks for processing;
determining, by the in-memory engine, a first portion of the data to be processed by the in-memory engine according to a first group of the query tasks and a second portion of the data to be processed by a data engine according to a second group of the query tasks to balance a processing load of the in-memory engine and a processing load of the data engine;
transmitting, by the in-memory engine and to the data engine, a request (i) for the first portion of the data and (ii) to process the second portion of the data according to the second group of the query tasks;
receiving, by the in-memory engine and from the data engine, the first portion of the data and the second portion of the data that is processed by the data engine;
processing, by the in-memory engine, the first portion of the data according to the first group of the query tasks; and
in response to the data request, providing, by the in-memory engine, the first portion of the data that is processed by the in-memory engine and the second portion of the data that is processed by the data engine.
12. The medium of claim 11 , wherein processing performed by the in-memory engine and processing performed by the data engine comprises:
filtering, aggregation, wrangling, searching, data mining, text analytics, on demand loading, incremental refreshing, streaming, data blending, complex ETL workflows, or multi-sourcing.
13. The medium of claim 11 , wherein the operations further comprise:
after transmitting, by the in-memory engine and to the data engine, a request (i) for the first portion of the data and (ii) to process the second portion of the data according to the second group of the query tasks, receiving, by the in-memory engine and from the data engine, heartbeats and execution status updates.
14. The medium of claim 11 , wherein transmitting, by the in-memory engine and to the data engine, a request (i) for the first portion of the data and (ii) to process the second portion of the data according to the second group of the query tasks comprises:
identifying a processing node of the data engine that stores the second portion of the data; and
transmitting, to the processing node of the data engine, the request to process the second portion of the data according to the second groups of the query tasks.