Data analytics platform over parallel databases and distributed file systems
Performing data analytics processing in the context of a large scale distributed system that includes a massively parallel processing (MPP) database and a distributed storage layer is disclosed. In various embodiments, a data analytics request is received. A plan is created to generate a response to the request. A corresponding portion of the plan is assigned to each of a plurality of distributed processing segments, including by invoking as indicated in the assignment one or more data analytical functions embedded in the processing segment.
1. A method, comprising:
embedding in each of a plurality of distributed processing segments a library or other shared object comprising one or more data analytical functions, wherein the library or other shared object is included in the processing segments as deployed;
receiving, by a master node, a data analysis request;
creating, by the master node, a plan to generate a response to the request;
selecting one or more of the plurality of distributed processing segments to process a corresponding portion of the plan, wherein selecting the one or more of the plurality of distributed processing segments to process the corresponding portion of the plan includes:
obtaining, by the master node, metadata associated with one or more portions of the plan to be performed by the one or more corresponding distributed processing segments of the plurality of distributed processing segments; and
embedding the metadata in an assignment communication to be sent to the one or more of the plurality of distributed processing segments, wherein the metadata indicates a location, within a distributed data storage layer, of data to be processed by the corresponding one or more distributed processing segments;
sending, by the master node, to each of the plurality of distributed processing segments for which a portion of the plan is to be processed, the corresponding portion of the plan to be processed by that corresponding segment and the metadata, wherein the metadata is used to locate or access a subset of data on which the segment is to perform an indicated processing, and wherein the metadata sent to each of the plurality of distributed processing segments for which the corresponding portion of the plan is assigned includes an identification of at least one data analytics function to be used to process the portion of the plan of the one or more data analytics functions that is embedded in each of the plurality of distributed processing segments;
receiving, from each of the plurality of distributed processing segments for which a portion of the plan is assigned, a corresponding result of processing the portion of the plan; and
generating, a master response to the data analysis request based at least in part on the corresponding result of processing the portion of the plan received from each of the plurality of distributed processing segments for which a portion of the plan is assigned.
2. The method of claim 1 , wherein the plurality of distributed processing segments correspondingly invoke one or more data analytical functions respectively embedded in the corresponding plurality of distributed processing segments.
3. The method of claim 2 , wherein the one or more data analytical functions that are invoked are included in an assignment of the plan to the plurality of distributed processing segments by the master node.
4. The method of claim 1 , wherein the metadata identifies a location data corresponding to the one or more portions of the plan and at least a part of one or more data analytic processing to be performed in connection with processing the corresponding one or more portions of the plan.
5. The method of claim 1 , wherein the data analysis request comprises one or more SQL statements.
6. The method of claim 1 , wherein the data analysis request comprises one or more SQL statements to compute one or more of the following: Logistic Regression, Multinomial Logistic Regression, K-means clustering, Association Rules based market basket analysis, and Latent Dirichlet based topic modeling.
7. The method of claim 1 , wherein the data analysis request is received at a master node of a large scale distributed system.
8. The method of claim 1 , wherein creating a plan to generate a response to the request includes creating a query plan, slicing the query plan into a plurality of slices, and identifying for each slice a group of processing segments to perform tasks comprising that slice of the query plan.
9. The method of claim 1 , wherein each of said distributed processing segments is configured to use the metadata to access said data to be processed by that segment.
10. The method of claim 1 , wherein the distributed data storage layer comprises data stored in an instance of a Hadoop Distributed File System (HDFS) and the metadata indicates a location within the HDFS of data to be processed by that segment.
11. The method of claim 1 , wherein the library or other shared object embodies the one or more data analytical functions in the form of one or more of the following: compiled C++ code, compiled Java, compiled Fortran, or other compiled code.
12. The method of claim 1 , wherein the plurality of distributed processing segments comprise a subset of parallel processing segments comprising a massively parallel processing (MPP) database system.
13. The method of claim 1 , wherein the embedded data analytics function includes a User-Defined function, a step function, or a final function of a User-Defined Aggregator.
14. The method of claim 1 , wherein the metadata sent to each of the plurality of distributed processing segments is sent in conjunction with the corresponding portion of the plan to be performed by that segment.
15. The method of claim 14 , wherein the metadata sent to each of the plurality of distributed processing segments is sent as part of the corresponding portion of the plan to be performed by that segment.
16. The method of claim 1 , wherein the metadata sent to each of the plurality of distributed processing segments for which a portion of the plan is to be processed is obtained from a central metadata store.
17. A system, comprising:
a communication interface; and
one or more processors coupled to the communication interface and configured to:
embed in each of a plurality of distributed processing segments a library or other shared object comprising one or more data analytical functions, wherein the library or other shared object is included in the processing segments as deployed;
receive a data analysis request;
create a plan to generate a response to the request;
select one or more of the plurality of distributed processing segments to process a corresponding portion of the plan, wherein to select the one or more of the plurality of distributed processing segments to process the corresponding portion of the plan includes:
obtain metadata associated with one or more portions of the plan to be performed by the one or more corresponding distributed processing segments of the plurality of distributed processing segments; and
embed the metadata in an assignment communication to be sent to the one or more of the plurality of distributed processing segments, wherein the metadata indicates a location, within a distributed data storage layer, of data to be processed by the corresponding one or more distributed processing segments;
send, to each of the plurality of distributed processing segments for which a portion of the plan is to be processed, the corresponding portion of the plan to be processed by that corresponding segment and the metadata, wherein the metadata is used to locate or access a subset of data on which the segment is to perform an indicated processing, and wherein the metadata sent to each of the plurality of distributed processing segments for which the corresponding portion of the plan is assigned includes an identification of at least one data analytics function to be used to process the portion of the plan of the one or more data analytics functions that is embedded in each of the plurality of distributed processing segments;
receive, from each of the plurality of distributed processing segments for which a portion of the plan is assigned, a corresponding result of processing the portion of the plan; and
generate, a master response to the data analysis request based at least in part on the corresponding result of processing the portion of the plan received from each of the plurality of distributed processing segments for which a portion of the plan is assigned.
18. A computer program product embodied in a tangible, non-transitory computer readable storage medium, comprising computer instructions for:
embedding in each of a plurality of distributed processing segments a library or other shared object comprising one or more data analytical functions, wherein the library or other shared object is included in the processing segments as deployed;
receiving a data analysis request;
creating a plan to generate a response to the request;
selecting one or more of a plurality of distributed processing segments to process a corresponding portion of the plan, wherein selecting the one or more of the plurality of distributed processing segments to process the corresponding portion of the plan includes
obtaining metadata associated with one or more portions of the plan to be performed by the one or more corresponding distributed processing segments; and
embedding the metadata in an assignment communication to be sent to the one or more of the plurality of distributed processing segments, wherein the metadata indicates a location, within a distributed data storage layer, of data to be processed by the corresponding one or more distributed processing segments;
sending, to each of the plurality of distributed processing segments for which a portion of the plan is to be processed, the corresponding portion of the plan to be processed by that corresponding segment and metadata used to locate or access a subset of data on which the segment is to perform an indicated processing, and wherein the metadata sent to each of the plurality of distributed processing segments for which the corresponding portion of the plan is assigned includes an identification of at least one data analytics function to be used to process the portion of the plan of the one or more data analytics functions that is embedded in each of the plurality of distributed processing segments;
receiving, from each of the plurality of distributed processing segments for which a portion of the plan is assigned, a corresponding result of processing the portion of the plan; and
generating, a master response to the data analysis request based at least in part on the corresponding result of processing the portion of the plan received from each of the plurality of distributed processing segments for which a portion of the plan is assigned.