Systems and methods for analysis of data at disparate data sources
Embodiments are described for a system and method to analyze data at a plurality of data sources. A data analytic workflow may be received. The data analytic workflow may include at least one operation to be performed on a plurality of data sets stored at a plurality of data sources. Instructions may be created based on the operation to be performed and a type of platform that operates the data sources. Furthermore, the instructions may be transmitted to the data sources such that the data sources may execute the operations on the data sets stored at the data sources.
1. A method to perform an analysis of a plurality of data sets stored at a plurality of data sources, the method comprising:
generating a graphical user interface that specifies a first remote data source and a plurality of operations that can be performed on a first data set and that specifies a second remote data source and another plurality of operations that can be performed on a second data set, wherein the first data source is of a first platform type and the second data source is of a second platform type, the first platform type being different than the second platform type;
generating a workflow that identifies the first remote data source and the second remote data source, an operation to combine, at the second data source running the second platform type, the first data set and the second data set that are stored at different data sources running different platform types, and at least one other operation;
receiving the workflow;
dynamically determining whether to translate the workflow;
in response to a determination to dynamically translate the workflow, translating the workflow into a set of instructions, wherein the translating is at least partially based on a platform type of the second data source at which the operation to combine is performed; and
transmitting the first data set stored at the first remote data source and the set of instructions to the second remote data source at which the first data set and the second data set are combined after the first data set being transmitted to the second remote data source;
wherein the at least one other operation is a modeling algorithm operation, model validation operation, prediction operation, sampling operations, or any combination thereof.
2. The method of claim 1 , wherein the first platform type is a relational database management system (RDBMS) and the second platform type is a distributed file system.
3. The method of claim 1 , wherein the at least one other operation is an operation to be performed on the combined first data set and second data set by the second data source, the set of instructions for the second data source includes a step associated with the other operation.
4. The method of claim 1 , wherein the set of instructions comprise at least one of programming code or function calls based on a type of platform associated with the first data source or second data source.
5. A system for analyzing a plurality of data sets stored at a plurality of data sources, the system comprising:
one or more processors; and
a non-transitory computer-readable medium storing a processor-based application, which when executed on a computer, will cause the one or more processors to:
generate a graphical user interface that specifies a first remote data source and a plurality of operations that can be performed on a first data set and that specifies a second remote data source and another plurality of operations that can be performed on a second data set, wherein the first data source is of a first platform type and the second data source is of a second platform type, the first platform type being different than the second platform type;
generate a workflow that identifies the first remote data source and the second remote data source, an operation to combine, at the second data source running the second platform type, the first data set and the second data set that are stored at different data sources running different platform types, and at least one other operation;
receive the workflow;
dynamically determining whether to translate the workflow;
in response to a determination to translate the workflow, translating the workflow into a set of instructions, wherein the translating is at least partially based on a platform type of the second data source at which the operation to combine is performed;
and
transmit the first data set stored at the first remote data source and the set of instructions to the second remote data source at which the first data set and the second data set are combined after the first data set being transmitted to the second remote data source;
wherein the at least one other operation is a modeling algorithm operation, model validation operation, prediction operation, sampling operations, or any combination thereof.
6. The system of claim 5 , wherein the first platform type is a relational database management system (RDBMS) and the second platform type is a distributed file system.
7. The system of claim 5 , wherein the at least one other operation is an operation to be performed on the combined first data set and second data set by the second data source, the set of instructions for the second data source includes a step associated with the other operation.
8. The system of claim 5 , wherein the set of instructions comprise at least one of programming code or function calls based on a type of platform associated with the first data source or second data source.
9. A computer program product, comprising a non-transitory computer readable medium carrying one or more sequences of instructions to analyze a plurality of data sets stored at a plurality of data sources which, when executed by one or more processors, causes the one or more processors to carry out the steps of:
generating a graphical user interface that specifies a first remote data source and a plurality of operations that can be performed on a first data set and that specifies a second remote data source and another plurality of operations that can be performed on the second data set, wherein the first data source is of a first platform type and the second data source is of a second platform type, the first platform type being different than the second platform type;
generating a workflow that identifies the first remote data source and the second remote data source, an operation to combine, at the second data source running the second platform type, the first data set and the second data set that are stored at different data sources running different platform types, and at least one other operation;
receiving the workflow;
dynamically determining whether to translate the workflow;
in response to a determination to dynamically translate the workflow, translating the workflow into a set of instructions, wherein the translating is at least partially based on a platform type of the second data source at which the operation to combine is performed;
and
transmitting the first data set stored at the first remote data source and the set of instructions to the second remote data source at which the first data set and the second data set are combined after the first data set being transmitted to the second remote data source;
wherein the at least one other operation is a modeling algorithm operation, model validation operation, prediction operation, sampling operations, or any combination thereof.
10. The computer program product of claim 9 , wherein the first platform type is a relational database management system (RDBMS) and the second platform type is a distributed file system.
11. The computer program product of claim 9 , wherein the at least one other operation is an operation to be performed on the combined first data set and second data set by the second data source, the set of instructions for the second data source includes a step associated with the other operation.
12. The computer program product of claim 9 , wherein the set of instructions comprise at least one of programming code or function calls based on a type of platform associated with the first data source or second data source.
13. The method of claim 1 , wherein the operation to combine is a join operation.
14. The system of claim 5 , wherein the operation to combine is a join operation.
15. The computer program product of claim 9 , wherein the operation to combine is a join operation.
16. The method of claim 1 , wherein the at least one other operation further includes at least one selected from a group consisting of:
data extraction operations, exploration operations, transformation operations, and analysis operations to be performed on the combined first data set and second data set by the second data source, a second set of instructions for the second data source includes a step associated with the at least one other operation.
17. The system of claim 5 , wherein the at least one other operation further includes at least one selected from a group consisting of:
data extraction operations, exploration operations, transformation operations, and analysis operations to be performed on the combined first data set and second data set by the second data source, a second set of instructions for the second data source includes a step associated with the at least one other operation.
18. The computer program product of claim 9 , wherein the at least one other operation further includes at least one selected from a group consisting of:
data extraction operations, exploration operations, transformation operations, and analysis operations to be performed on the combined first data set and second data set by the second data source, a second set of instructions for the second data source includes a step associated with the at least one other operation.
19. The method of claim 1 , further comprising:
generating the workflow that identifies the first remote data source and the second remote data source, the operation to combine the first data set and the second data set, and the at least one other operation, and, in addition thereto, at least one additional operation;
translating, by the computer, the workflow into another set of instructions specific to the first platform type and the at least one additional operation; and
transmitting the other set of instructions to the first remote data source;
wherein the additional operation is at least one selected from a group consisting of: a data set combine operation, a data extraction operation, exploration operation, sampling operation, transformation operation, and analysis operation to be performed on the first data set of the first data source, the other set of instructions for the first data source includes a step associated with the additional operation.
20. The system of claim 5 , wherein the non-transitory computer-readable medium storing the processor-based application, which when executed on a computer, will further cause the one or more processors to:
generate the workflow that identifies the first remote data source and the second remote data source, the operation to combine on the first data set and the second data set, and the at least one other operation, and, in addition thereto, at least one additional operation;
translate the workflow into another set of instructions specific to the first platform type and the at least one additional operation; and
transmit the other set of instructions to the first remote data source;
wherein the additional operation is at least one selected from a group consisting of: a data set combine operation, a data extraction operation, exploration operation, sampling operation, transformation operation, and analysis operation to be performed on the first data set of the first data source, the other set of instructions for the first data source includes a step associated with the additional operation.
21. The computer program product of claim 9 , further comprising the steps of:
generating the workflow that identifies the first remote data source and the second remote data source, the operation to combine the first data set and the second data set, and the at least one other operation, and, in addition thereto, at least one additional operation;
translating, by the computer, the workflow into another set of instructions specific to the first platform type and the at least one additional operation; and
transmitting the other set of instructions to the first remote data source;
wherein the additional operation is at least one selected from a group consisting of: a data set combine operation, a data extraction operation, exploration operation, sampling operation, transformation operation, and analysis operation to be performed on the first data set of the first data source, the other set of instructions for the first data source includes a step associated with the additional operation.
22. The method of claim 1 , further comprising retrieving the first data set from the first data source.