Methods and systems for distributed data analysis
The present disclosure relates to computer implemented methods and systems for data management, data analysis, and processing. The disclosed methods and systems can incorporate external data analysis into an otherwise closed data analysis environment.
1 . A method comprising:
generating, by a computing device, based on a user selection of a first graphical object output via a user interface at the computing device, a data subset associated with a data model stored in primary memory of the computing device, wherein the user selection of the first graphical object is indicative of a request for external processing;
in response to the request for external processing, sending, to at least one remote computing device comprising an external engine to perform the external processing, a table comprising at least one tuple associated with the data subset, the at least one tuple comprising an expression and at least one value for each of at least two dimensions of the data subset, wherein the table is generated based on a format associated with the external engine, wherein the external processing comprises an evaluation of the expression on at least a portion of the data subset comprising one or more data records associated with the at least one value for each of the at least two dimensions, and wherein the external engine is configured to provide a result of the external processing that is consonant to a cardinality of the at least one tuple;
receiving, from the external engine via the at least one remote computing device, the table comprising the result of the external processing, wherein the result comprises at least one result tuple, and wherein a cardinality of the at least one result tuple at least partially differs from the cardinality of the at least one tuple; and
based on a determination that the cardinality of the at least one result tuple is consonant to the cardinality of the at least one tuple, appending the result directly to the primary memory of the computing device storing the data model without requiring the result to be formatted.
2 . The method of claim 1 , wherein appending the result directly to the primary memory of the computing device storing the data model without requiring the result to be formatted comprises appending, without intervention from a script engine, the result of the external processing directly to the primary memory of the computing device storing the data model.
3 . The method of claim 2 , wherein the script engine loads the data model into memory.
4 . The method of claim 1 , wherein receiving the table comprising the result of the external processing comprises comparing the result of the external processing to values of the one or more data records to preserve associativity.
5 . The method of claim 1 , wherein the external engine comprises one or more data processing applications.
6 . The method of claim 5 , wherein the one or more data processing applications comprise one or more of a simulation application, a statistical application, a mathematical computation application, or a database application.
7 . The method of claim 1 , wherein the table comprising the result of the external processing comprises at least one value derived from the evaluation of the expression.
8 . The method of claim 7 , wherein appending the result directly to the primary memory of the computing device storing the data model without requiring the result to be formatted comprises appending the at least one value derived from the evaluation of the expression directly to the primary memory of the computing device storing the data model.
9 . A method comprising:
determining, by a computing device, based on a user selection of in-memory data at the computing device, one or more related tables associated with the in-memory data, wherein the user selection is indicative of a request for external processing;
in response to the request for external processing, sending, to at least one remote computing device comprising an external engine to perform the external processing, a table comprising at least one tuple, the at least one tuple comprising at least one expression and at least one value for each of at least two dimensions of the one or more related tables, wherein the table is generated based on a format associated with the external engine, wherein the external processing comprises an evaluation of the at least one expression on at least a portion of the one or more related tables comprising one or more data records associated with the at least one value for each of the at least two dimensions, and wherein the external engine is configured to provide a result of the evaluation that is consonant to a cardinality of the at least one tuple;
receiving, from the external engine via the at least one remote computing device, the table comprising the result of the evaluation, wherein the result comprises at least one result tuple, and wherein a cardinality of the at least one result tuple at least partially differs from the cardinality of the at least one tuple; and
based on a determination that the cardinality of the at least one result tuple is consonant to the cardinality of the at least one tuple, appending the result of the evaluation directly to the in-memory data at the computing device without requiring the result to be formatted.
10 . The method of claim 9 , wherein appending the result of the evaluation directly to the in-memory data at the computing device without requiring the result to be formatted comprises appending, without intervention from a script engine, the result of the evaluation directly to the in-memory data at the computing device.
11 . The method of claim 10 , wherein the script engine loads the in-memory data at the computing device.
12 . The method of claim 9 , wherein the external engine comprises one or more data processing applications.
13 . The method of claim 12 , wherein the one or more data processing applications comprise one or more of a simulation application, a statistical application, a mathematical computation application, or a database application.
14 . The method of claim 9 , wherein the table comprising the result of the external processing comprises at least one value derived from the evaluation of the at least one expression.
15 . A non-transitory computer-readable storage medium comprising processor-executable instructions that, when executed by at least one processor of a computing device, cause the at least one processor to:
determine, based on a user selection of in-memory data at the computing device, one or more related tables associated with the in-memory data, wherein the user selection is indicative of a request for external processing;
in response to the request for external processing, send, to at least one remote computing device comprising an external engine to perform the external processing, a table comprising at least one tuple, the at least one tuple comprising at least one expression and at least one value for each of at least two dimensions of the one or more related tables, wherein the table is generated based on a format associated with the external engine, wherein the external processing comprises an evaluation of the at least one expression on at least a portion of the one or more related tables comprising one or more data records associated with the at least one value for each of the at least two dimensions, and wherein the external engine is configured to provide a result of the evaluation that is consonant to a cardinality of the at least one tuple;
receive, from the external engine via the at least one remote computing device, the table comprising the result of the evaluation, wherein the result comprises at least one result tuple, and wherein a cardinality of the at least one result tuple at least partially differs from the cardinality of the at least one tuple; and
based on a determination that the cardinality of the at least one result tuple is consonant to the cardinality of the at least one tuple, appending the result of the evaluation directly to the in-memory data without requiring the result to be formatted.
16 . The non-transitory computer-readable storage medium of claim 15 , wherein the processor-executable instructions that cause the at least one processor to append the result of the evaluation directly to the in-memory data without requiring the result to be formatted further cause the at least one processor to append, without intervention from a script engine, the result of the evaluation directly to the in-memory data at the computing device.
17 . The non-transitory computer-readable storage medium of claim 16 , wherein the script engine loads the in-memory data at the computing device.
18 . The non-transitory computer-readable storage medium of claim 15 , wherein the external engine comprises one or more data processing applications.
19 . The non-transitory computer-readable storage medium of claim 18 , wherein the one or more data processing applications comprise one or more of a simulation application, a statistical application, a mathematical computation application, or a database application.
20 . The non-transitory computer-readable storage medium of claim 15 , wherein the table comprising the result of the external processing comprises at least one value derived from the evaluation of the at least one expression.