Encapsulating access algorithms for data processing engines
A first data processing engine with an unpublished data format may store data on a shared storage system, where the data is in an unpublished data format. The first data processing engine may generate code, executable by a second data processing engine, where the code is configured to allow the second data processing engine to directly access a portion of the data without having access to knowledge of the unpublished data format.
1 . A system comprising:
a memory storing program instructions; and
a processor in communication with the memory, the processor being configured to execute the program instructions to perform processes comprising:
storing, by a first data processing engine with an unpublished data format, data on a shared storage system; and
generating, by the first data processing engine, executable code, wherein the code is configured to allow a second data processing engine to directly access and read a portion of the data without having access to knowledge of the unpublished data format.
2 . The system of claim 1 , wherein the memory stores further program instructions, and wherein the processor is configured to execute the further program instructions to perform the processes further comprising:
generating worker node partitions to process portions of the generated code in parallel.
3 . The system of claim 1 , wherein the memory stores further program instructions, and wherein the processor is configured to execute the further program instructions to perform the processes further comprising:
receiving a SQL query from a client requesting access to the data.
4 . The system of claim 3 , wherein the memory stores additional program instructions, and wherein the processor is configured to execute the additional program instructions to perform the processes further comprising:
generating a subquery from the query prompting the first data processing engine to generate the code.
5 . The system of claim 1 , wherein the memory stores further program instructions, and wherein the processor is configured to execute the further program instructions to perform the processes further comprising:
processing the generated code in batches.
6 . The system of claim 1 , wherein the memory stores further program instructions, and wherein the processor is configured to execute the further program instructions to perform the processes further comprising:
bridging the generated code to be processed by the second data processing engine.
7 . The system of claim 1 , wherein the generated code comprises metadata with data size and data location.
8 . A method comprising:
storing, by a first data processing engine with an unpublished data format, data on a shared storage system; and
generating, by the first data processing engine, executable code, wherein the code is configured to allow a second data processing engine to directly access and read a portion of the data without having access to knowledge of the unpublished data format.
9 . The method of claim 8 , wherein the method further comprises:
generating worker node partitions to process portions of the generated code in parallel.
10 . The method of claim 8 , wherein the method further comprises:
receiving a SQL query from a client requesting access to the data.
11 . The method of claim 10 , wherein the method further comprises:
generating a subquery from the query prompting the first data processing engine to generate the code.
12 . The method of claim 8 , wherein the method further comprises:
processing the generated code in batches.
13 . The method of claim 8 , wherein the method further comprises:
bridging the generated code to be processed by the second data processing engine.
14 . The method of claim 8 , wherein the generated code contains metadata with data size and data location.
15 . A computer program product comprising one or more computer readable storage media having program instructions collectively embodied therewith, the program instructions executable by one or more processors to cause the one or more processors to perform a method, the method comprising:
storing, by a first data processing engine with an unpublished data format, data on a shared storage system; and
generating, by the first data processing engine, executable code, wherein the code is configured to allow a second data processing engine to directly access and read a portion of the data without having access to knowledge of the unpublished data format.
16 . The computer program product of claim 15 , further comprising additional program instructions collectively stored on the one or more computer readable storage media and configured to cause the one or more processors to perform the method further comprising:
generating worker node partitions to process portions of the generated code in parallel.
17 . The computer program product of claim 15 , further comprising additional program instructions stored on the one or more computer readable storage media and configured to cause the one or more processors to perform the method further comprising:
receiving a SQL query from a client requesting access to the data.
18 . The computer program product of claim 17 , further comprising additional program instructions stored on the one or more computer readable storage media and configured to cause the one or more processors to perform the method further comprising:
generating a subquery from the query prompting the first data processing engine to generate the code.
19 . The computer program product of claim 15 , further comprising additional program instructions stored on the one or more computer readable storage media and configured to cause the one or more processors to perform the method further comprising:
processing the generated code in batches.
20 . The computer program product of claim 15 , further comprising additional program instructions stored on the one or more computer readable storage media and configured to cause the one or more processors to perform the method further comprising:
bridging the generated code to be processed by the second data processing engine.