PARALLEL DECOMPOSITION AND RESTORATION OF DATA CHUNKS
A system for parallel decomposition and restoration of data chunks is provided, wherein a decomposable transformer service module analyzes and decomposes data into data chunks and transformations that restore the original data from the chunks, enabling efficient storage, modification, and restoration of program code across a number of target devices using a central repository.
1 . A system for parallel decomposition and restoration of data chunks, comprising:
a directed computation graph service module comprising a memory, a processor, and a plurality of programming instructions stored in the memory thereof and operable on the processor thereof, wherein the programmable instructions, when operating on the processor, cause the processor to:
receive input data from a plurality of sources;
analyze the input data to determine a best course of action for analyzing the input data based on measuring shared state requirements between pooled workers in the distributed computing environment with a declarative formalism for specifying data analysis and transformation tasks; and
queue at least a portion of the input data for processing using a decomposable service module based at least in part by analysis of the input data;
a decomposable transformer service module comprising a memory, a processor, and a plurality of programming instructions stored in the memory thereof and operable on the processor thereof, wherein the programmable instructions, when operating on the processor, cause the processor to:
receive data from the directed computation graph module;
analyze the received data to determine a plurality of decomposition operations that may be performed in parallel, wherein a decomposition operation comprises dividing a portion of the received data into a data chunk and a transformation that, when performed on the data chunk, will restore the portion of the received data from the data chunk;
instantiate a plurality of child processes, wherein each child process executes one decomposition operation and all child processes operate in parallel;
destroy each of the plurality of child processes when its respective decomposition operation is concluded; and
store the data chunk and the transformation.
2 . The system of claim 1 , wherein the data chunk comprises program code.
3 . The system of claim 2 , wherein the transformation comprises instructions for compiling the program code.
4 . The system of claim 1 , wherein at least a portion of the data input into the system comes from actions of a user while using an application.
5 . The system of claim 1 , wherein at least a portion of the data input into the system comes from a distributed database.
6 . A method for parallel decomposition and restoration of data chunks, comprising the steps of:
(a) receiving input data from a plurality of sources, using a directed computation graph service module;
(b) analyzing the input data to determine a best course of action for analyzing the input data, using the directed computation graph service module;
(c) queueing at least a portion of the input data for processing using a decomposable service module based at least in part by analysis of the input data;
(d) receiving data from the directed computation graph service module, using a decomposable transformer service module;
(e) analyzing the received data to determine a plurality of decomposition operations that may be performed in parallel, wherein a decomposition operation comprises dividing a portion of the received data into a data chunk and a transformation that, when performed on the data chunk, will restore the portion of the received data from the data chunk;
(f) instantiating a plurality of child processes, wherein each child process executes one decomposition operation and all child processes operate in parallel;
(g) destroying each of the plurality of child processes when its respective decomposition operation is concluded; and
(h) storing the data chunk and the transformation.
7 . The method of claim 6 , wherein the data chunk comprises program code.
8 . The method of claim 7 , wherein the transformation comprises instructions for compiling the program code.
9 . The method of claim 6 , wherein at least a portion of the data input into the system comes from actions of a user while using an application.
10 . The method of claim 6 , wherein at least a portion of the data input into the system comes from a distributed database.