CODE EXECUTION AND DATA PROCESSING PIPELINE
A method performed by one or more processors comprises displaying code, receiving user selection of a portion of code, determining one or more settable data items, generating a template, displaying the template, receiving a user input value for the settable data items by the template, and executing the code with each of the settable data items set to the received user input value. A data processing pipeline is configured to pass a data item to a first transformer to provide first transformed data, store the first transformed data in a temporary memory, write the first transformed data to the data storage system, and pass the transformed data from the temporary memory to a second transformer.
1 . A computerized method, performed by a computing system having one or more hardware computer processors and one or more non-transitory computer readable storage device storing software instructions executable by the computing system to perform the computerized method comprising:
initiating a data transformation, wherein the data transformation comprises creating a data object;
initiating a first write operation, wherein the first write operation comprises storing the data object in a temporary memory;
in parallel with or subsequent to initiating the first write operation, initiating a second write operation, wherein the second write operation comprises storing the data object in a database; and
initiating a read operation wherein:
the read operation comprises accessing the data object; and
if the read operation is initiated before the second write operation has completed, the data object is accessed from the temporary memory.
2 . The computerized method of claim 1 , wherein the temporary memory comprises a volatile memory and a non-volatile memory, wherein the volatile memory is configured to be used initially with the non-volatile memory configured to be used as a fallback to the volatile memory.
3 . The computerized method of claim 1 , wherein the read operation comprises reading the data object for display in a user interface before the second write operation completes.
4 . The computerized method of claim 1 , wherein the read operation comprises passing the data object from the temporary memory to a subsequent transformer in a data processing pipeline.
5 . The computerized method of claim 1 , further comprising persisting intermediary results of the data transformation.
6 . The computerized method of claim 1 , wherein the temporary memory is located on a machine remote from the one or more hardware computer processors and is accessible via a high bandwidth, low latency network.
7 . The computerized method of claim 1 , wherein the second write operation is performed at regular intervals as a safeguard against system failure.
8 . The computerized method of claim 7 , wherein the regular intervals are a fixed value.
9 . The computerized method of claim 1 , wherein the read operation continues to access the data object from the temporary memory even after the second write operation has completed.
10 . The computerized method of claim 1 , further comprising allowing other data pipelines or processes to access the data object from the temporary memory while the second write operation is in progress.
11 . The computerized method of claim 1 , wherein initiating the data transformation comprises executing computer-executable code in one or more of: Python, Java, Scala, C#, XML, HTML, or YAML.
12 . A computing system comprising:
a hardware computer processor; and
a non-transitory computer readable medium having software instructions stored thereon, the software instructions executable by the hardware computer processor to cause the computing system to perform operations comprising:
initiating a data transformation, wherein the data transformation comprises creating a data object;
initiating a first write operation, wherein the first write operation comprises storing the data object in a temporary memory;
in parallel with or subsequent to initiating the first write operation, initiating a second write operation, wherein the second write operation comprises storing the data object in a database; and
initiating a read operation wherein:
the read operation comprises accessing the data object; and
if the read operation is initiated before the second write operation has completed, the data object is accessed from the temporary memory.
13 . The computing system of claim 12 , wherein the temporary memory comprises a volatile memory and a non-volatile memory, wherein the volatile memory is configured to be used initially with the non-volatile memory configured to be used as a fallback to the volatile memory.
14 . The computing system of claim 12 , wherein the read operation comprises reading the data object for display in a user interface before the second write operation completes.
15 . The computing system of claim 12 , wherein the read operation comprises passing the data object from the temporary memory to a subsequent transformer in a data processing pipeline.
16 . The computing system of claim 12 , wherein the operations further comprise persisting intermediary results of the data transformation.
17 . The computing system of claim 12 , wherein the temporary memory is located on a machine remote from the hardware computer processor and is accessible via a high bandwidth, low latency network.
18 . The computing system of claim 12 , wherein the second write operation is performed at regular intervals as a safeguard against system failure.
19 . The computing system of claim 18 , wherein the regular intervals are a fixed value.
20 . The computing system of claim 12 , wherein the read operation continues to access the data object from the temporary memory even after the second write operation has completed.