Code execution and data processing pipeline
A method performed by one or more processors comprises displaying code, receiving user selection of a portion of code, determining one or more settable data items, generating a template, displaying the template, receiving a user input value for the settable data items by the template, and executing the code with each of the settable data items set to the received user input value. A data processing pipeline is configured to pass a data item to a first transformer to provide first transformed data, store the first transformed data in a temporary memory, write the first transformed data to the data storage system, and pass the transformed data from the temporary memory to a second transformer.
1. A method performed by one or more processors, the method comprising:
generating a data transformation template by:
displaying a user interface depicting data transformation code including a plurality of code segments, the data transformation code configured to transform data items;
receiving a user selection of a first code segment of the data transformation code;
determining and storing a name or a reference to each of one or more settable items from the first code segment;
generating the data transformation template, wherein the data transformation template indicates the one or more settable items from the first code segment; and
storing the generated data transformation template in a markup language;
initiating transformation of data items from a data source by:
determining the data source storing the data items;
dynamically displaying a user interface including the stored data transformation template, the user interface allowing setting of the one or more settable items;
receiving, via the user interface, a user input value for each of the one or more settable items; and
executing the data transformation code with the one or more settable items set to the corresponding received user input value on the data items from the data source.
2. The method of claim 1 , further comprising receiving multiple user selections and using the user selections to determine the one or more settable items.
3. The method of claim 2 , wherein at least one of the settable items is set, during the executing, by changing text characters in the data transformation code.
4. The method of claim 2 , wherein at least one of the settable items is set, during the executing, by application programming interface calls.
5. A computing system comprising:
a hardware computer processor;
a non-transitory computer readable medium having software instructions stored thereon, the software instructions executable by the hardware computer processor to cause the computing system to perform operations comprising:
generating a data transformation template by:
displaying a user interface depicting data transformation code including a plurality of code segments, the data transformation code configured to transform data items;
receiving a user selection of a first code segment of the data transformation code;
determining and storing a name or a reference to each of one or more settable items from the first code segment;
generating the data transformation template, wherein the data transformation template indicates the one or more settable items from the first code segment; and
storing the generated data transformation template in a markup language;
initiating transformation of data items from a data source by:
determining the data source storing the data items;
dynamically displaying a user interface including the stored data transformation template, the user interface allowing setting of the one or more settable items;
receiving, via the user interface, a user input value for each of the one or more settable items; and
executing the data transformation code with the one or more settable items set to the corresponding received user input value on the data items from the data source.
6. The computing system of claim 5 , further comprising receiving multiple user selections and using the user selections to determine the one or more settable items.
7. The computing system of claim 6 , wherein at least one of the settable items is set, during the executing, by changing text characters in the data transformation code.
8. The computing system of claim 6 , wherein at least one of the settable items is set, during the executing, by application programming interface calls.
9. A non-transitory computer readable medium having software instructions stored thereon, the software instructions executable by a hardware computer processor to cause a computing system to perform operations comprising:
generating a data transformation template by:
displaying a user interface depicting data transformation code including a plurality of code segments, the data transformation code configured to transform data items;
receiving a user selection of a first code segment of the data transformation code;
determining and storing a name or a reference to each of one or more settable items from the first code segment; and
generating the data transformation template, wherein the data transformation template indicates the one or more settable items from the first code segment;
storing the generated data transformation template in a markup language;
initiating transformation of data items from a data source by:
determining the data source storing the data items;
dynamically displaying a user interface including the stored data transformation template, the user interface allowing setting of the one or more settable items;
receiving, via the user interface, a user input value for each of the one or more settable items; and
executing the data transformation code with the one or more settable items set to the corresponding received user input value on the data items from the data source.