Shipping of data through ETL stages
Performing an extract, transform, and load (ETL) process. Column data is received by a stage of the ETL process. The size of the received column data is ascertained by the stage. In response to determining that the size of the column data exceeds a predefined threshold, the stage saves the column data and creates a data locator associated with the column data. The created data locator advances through successive downstream stages of the ETL process as a replacement for the column data.
1. A computer system comprising one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage media, and program instructions stored on at least one of the one or more computer-readable tangible storage media for execution by at least one of the one or more processors via at least one of the one or more computer-readable memories, wherein the computer system is capable of performing a method of shipping data between stages of an extract, transform, and load (ETL) process, wherein the method comprising:
receiving, by a stage of a plurality of stages associated with the ETL process, column data;
ascertaining, by the stage, a size of the received column data; and
in response to determining, by the stage, that the size of the received column data exceeds a predefined threshold:
saving, by the stage, the column data;
creating, by the stage, a data locator associated with the column data, wherein the created data locator is a data structure that includes metadata comprising (i) a hash key, (ii) a source indicator, and (iii) a timestamp, wherein the hash key corresponds to the saved column data associated with the data locator, the source indicator corresponds to the stage, and the timestamp indicates when the column data was received in the stage,
advancing the created data locator through each of a plurality of successive downstream stages;
receiving, at the stage, an indication that a respective downstream stage requires the column data based on the data locator received at the respective downstream stage;
determining, by the respective successive stage and based on the received data locator, whether the saved column data is a latest version;
in response to determining that the saved column data is the latest version, forwarding the saved column data directly from the stage to the respective downstream stage; and
in response to determining that the saved column data is not the latest version, forwarding the latest version of the column data directly to the respective downstream stage from an upstream stage, different than the stage, containing the latest version of the column data.