Extensible data transformation authoring and validation system
View Patent ↗Data transformation in a distributed system of applications and data repositories is described. The subsystems for the overall framework are distributed, thereby allowing for customization to require only isolated changes to one or more subsystems. In one embodiment, a source code repository is used to receive and store source code. A build subsystem can retrieve source code from the source code repository and build it, using one or more criteria. By building the source code, the build subsystem can generate an artifact, which is executable code, such as a JAR or SQL file. Likewise, by building the source code, the build subsystem can generate one or more job specifications for executing the executable code. In one embodiment, the artifact and job specification may be used to launch an application server in a cluster. The application server can then receive data transformation instructions and execute the data transformation instructions.
1. A method performed by one or more processors, the method comprising:
receiving user code comprising at least a portion of a computer program via data retrieval from a computer data storage device or input from a computer input device;
digitally storing the user code at a first code repository;
building the user code into executable code comprising one or more machine executable computer program files, wherein building the user code comprises:
generating the executable code, based on the user code and a first criteria, and digitally storing the executable code in a second repository which is different from the first code repository, wherein the first criteria comprises a criteria for obfuscating sensitive data and the second repository is a blobstore;
generating a job specification, based on the user code and a second criteria, and digitally storing the job specification in a computer memory;
launching computer execution of an application server, using the executable code and the job specification;
receiving data transformation instructions that specify a process for transforming a first dataset to a second dataset;
in response to receiving data transformation instructions, executing, using the executable code, the data transformation instructions in the application server.
2. The method of claim 1 , wherein the first criteria comprises information related to scheduling of a build job for generating executable code.
3. The method of claim 1 , wherein the application server is launched in a cluster of a plurality of application servers.
4. The method of claim 1 , further comprising providing status information to a client device regarding the execution of the data transformation instructions.
5. The method of claim 1 , further comprising:
prior to storing the user code at the first code repository, determining that the user code is authorized based on a first permission setting;
in response to determining that the user code is authorized, performing the step of storing the user code at the first code repository;
prior to executing the data transformation instructions in the application server,
determining that the data transformation instructions are authorized based on a second permission setting; and
in response to determining that the data transformation instructions are authorized, performing the step of executing the data transformation instructions in the application server.
6. The method of claim 1 , wherein the method further comprises:
during the generating the executable code, including, in the executable code, one or more libraries capable of obfuscating sensitive data.
7. One or more non-transitory computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, cause:
receiving user code comprising at least a portion of a computer program via data retrieval from a computer data storage device or input from a computer input device;
digitally storing the user code at a first code repository;
building the user code into executable code comprising one or more machine executable computer program files, wherein building the user code comprises:
generating the executable code, based on the user code and a first criteria, and digitally storing the executable code in a second repository which is different from the first code repository, wherein the first criteria comprises a criteria for obfuscating sensitive data and the second repository is a blobstore;
generating a job specification, based on the user code and a second criteria, and digitally storing the job specification in a computer memory;
launching computer execution of an application server, using the executable code and the job specification;
receiving data transformation instructions that specify a process for transforming a first dataset to a second dataset; and
in response to receiving data transformation instructions, executing, using the executable code, the data transformation instructions in the application server.
8. The one or more non-transitory computer-readable storage medium of claim 7 , wherein the first criteria comprises information related to scheduling of a build job for generating executable code.
9. The one or more non-transitory computer-readable storage medium of claim 7 , wherein the application server is launched in a cluster of a plurality of application servers.
10. The one or more non-transitory computer-readable storage medium of claim 7 , further comprising instructions which, when executed by one or more processors, cause providing status information to a client device regarding the execution of the data transformation instructions.
11. The one or more non-transitory computer-readable storage medium of claim 7 , further comprising instructions which, when executed by one or more processors, cause:
prior to storing the user code at the first code repository, determining that the user code is authorized based on a first permission setting;
in response to determining that the user code is authorized, performing the step of storing the user code at the first code repository;
determining that the data transformation instructions are authorized based on a second permission setting; and
in response to determining that the data transformation instructions are authorized, performing the step of executing the data transformation instructions in the application server.
12. The one or more non-transitory computer-readable storage medium of claim 7 , further comprising instructions which, when executed by one or more processors, cause:
during the generating the executable code, including, in the executable code, one or more libraries capable of obfuscating sensitive data.