Crawling data transforms

1 . A method of operating a computing system, the method comprising:

receiving information defining a data transformation;

generating a data transformation plan comprising multiple steps, wherein each step of the multiple steps comprises at least one table storing data to transform according to the data transformation;

determining, by a scheduler operating at regular intervals, that a number of running steps is less than a maximum number of running steps;

upon determining that the number of running steps is less than the maximum number of running steps, identifying, by the scheduler, an uncompleted table in the data transformation plan;

referencing a progress table to identify a last transformed record in the uncompleted table, wherein the progress table stores an identifier of the last transformed record;

locking an uncompleted record in the uncompleted table after the last transformed record, wherein locking the uncompleted record prevents the uncompleted record from being changed by other processes;

transforming the uncompleted record in the uncompleted table to generate a transformed record and updating the uncompleted table with the transformed record; and

updating the progress table to store an identifier of the transformed record, wherein the scheduler, upon a subsequent interval, references the progress table to resume transforming the uncompleted table from the last transformed record.

2 . The method of claim 1 , further comprising, prior to identifying the uncompleted table in the data transformation plan, determining that a number of running data transformation plans is less than a maximum number of running data transformation plans.

3 . The method of claim 2 , further comprising:

upon determining that that the number of running data transformation plans is less than the maximum number of running data transformation plans, determining that a next data transformation plan that is not running is inside a transform time window; and

initiating the next data transformation plan.

4 . The method of claim 1 , further comprising:

determining that the data transformation plan is outside a transform time window; and

terminating the data transformation plan.

5 . The method of claim 1 , further comprising generating a second data transformation plan comprising multiple steps, wherein each step of the multiple steps of the second data transformation plan comprises at least one table storing data to transform according to the data transformation.

6 . The method of claim 1 , further comprising, upon transforming the uncompleted record and updating the uncompleted table with the transformed record, updating the progress table indicating that the uncompleted record was transformed.

7 . The method of claim 1 , further comprising, after transforming the uncompleted record and updating the uncompleted table with the transformed record:

determining that the uncompleted table comprises no additional uncompleted records; and

updating the progress table indicating that transforming the uncompleted table according to the data transformation is complete.

8 . The method of claim 7 , further comprising, after determining that the uncompleted table comprises no additional uncompleted records:

identifying a second uncompleted table;

transforming every uncompleted record in the second uncompleted table; and

updating the progress table indicating that transforming the second uncompleted table according to the data transformation is complete.

9 . The method of claim 8 , further comprising:

determining that every table of a step of the multiple steps of the data transformation plan has been transformed according to the data transformation; and

in response to determining that every table in the step of the multiple steps of the data transformation plan has been transformed, updating the progress table indicating that the step of the multiple steps of the data transformation plan is complete.

10 . The method of claim 9 , further comprising:

determining that every step of the multiple steps of the data transformation plan has been completed according to the data transformation; and

updating the progress table indicating that the data transformation plan is complete.

11 . One or more computer-readable storage media having program instructions stored thereon for performing crawling data transforms, wherein the program instructions, when read and executed by a processing system, direct the processing system to at least:

receive information defining a data transformation;

generate a data transformation plan comprising multiple steps, wherein each step of the multiple steps comprises at least one table storing data to transform according to the data transformation;

determine, by a scheduler operating at regular intervals, that a number of running steps is less than a maximum number of running steps;

upon determining that the number of running steps is less than the maximum number of running steps, identify, by the scheduler, an uncompleted table in the data transformation plan;

reference a progress table to identity a last transformed record in the uncompleted table, wherein the progress table stores an identifier of the last transformed record;

lock an uncompleted record in the uncompleted table after the last transformed record, wherein locking the uncompleted record prevents the uncompleted record from being changed by other processes;

transform the uncompleted record in the uncompleted table to generate a transformed record and update the uncompleted table with the transformed record; and

update the progress table to store an identifier of the transformed record, wherein the scheduler, upon a subsequent interval, references the progress table to resume transforming the uncompleted table from the last transformed record.

12 . The one or more computer-readable storage media of claim 11 , wherein the program instructions, when read and executed by the processing system, further direct the processing system to, prior to identifying the uncompleted table in the data transformation plan, determine that a number of running data transformation plans is less than a maximum number of running data transformation plans.

13 . The one or more computer-readable storage media of claim 12 , wherein the program instructions, when read and executed by the processing system, further direct the processing system to:

upon determining that that the number of running data transformation plans is less than the maximum number of running data transformation plans, determine that a next data transformation plan that is not running is inside a transform time window; and

initiate the next data transformation plan.

14 . The one or more computer-readable storage media of claim 11 , wherein the program instructions, when read and executed by the processing system, further direct the processing system to:

determine that the data transformation plan is outside a transform time window; and

terminate the data transformation plan.

15 . The one or more computer-readable storage media of claim 11 , wherein the program instructions, when read and executed by the processing system, further direct the processing system to generate a second data transformation plan comprising multiple steps, wherein each step of the multiple steps of the second data transformation plan comprises at least one table storing data to transform according to the data transformation.

16 . The one or more computer-readable storage media of claim 11 ,

wherein the program instructions, when read and executed by the processing system, further direct the processing system to, upon transforming the uncompleted record and updating the uncompleted table with the transformed record, update the progress table indicating that the uncompleted record was transformed.

17 . The one or more computer-readable storage media of claim 11 , further comprising, after transforming the uncompleted record and updating the uncompleted table with the transformed record:

determine that the uncompleted table comprises no additional uncompleted records; and

update the progress table indicating that transforming the uncompleted table according to the data transformation is complete.

18 . The one or more computer-readable storage media of claim 17 , wherein the program instructions, when read and executed by the processing system, further direct the processing system to, after determining that the uncompleted table comprises no additional uncompleted records:

identify a second uncompleted table;

transform each uncompleted record in the second uncompleted table; and

update the progress table indicating that transforming the second uncompleted table according to the data transformation is complete.

19 . The one or more computer-readable storage media of claim 18 , wherein the program instructions, when read and executed by the processing system, further direct the processing system to:

determine that every table of a step of the multiple steps of the data transformation plan has been transformed according to the data transformation; and

in response to determining that every table of the step of the multiple steps of the data transformation has been transformed, update the progress table indicating that the step of the multiple steps of the data transformation plan is complete.

20 . A system comprising:

one or more computer-readable storage media;

a processing system operatively coupled with the one or more computer-readable storage media; and

program instructions stored on the one or more computer-readable storage media for providing crawling data transformation capabilities, wherein the program instructions, when read and executed by the processing system, direct the processing system to at least:

receive information defining a data transformation;

generate a data transformation plan comprising multiple steps, wherein each step of the multiple steps comprises at least one table storing data to transform according to the data transformation;

determine, by a scheduler operating at regular intervals, that a number of running steps is less than a maximum number of running steps;

upon determining that the number of running steps is less than the maximum number of running steps, identify, by the scheduler, an uncompleted table in the data transformation plan;

reference a progress table to identify a last transformed record in the uncompleted table, wherein the progress table stores an identifier of the last transformed record;

lock an uncompleted record in the uncompleted table after the last transformed record, wherein locking the uncompleted record prevents the uncompleted record from being changed by other processes;

transform the uncompleted record in the uncompleted table to generate a transformed record and update the uncompleted table with the transformed record; and

update the progress table to store an identifier of the transformed record, wherein the scheduler, upon a subsequent interval, references the progress table to resume transforming the uncompleted table from the last transformed record.