Schedule unplannable workload via static runtime

1 . A computer-implemented method for scheduling unplannable workloads via static runtime, comprising:

establishing an inbound channel based on a setup order and associating the inbound channel to an Ingestion-Transformation-Load (ITL) task;

storing incoming data received via the inbound channel to a staging area;

organizing the incoming data in the staging area into a plurality of batches;

monitoring the staging area to determine a number of unprocessed batches;

generating a work order for each of the plurality of batches in the staging area in response to determining that the number of unprocessed batches meets or exceeds a first predetermined threshold;

assigning a priority number to the work order generated for each of the plurality of batches;

executing work orders on a runtime instance in a sequence based on the priority number assigned to each of the work orders; and

removing data in a batch corresponding to the work order being executed on the runtime instance from the staging area when a status for the batch indicates that the batch has been processed and loaded to a target.

2 . The computer-implemented method of claim 1 , further comprising:

pausing the generation of work orders in response to determining that all batches in the staging area have been processed.

3 . The computer-implemented method of claim 1 , further comprising:

pausing the generation of work orders in response to determining that the number of unprocessed batches in the staging area is less than or equal to a second predetermined threshold.

4 . The computer-implemented method of claim 3 , further comprising resuming the generation of work orders after a predefined time period has elapsed.

5 . The computer-implemented method of claim 1 , wherein the incoming data comprises at least one change to existing data for an entity.

6 . The computer-implemented method of claim 5 , wherein the organizing the incoming data in the staging area into a plurality of batches further comprises:

adding, by the ingestion service, the at least one change to the existing data for the entity to a current batch when the entity is not present in the current batch; or

adding, by the ingestion service, the at least one change to the existing data for the entity to a new batch when the entity is present in the current batch.

7 . A system for scheduling unplannable workloads via static runtime, comprising:

at least one processor; and

a memory storing instructions therein that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:

establishing an inbound channel based on a setup order and associating the inbound channel to an Ingestion-Transformation-Load (ITL) task;

storing incoming data received via the inbound channel to a staging area;

organizing the incoming data in the staging area into a plurality of batches;

monitoring the staging area to determine a number of unprocessed batches; and

generating a work order for each of the plurality of batches in the staging area in response to determining that the number of unprocessed batches meets or exceeds a first predetermined threshold;

assigning a priority number to the work order generated for each of the plurality of batches;

executing work orders on a runtime instance in a sequence based on the priority number assigned to each of the work orders; and

removing data in a batch corresponding to the work order being executed on the runtime instance from the staging area when a status for the batch indicates that the batch has been processed and loaded to a target.

8 . The system of claim 7 , wherein the operations further comprise:

pausing the generation of work orders in response to determining that all batches in the staging area have been processed.

9 . The system of claim 7 , wherein the operations further comprise:

pausing the generation of work orders in response to determining that the number of unprocessed batches in the staging area is less than or equal to a second predetermined threshold.

10 . The system of claim 9 , wherein the operations further comprise: resuming the generation of work orders after a predefined time period has elapsed.

11 . The system of claim 7 , wherein the incoming data comprises at least one change to existing data for an entity.

12 . The system of claim 11 , wherein the organizing the incoming data in the staging area into a plurality of batches further comprises:

adding, by the ingestion service, the at least one change to the existing data for the entity to a current batch when the entity is not present in the current batch; or

adding, by the ingestion service, the at least one change to the existing data for the entity to a new batch when the entity is present in the current batch.

13 . A non-transitory computer-readable medium having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations comprising:

establishing an inbound channel based on a setup order and associating the inbound channel to an Ingestion-Transformation-Load (ITL) task;

storing incoming data received via the inbound channel to a staging area;

organizing the incoming data in the staging area into a plurality of batches;

monitoring the staging area to determine a number of unprocessed batches; and

generating a work order for each of the plurality of batches in the staging area in response to determining that the number of unprocessed batches meets or exceeds a first predetermined threshold;

assigning a priority number to the work order generated for each of the plurality of batches;

executing work orders on a runtime instance in a sequence based on the priority number assigned to each of the work orders; and

removing data in a batch corresponding to the work order being executed on the runtime instance from the staging area when a status for the batch indicates that the batch has been processed and loaded to a target.

14 . The non-transitory computer-readable medium of claim 13 , wherein the operations further comprise:

pausing the generation of work orders in response to determining that all batches in the staging area have been processed.

15 . The non-transitory computer-readable medium of claim 13 , wherein the operations further comprise:

pausing the generation of work orders in response to determining that the number of unprocessed batches in the staging area is less than or equal to a second predetermined threshold.

16 . The non-transitory computer-readable medium of claim 13 , wherein the incoming data comprises at least one change to existing data for an entity.

17 . The non-transitory computer-readable medium of claim 16 , wherein the organizing the incoming data in the staging area into a plurality of batches further comprises:

adding, by the ingestion service, the at least one change to the existing data for the entity to a current batch when the entity is not present in the current batch; or

adding, by the ingestion service, the at least one change to the existing data for the entity to a new batch when the entity is present in the current batch.

18 . The computer-implemented method of claim 1 , wherein the inbound channel comprises at least one of:

an endpoint for a Representational state Transfer (REST) Application Programming Interface (API); or

an event broker.

19 . The system of claim 7 , wherein the inbound channel comprises at least one of:

an endpoint for a Representational state Transfer (REST) Application Programming Interface (API); or

an event broker.

20 . The non-transitory computer-readable medium of claim 13 , wherein the inbound channel comprises at least one of:

an endpoint for a Representational state Transfer (REST) Application Programming Interface (API); or

an event broker.