Pseudo-synchronous processing by an analytic query and build cluster
The technology disclosed relates to creating and frequently updating multiple online analytic processing (OLAP) analytic databases from an online transaction processing (OLTP) transaction updatable system that includes transaction commit, rollback, and field level security capabilities. It also relates to transparently decoupling extraction from rebuilding of frequently updated OLAP analytic databases from the OLTP transaction updatable system.
1. A method for controlling deployment of units of work at an execution plan controller, comprising:
generating an execution plan indicating a plurality of unit of work lists based at least in part on dependencies between the units of work, wherein at least one unit of work list comprises one or more units of work and at least one unit of work comprises at least a portion of instructions for an online analytic processing (OLAP) analytic query and build cluster (AQBC) to build an OLAP dataset, the dependencies indicating one or more prerequisite units of work to complete before one or more dependent units of work;
dispatching, to the AQBC, one or more first units of work for a first unit of work list according to an order of the execution plan, the order of the execution plan based at least in part on the one or more prerequisite units of work, and wherein the AQBC is configured to implement a data extraction process to build the OLAP dataset;
receiving, from the AQBC, one or more responses indicating completion of the one or more first units of work;
determining completion of all units of work comprised in the first unit of work list based at least in part on receiving the one or more responses; and
dispatching one or more second units of work for a second unit of work list to the AQBC based at least in part on the determined completion and according to the order of the execution plan.
2. The method of claim 1 , wherein:
a first unit of work of the one or more first units of work is a prerequisite to a second unit of work of the one or more second units of work; and
the execution plan is generated based at least in part on the first unit of work being the prerequisite to the second unit of work.
3. The method of claim 2 , wherein generating the execution plan comprises:
storing the first unit of work in the first unit of work list; and
storing the second unit of work in the second unit of work list based at least in part on the first unit of work being the prerequisite to the second unit of work.
4. The method of claim 1 , further comprising:
receiving a plurality of units of work and a plurality of prerequisites, wherein each prerequisite of the plurality of prerequisites corresponds to a unit of work of the plurality of units of work, and the execution plan is generated based at least in part on the plurality of units of work and the plurality of prerequisites.
5. The method of claim 4 , wherein the plurality of units of work and the plurality of prerequisites are received via a user interface, an application programming interface, an external file, or a combination thereof.
6. The method of claim 1 , further comprising:
receiving, from the AQBC, a response indicating completion of a first unit of work for the first unit of work list;
determining that all units of work comprised in the first unit of work list are not dispatched to the AQBC; and
dispatching, to the AQBC, a second unit of work for the first unit of work list based at least in part on the response, the determining that all units of work comprised in the first unit of work list are not dispatched to the AQBC, and the order of the execution plan.
7. The method of claim 1 , further comprising:
receiving, from the AQBC, a response indicating completion of a first unit of work for the first unit of work list;
determining that all units of work comprised in the first unit of work list are not complete; and
refraining from dispatching any units of work for the second unit of work list to the AQBC based at least in part on the determining that all units of work comprised in the first unit of work list are not complete.
8. The method of claim 1 , wherein dispatching the one or more first units of work further comprises:
dispatching, to the AQBC, the one or more first units of work for the first unit of work list based at least in part on a concurrency limit for the AQBC.
9. The method of claim 1 , further comprising:
assigning a request identifier (ID) to a first unit of work of the one or more first units of work;
dispatching, to the AQBC, the first unit of work;
assigning an instance ID to the first unit of work based at least in part on dispatching the first unit of work;
receiving, from the AQBC, a response indicating completion of the first unit of work; and
assigning a complete ID to the first unit of work based at least in part on receiving the response.
10. The method of claim 1 , further comprising:
creating a snapshot of the execution plan; and
saving the snapshot of the execution plan in a persistent data store.
11. The method of claim 10 , further comprising:
executing the execution plan according to the snapshot of the execution plan, wherein dispatching the one or more first units of work and dispatching the one or more second units of work are based at least in part on executing the execution plan;
receiving a change to the execution plan; and
refraining from incorporating the change to the execution plan in the snapshot of the execution plan until completion of executing the execution plan.
12. The method of claim 11 , further comprising:
identifying completion of executing the execution plan; and
incorporating the change to the execution plan in the snapshot of the execution plan in the persistent data store based at least in part on the identifying.
13. The method of claim 1 , further comprising:
saving a current state of the execution plan in persistent data storage;
identifying a failure event during execution of the execution plan; and
restarting execution of the execution plan from the saved current state of the execution plan based at least in part on the identified failure event.
14. The method of claim 1 , wherein a unit of work comprises an analytic query or a build request for one or more analytic databases.
15. An apparatus for controlling deployment of units of work at an execution plan controller, comprising:
a processor;
memory coupled with the processor; and
instructions stored in the memory and executable by the processor to cause the apparatus to:
generate an execution plan indicating a plurality of unit of work lists based at least in part on dependencies between the units of work, wherein at least one unit of work list comprises one or more units of work and at least one unit of work comprises at least a portion of instructions for an online analytic processing (OLAP) analytic query and build cluster (AQBC) to build an OLAP dataset, the dependencies indicating one or more prerequisite units of work to complete before one or more dependent units of work;
dispatch, to the AQBC, one or more first units of work for a first unit of work list according to an order of the execution plan, the order of the execution plan based at least in part on the one or more prerequisite units of work, and wherein the AQBC is configured to implement a data extraction process to build the OLAP dataset;
receive, from the AQBC, one or more responses indicating completion of the one or more first units of work;
determine completion of all units of work comprised in the first unit of work list based at least in part on receiving the one or more responses; and
dispatch one or more second units of work for a second unit of work list to the AQBC based at least in part on the determined completion and according to the order of the execution plan.
16. The apparatus of claim 15 , wherein:
a first unit of work of the one or more first units of work is a prerequisite to a second unit of work of the one or more second units of work; and
the execution plan is generated based at least in part on the first unit of work being the prerequisite to the second unit of work.
17. The apparatus of claim 16 , wherein the instructions to generate the execution plan are executable by the processor to cause the apparatus to:
store the first unit of work in the first unit of work list; and
store the second unit of work in the second unit of work list based at least in part on the first unit of work being the prerequisite to the second unit of work.
18. The apparatus of claim 15 , wherein the instructions are further executable by the processor to cause the apparatus to:
receive, from the AQBC, a response indicating completion of a first unit of work for the first unit of work list;
determine that all units of work comprised in the first unit of work list are not dispatched to the AQBC; and
dispatch, to the AQBC, a second unit of work for the first unit of work list based at least in part on the response, the determining that all units of work comprised in the first unit of work list are not dispatched to the AQBC, and the order of the execution plan.
19. The apparatus of claim 15 , wherein the instructions are further executable by the processor to cause the apparatus to:
receive, from the AQBC, a response indicating completion of a first unit of work for the first unit of work list;
determine that all units of work comprised in the first unit of work list are not complete; and
refrain from dispatching any units of work for the second unit of work list to the AQBC based at least in part on the determining that all units of work comprised in the first unit of work list are not complete.
20. A non-transitory computer-readable medium storing code for controlling deployment of units of work at an execution plan controller, the code comprising instructions executable by a processor to:
generate an execution plan indicating a plurality of unit of work lists based at least in part on dependencies between the units of work, wherein at least one unit of work list comprises one or more units of work and at least one unit of work comprises at least a portion of instructions for an online analytic processing (OLAP) analytic query and build cluster (AQBC) to build an OLAP dataset, the dependencies indicating one or more prerequisite units of work to complete before one or more dependent units of work;
dispatch, to the AQBC, one or more first units of work for a first unit of work list according to an order of the execution plan, the order of the execution plan based at least in part on the one or more prerequisite units of work, and wherein the AQBC is configured to implement a data extraction process to build the OLAP dataset;
receive, from the AQBC, one or more responses indicating completion of the one or more first units of work;
determine completion of all units of work comprised in the first unit of work list based at least in part on receiving the one or more responses; and
dispatch one or more second units of work for a second unit of work list to the AQBC based at least in part on the determined completion and according to the order of the execution plan.