Click-to-script reflection
A click-to-script service enables developers of big-data job scripts to quickly see the underlying script operations from optimized execution plans. Once a big-data job is received, the disclosed examples compile it and generate tokens that are associated with each operation of the big-data job. These tokens include may include the file name of the job, the line number of the operation, and/or an Abstract Syntax Tree (AST) node for the given operations. An original execution plan is optimized into an optimized execution plan, and the tokens for the original operations of the job script are assigned to the optimized operations of the optimized execution plan. The optimized execution plan is graphically displayed in an interactive manner such that users may view the optimized execution plan and click on its optimized operations to find the original operations of the job script.
1. A method for tracking a job script operation, the method comprising:
applying a token to each line of the job script operation for a job during compiling of the job, wherein the token comprises a file name of the job script and a line number of the job script operation of the job to which the token is applied;
optimizing the job script operation to create an optimized execution plan of the job, wherein the optimized execution plan comprises a different operation than the job script operation;
associating a token applied to a job script operation with the different operation of the optimized execution plan; and
providing a graphical representation of the operation in a user interface that allows a user to navigate back to the job script operation.
2. The method of claim 1 , further comprising optimizing the job into the optimized execution plan through combining at least two job script operations into an optimized operation of the optimized execution plan.
3. The method of claim 1 , further comprising optimizing the job into the optimized execution plan through generating a tree of optimized operations for the optimized execution plan.
4. The method of claim 3 , wherein generation of the tree comprises expressing either a COptExpr or a GroupExpr class of tree.
5. The method of claim 1 , wherein the token additionally includes an Abstract Syntax Tree (AST) of an original plan of the job.
6. The method of claim 1 , further comprising running the optimized execution plan of the job in a container of a cloud environment.
7. The method of claim 6 , wherein the container comprises a Yet Another Resource Negotiator (YARN) container.
8. The method of claim 6 , wherein the cloud environment comprises a virtual machine running the container.
9. The method of claim 1 , wherein the job is a Structured Computations Optimize for Parallel Execution (SCOPE) job.
10. The method of claim 1 , wherein the job is a Structured Query Language (SQL) job.
11. A method for tracking a job script operation, the method comprising:
applying a token to each line of the job script operation for a job during compiling of the job, wherein the token comprises a line number of the job script operation to which the token is applied and positions of an Abstract Syntax Tree (AST) node for an original plan of the job;
optimizing the job script operation to create an optimized execution plan of the job, wherein the optimized execution plan comprises a different operation than the job script operation;
associating a token applied to a job script operation with the different operation of the optimized execution plan; and
providing a graphical representation of the operation in a user interface that allows a user to navigate back to the job script operation.
12. The method of claim 11 , further comprising optimizing the job into the optimized execution plan through combining at least two job script operations into an optimized operation of the optimized execution plan.
13. The method of claim 11 , further comprising optimizing the job into the optimized execution plan through generating a tree of optimized operations for the optimized execution plan.
14. The method of claim 11 , wherein the graphical representation comprises interactive tree leaves comprising the different operation that a user is able to navigate to view the job script operation associated with the different operation.
15. The method of claim 11 , wherein the different operation is generated through usage of a COpArg abstract class representing tree node details.
16. A system for tracking a job script operation, the system comprising:
memory embodied with instructions to compile and add a token to a job script operation of a job and optimize the job script operations with the added token into an optimized execution plan; and
a processor configured to:
apply the token to each line of the job script operation for a job during compiling of the job, wherein the token comprises a line number of the job script operation of the job to which the token is applied;
optimize the job script operation to create the optimized execution plan of the job, wherein the optimized execution plan comprises a different operation than the job script operation;
associate a token applied to a job script operation with the different operation of the optimized execution plan; and
provide a graphical representation of the different operation to a client computing device.
17. The system of claim 16 , wherein said optimizing the job into the optimized execution plan comprises combining at least two job script operations into an optimized operation of the optimized execution plan.
18. The system of claim 16 , wherein said optimizing the job into the optimized execution plan comprises generating a tree of optimized operations for the optimized execution plan.
19. The system of claim 16 , wherein the job is a Structured Computations Optimize for Parallel Execution (SCOPE) job.
20. The system of claim 16 , wherein the job is a Structured Query Language (SQL) job.