Callable notebook for cluster execution
A system for processing a notebook includes an input interface and a processor. The input interface is to receive a first notebook. The notebook comprises code for interactively querying and viewing data. The processor is to load the first notebook into a shell. The shell receives one or more parameters associated with the first notebook. The shell executes the first notebook using a cluster.
1. A system for processing a notebook, comprising:
an input interface to receive a first notebook, wherein the notebook comprises code for interactively querying and viewing data; and
a processor to:
load the first notebook into a shell, wherein the first notebook includes at least one command, and wherein the shell receives one or more parameters associated with the first notebook; and
execute a command of the at least one command, comprising to:
determine whether a selected command relates to a local command; and
in response to a determination that the selected command does not relate to the local command:
determine whether the selected command relates to a cluster command, comprising to:
determine whether the selected command relates to a cluster command based on an estimated time to complete the command locally, an estimated processor load to complete the command locally, a data usage, an indication of whether the command is the cluster command, or any combination thereof; and
in response to a determination that the selected command does not relate to the cluster command:
determine that the selected command relates to a special case; and
execute the special case command.
2. The system of claim 1 , wherein the shell replaces one or more variables in the first notebook with the one or more input parameters.
3. The system of claim 1 , wherein the shell executes the first notebook a second time using one or more different input parameters.
4. The system of claim 1 , wherein the first notebook comprises a command to execute a second notebook.
5. The system of claim 4 , wherein the second notebook and the first notebook comprise different notebooks.
6. The system of claim 4 , wherein the second notebook is written in a different language than the first notebook.
7. The system of claim 4 , wherein the second notebook and the first notebook comprise the same notebook.
8. The system of claim 4 , wherein the second notebook produces output.
9. The system of claim 1 , wherein the first notebook produces output.
10. The system of claim 1 , wherein the shell is configured to prevent recursion.
11. The system of claim 1 , wherein the processor provides an indication for display based at least in part on an output from the first notebook.
12. The system of claim 1 , wherein the first notebook comprises multiple code sections.
13. The system of claim 12 , wherein code sections within the first notebook are in different languages.
14. The system of claim 1 , wherein the shell re-executes the first notebook.
15. The system of claim 1 , wherein the shell re-executes a code section of the first notebook.
16. The system of claim 1 , wherein executing the first notebook comprises accessing a big data repository.
17. The system of claim 16 , wherein updated data is accessed from the big data repository.
18. The system of claim 17 , wherein the shell re-executes the first notebook using the updated data.
19. A method for processing a notebook, comprising:
receiving a first notebook, wherein the notebook comprises code for interactively querying and viewing data;
loading, using a processor, the first notebook into a shell, wherein the first notebook includes at least one command, and wherein the shell receives one or more parameters associated with the first notebook; and
executing a command of the at least one command, comprising:
determining whether a selected command relates to a local command; and
in response to a determination that the selected command does not relate to the local command:
determining whether the selected command relates to a cluster command, comprising:
determining whether the selected command relates to a cluster command based on an estimated time to complete the command locally, an estimated processor load to complete the command locally, a data usage, an indication of whether the command is the cluster command, or any combination thereof; and
in response to a determination that the selected command does not relate to the cluster command:
determining that the selected command relates to a special case; and
executing the special case command.
20. A computer program product for processing a notebook, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
receiving a first notebook, wherein the notebook comprises code for interactively querying and viewing data;
loading the first notebook into a shell, wherein the first notebook includes at least one command, and wherein the shell receives one or more parameters associated with the first notebook; and
executing a command of the at least one command, comprising:
determining whether a selected command relates to a local command; and
in response to a determination that the selected command does not relate to the local command:
determining whether the selected command relates to a cluster command, comprising:
determining whether the selected command relates to a cluster command based on an estimated time to complete the command locally, an estimated processor load to complete the command locally, a data usage, an indication of whether the command is the cluster command, or any combination thereof; and
in response to a determination that the selected command does not relate to the cluster command:
determining that the selected command relates to a special case; and
executing the special case command.