Checkpoint and restore based startup of executor nodes of a distributed computing engine for processing queries
A system performs efficient startup of executors of a distributed computing engine used for processing queries, for example, database queries. The system starts an executor node and processes a set of queries using the executor node to warm up the executor node. The system performs a checkpoint of the warmed-up executor node to create an image. The image is restored in the target executor nodes. The system may store a checkpoint image for each configuration of an executor node. The configuration is determined based on various factors including the hardware of the executor node, memory allocation of the processes, and so on. The user or restore based on checkpoint images improves efficiency of execution of the startup of executor nodes.
1. A method, comprising:
starting execution of one or more processes for processing queries in a source executor node of a distributed computing engine configured for receiving and processing queries, the one or more processes including a runtime execution engine of a programming language used for implementing instructions for the processing queries;
sending the queries to the source executor node for execution of the queries on the source executor node;
performing checkpoint of the one or more processes executing on the source executor node to generate a checkpoint image, wherein the execution of the queries on the source executor node causes just-in-time compilation of at least a portion of code used for the processing queries, the just-in-time compilation causing generation of native instructions for the source executor node, wherein the native instructions are stored in the checkpoint image;
storing the checkpoint image in a checkpoint image repository in the source executor node;
providing the checkpoint image from the checkpoint image repository to a plurality of target executor nodes of the distributed computing engine;
for each target executor node of the plurality of target executor nodes of the distributed computing engine, performing restore of checkpoint image to restore a check point image on said each target executor node of the plurality of target executor nodes based on the native instructions of the provided checkpoint image; and
processing additional queries using the plurality of target executor nodes in the distributed computing engine to perform operations related to creation of data structures after the performing the restore of checkpoint image.
2. The method of claim 1 , wherein the execution of the queries on the source executor node causes loading of classes used for execution of the queries, wherein the classes are stored in the checkpoint image.
3. The method of claim 1 , wherein the execution of the queries on the source executor node causes initialization of variables and creation of data structures used for processing the queries, wherein the variables and data structures are stored in the checkpoint image.
4. The method of claim 1 , wherein the checkpoint image repository stores a plurality of checkpoint images, each checkpoint image associated with a particular configuration of executor nodes.
5. The method of claim 4 , wherein a configuration of an executor node is associated with one or more of: a CPU (central processing unit) type of the executor node, an amount of memory allocation of the executor node, or a version of a particular software running on the executor node.
6. The method of claim 1 , further comprising:
generating a plurality of checkpoint images for a configuration of executor nodes, wherein different checkpoint images of the plurality of checkpoint images have different address space layouts.
7. The method of claim 1 , further comprising:
responsive to restoring the checkpoint image on a target executor node, modifying one of more random numbers used by instructions for processing queries.
8. The method of claim 1 , further comprising:
detecting a new configuration of an executor node;
triggering generation of a new checkpoint image corresponding to the new configuration; and
storing the new checkpoint image in the checkpoint image repository.
9. A non-transitory computer readable medium comprising stored instructions, the stored instructions when executed by at least one processor of one or more computing devices, cause the one or more computing devices to:
start execution of one or more processes for processing queries in a source executor node of a distributed computing engine configured for receiving and processing queries, the one or more processes including a runtime execution engine of a programming language used for implementing instructions for the processing queries;
send the queries to the source executor node for execution of the queries on the source executor node;
perform checkpoint of the one or more processes executing on the source executor node to generate a checkpoint image, wherein the execution of the queries on the source executor node causes just-in-time compilation of at least a portion of code used for the processing queries, the just-in-time compilation causing generation of native instructions for the source executor node, wherein the native instructions are stored in the checkpoint image;
store the checkpoint image in a checkpoint image repository in the source executor node;
provide the checkpoint image from the checkpoint image repository to a plurality of target executor nodes of the distributed computing engine;
for each target executor node of the plurality of target executor nodes of the distributed computing engine, perform restore of checkpoint image to restore a check point image on said each target executor node of the plurality of target executor nodes based on the native instructions of the provided checkpoint image; and
process additional queries using the plurality of target executor nodes in the distributed computing engine to perform operations related to creation of data structures after the performing the restore of checkpoint image.
10. The non-transitory computer readable medium of claim 9 , wherein the execution of the queries on the source executor node causes loading of classes used for execution of the queries, wherein the classes are stored in the checkpoint image.
11. The non-transitory computer readable medium of claim 9 , wherein the execution of the queries on the source executor node causes initialization of variables and creation of data structures used for processing the queries, wherein the variables and data structures are stored in the checkpoint image.
12. The non-transitory computer readable medium of claim 9 , wherein the checkpoint image repository stores a plurality of checkpoint images, each checkpoint image associated with a particular configuration of executor nodes.
13. The non-transitory computer readable medium of claim 12 , wherein a configuration of an executor node is associated with one or more of: a CPU (central processing unit) type of the executor node, an amount of memory allocation of the executor node, or a version of a particular software running on the executor node.
14. The non-transitory computer readable medium of claim 9 , the instructions causing the one or more computing devices to:
generate a plurality of checkpoint images for a configuration of executor nodes, wherein different checkpoint images of the plurality of checkpoint images have different address space layouts.
15. The non-transitory computer readable medium of claim 9 , the instructions causing the one or more computing devices to:
responsive to restoring the checkpoint image on a target executor node, modify one of more random numbers used by instructions for processing queries.
16. The non-transitory computer readable medium of claim 9 , the instructions causing the one or more computing devices to:
detect a new configuration of an executor node;
trigger generation of a new checkpoint image corresponding to the new configuration;
and store the new checkpoint image in the checkpoint image repository.
17. A computer system, comprising:
a computer processor; and
a non-transitory computer-readable storage medium comprising instructions that when executed by the computer processor, cause the computer system to:
start execution of one or more processes for processing queries in a source executor node of a distributed computing engine configured for receiving and processing queries, the one or more processes including a runtime execution engine of a programming language used for implementing instructions for the processing queries;
send the queries to the source executor node for execution of the queries on the source executor node;
perform checkpoint of the one or more processes executing on the source executor node to generate a checkpoint image, wherein the execution of the queries on the source executor node causes just-in-time compilation of at least a portion of code used for the processing queries, the just-in-time compilation causing generation of native instructions for the source executor node, wherein the native instructions are stored in the checkpoint image;
store the checkpoint image in a checkpoint image repository in the source executor node;
provide the checkpoint image from the checkpoint image repository to a plurality of target executor nodes of the distributed computing engine;
for each target executor node of the plurality of target executor nodes of the distributed computing engine, perform restore of checkpoint image to restore a check point image on said each target executor node of the plurality of target executor nodes based on the native instructions of the provided checkpoint image; and
process additional queries using the plurality of target executor nodes in the distributed computing engine to perform operations related to creation of data structures after the performing the restore of checkpoint image.
18. The computer system of claim 17 , wherein the instructions further cause the computer system to:
detect a new configuration of an executor node;
trigger generation of a new checkpoint image corresponding to the new configuration;
and store the new checkpoint image in the checkpoint image repository.
19. The computer system of claim 17 , wherein the execution of the queries on the source executor node causes loading of classes used for execution of the queries, wherein the classes are stored in the checkpoint image.
20. The computer system of claim 17 , wherein the execution of the queries on the source executor node causes initialization of variables and creation of data structures used for processing the queries, wherein the variables and data structures are stored in the checkpoint image.