Server group fetch in database backup
In some examples, a method of performing a backup of a group of relational databases comprises identifying database files to be fetched in the group of relational databases; grouping the identified database files into batches; based on configuration parameters of the identified database files, identifying, among the batches, a sub-set of batches of database files that are eligible to be fetched in parallel for the backup; configuring a single fetch call to a call stack to fetch the sub-set of eligible batches; and determining a push or pull model for the configured single fetch call based at least in part on feedback from a most resource-constrained element in the call stack.
1. A method of performing a backup of a group of relational databases, the method comprising:
identifying database files to be fetched from the group of relational databases;
grouping the identified database files into batches;
based on configuration parameters of the identified database files, identifying, among the batches, a sub-set of batches of database files that are eligible to be fetched in parallel for the backup;
configuring a single fetch call to a call stack of a data management node, the single fetch call to fetch the sub-set of eligible batches in parallel from the group of relational databases;
obtaining feedback from an element in the call stack of the data management node, wherein the feedback indicates resource constraints associated with the element;
determining whether to use a push model or a pull model for execution of the configured single fetch call to fetch the sub-set of eligible batches in parallel based at least in part on the feedback from the element in the call stack of the data management node, wherein both the push model and the pull model are operable to fetch the sub-set of batches of database files from a third-party agent that is separate from the group of relational databases and configured to read database files from the group of relational databases and transfer the database files to a server running on a data management node; and
executing the configured single fetch call based at least in part on the determined model, wherein:
when the determined model is the push model, executing the configured single fetch call according to the determined model comprises causing the third-party agent to push the sub-set of batches of database files to the data management node, and
when the determined model is the pull model, executing the configured single fetch call according to the determined model comprises causing the data management node to pull the sub-set of batches of database files from the third-party agent in accordance with a determination made by the data management node.
2. The method of claim 1 , further comprising:
splitting the database files in the sub-set of eligible batches into data chunks; and
eliminating a request for one or more data chunks from the configured single fetch call.
3. The method of claim 1 , further comprising reading the database files in the sub-set of batches in parallel and transmitting database file data and metadata to a patch file server.
4. The method of claim 2 , wherein eliminating the request for one or more data chunks from the configured single fetch call is based at least in part on a changed-block-tracking (CBT) analysis.
5. The method of claim 3 , wherein reading the database files in parallel includes allocating a same amount of read buffer per file.
6. The method of claim 3 , wherein the patch file server is located on a data node in a cluster of cooperating data nodes.
7. The method of claim 3 , further comprising generating a snapshot based on data or metadata of a patch file hosted at the patch file server.
8. A backup system comprising:
at least one processor; and
a memory storing instructions which, when executed by the at least one processor, cause the system to perform operations in a method of performing a backup of a group of relational databases, the operations comprising, at least:
identifying database files to be fetched from the group of relational databases;
grouping the identified database files into batches;
based on configuration parameters of the identified database files, identifying, among the batches, a sub-set of batches of database files that are eligible to be fetched in parallel for the backup;
configuring a single fetch call to a call stack of a data management node, the single fetch call to fetch the sub-set of eligible batches in parallel from the group of relational databases;
obtaining feedback from an element in the call stack of the data management node, wherein the feedback indicates resource constraints associated with the element;
determining whether to use a push model or a pull model for execution of the configured single fetch call to fetch the sub-set of eligible batches in parallel based at least in part on the feedback from the element in the call stack of the data management node, wherein both the push model and the pull model are operable to fetch the sub-set of batches of database files from a third-party agent that is separate from the group of relational databases and configured to read database files from the group of relational databases and transfer the database files to a server running on a data management node; and
executing the configured single fetch call based at least in part on the determined model, wherein:
when the determined model is the push model, executing the configured single fetch call according to the determined model comprises causing the third-party agent to push the sub-set of batches of database files to the data management node, and
when the determined model is the pull model, executing the configured single fetch call according to the determined model comprises causing the data management node to pull the sub-set of batches of database files from the third-party agent in accordance with a determination made by the data management node.
9. The system of claim 8 , wherein the operations further comprise:
splitting database files in the sub-set of eligible batches into data chunks; and
eliminating one or more data chunks from the configured single fetch call.
10. The system of claim 9 , wherein eliminating one or more data chunks from the configured single fetch call is based at least in part on a changed-block-tracking (CBT) analysis.
11. The system of claim 10 , wherein the operations further comprise reading the database files in the sub-set of batches in parallel and transmitting database file data and metadata to a patch file server.
12. The system of claim 11 , wherein reading the database files in parallel includes allocating a same amount of read buffer per file.
13. The system of claim 11 , wherein the patch file server is located on a data node in a cluster of cooperating data nodes.
14. The system of claim 11 , wherein the operations further comprise generating a snapshot based on data or metadata of a patch file hosted at the patch file server.
15. A non-transitory machine-readable medium including instructions which, when read by a machine, cause a machine to perform operations in a method of performing a backup of a group of relational databases, the operations comprising, at least:
identifying database files to be fetched from the group of relational databases;
grouping the identified database files into batches;
based on configuration parameters of the identified database files, identifying, among the batches, a sub-set of batches of database files that are eligible to be fetched in parallel for the backup;
configuring a single fetch call to a call stack of a data management node, the single fetch call to fetch the sub-set of eligible batches in parallel from the group of relational databases;
obtaining feedback from an element in the call stack of the data management node, wherein the feedback indicates resource constraints associated with the element;
determining whether to use a push model or a pull model for execution of the configured single fetch call to fetch the sub-set of eligible batches in parallel based at least in part on the feedback from the element in the call stack of the data management node, wherein both the push model and the pull model are operable to fetch the sub-set of batches of database files from a third-party agent that is separate from the group of relational databases and configured to read database files from the group of relational databases and transfer the database files to a server running on a data management node; and
executing the configured single fetch call based at least in part on the determined model, wherein:
when the determined model is the push model, executing the configured single fetch call according to the determined model comprises causing the third-party agent to push the sub-set of batches of database files to the data management node, and
when the determined model is the pull model, executing the configured single fetch call according to the determined model comprises causing the data management node to pull the sub-set of batches of database files from the third-party agent in accordance with a determination made by the data management node.
16. The medium of claim 15 , wherein the operations further comprise:
splitting database files in the sub-set of eligible batches into data chunks; and
eliminating one or more data chunks from the configured single fetch call.
17. The medium of claim 15 , wherein the operations further comprise reading the database files in the sub-set of batches in parallel and transmitting database file data and metadata to a patch file server.
18. The medium of claim 16 , wherein eliminating one or more data chunks from the configured single fetch call is based at least in part on a changed-block-tracking (CBT) analysis.
19. The medium of claim 17 , wherein reading the database files in parallel includes allocating a same amount of read buffer per file.
20. The medium of claim 17 , wherein the patch file server is located on a data node in a cluster of cooperating data nodes.