Intelligent data slicing
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for backing up a data object in blocks. One of the methods includes determining, for a data object of a backup process, whether a size of the data object or an estimated backup time of the data object satisfies a criterion that, when satisfied, indicates that at least two blocks of the data object should be separately fetched from the source system by different workers; determining one or more markers for end points of the at least two blocks using data from a prior backup of the data object; and causing, at least partially concurrently for two or more blocks from the at least two blocks, a respective backup worker to fetch the respective block from a source system using at least one marker from the one or more markers that defines an end of the respective block.
1. A computer-implemented method comprising:
determining, for a data object of a backup process for a source system, whether a size of the data object or an estimated backup time of the data object satisfies a criterion that, when satisfied, indicates that at least two blocks of the data object should be separately fetched from the source system by different workers;
determining one or more markers for end points of the at least two blocks using data from a prior backup of the data object; and
causing, at least partially concurrently for two or more blocks from the at least two blocks, a respective backup worker to fetch the respective block from the source system using at least one marker from the one or more markers that defines an end of the respective block.
2. The method of claim 1 , wherein determining the one or more markers comprises:
determining, using the data from the prior backup, a quantity of the at least two blocks of the data object that should be separately fetched from the source system; and
determining the one or more markers that each define at least one endpoint of a corresponding block using the quantity of the at least two blocks, a size of the data object and the data from the prior backup.
3. The method of claim 1 , comprising:
receiving a fetched block from the at least two blocks out of an order according to the marker for the fetched block;
stitching, according to the marker for the fetched block and other markers for the other fetched blocks, the fetched block into a file for the data object;
after stitching all blocks from the at least two blocks in the file for the data object, sorting the data in the file for the data object; and
storing, in memory, the sorted data in the file for the data object.
4. The method of claim 3 , comprising:
comparing, using the sorted data in the file for the data object, at least a first portion of a current backup of the data object with at least a second portion of a second prior backup of the data object; and
providing, as output, data for the comparison.
5. The method of claim 4 , wherein the prior backup and the second prior backup comprise the same backup.
6. The method of claim 1 , wherein determining whether the size of the data object satisfies the criterion occurs during the prior backup of the data object.
7. The method of claim 1 , wherein determining whether the size of the data object satisfies the criterion occurs during a current backup process of the data object that includes causing the respective backup worker to fetch the respective block from the source system.
8. The method of claim 1 , wherein determining the one or more markers occurs during the prior backup of the data object.
9. The method of claim 1 , wherein determining the one or more markers occurs during a current backup process of the data object that includes causing the respective backup worker to fetch the respective block from the source system.
10. The method of claim 1 , wherein:
the data object comprises a table;
each of the at least two blocks comprises a logic block of one or more records from the table; and
a marker from the one or more markers comprises a record identifier for a record in the table.
11. The method of claim 1 , wherein the prior backup comprises a most recent backup sequentially of the data object for the source system.
12. The method of claim 1 , wherein a first quantity of the one or more markers is one less than a second quantity of the two or more blocks.
13. A system comprising one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
determining, for a data object of a backup process for a source system, whether a size of the data object or an estimated backup time of the data object satisfies a criterion that, when satisfied, indicates that at least two blocks of the data object should be separately fetched from the source system by different workers;
determining one or more markers for end points of the at least two blocks using data from a prior backup of the data object; and
causing, at least partially concurrently for two or more blocks from the at least two blocks, a respective backup worker to fetch the respective block from the source system using at least one marker from the one or more markers that defines an end of the respective block.
14. The system of claim 13 , wherein determining the one or more markers comprises:
determining, using the data from the prior backup, a quantity of the at least two blocks of the data object that should be separately fetched from the source system; and
determining the one or more markers that each define at least one endpoint of a corresponding block using the quantity of the at least two blocks, a size of the data object and the data from the prior backup.
15. The system of claim 13 , the operations comprising:
receiving a fetched block from the at least two blocks out of an order according to the marker for the fetched block;
stitching, according to the marker for the fetched block and other markers for the other fetched blocks, the fetched block into a file for the data object;
after stitching all blocks from the at least two blocks in the file for the data object, sorting the data in the file for the data object; and
storing, in memory, the sorted data in the file for the data object.
16. The system of claim 15 , the operations comprising:
comparing, using the sorted data in the file for the data object, at least a first portion of a current backup of the data object with at least a second portion of a second prior backup of the data object; and
providing, as output, data for the comparison.
17. The system of claim 16 , wherein the prior backup and the second prior backup comprise the same backup.
18. The system of claim 13 , wherein determining whether the size of the data object satisfies the criterion occurs during the prior backup of the data object.
19. The system of claim 13 , wherein determining whether the size of the data object satisfies the criterion occurs during a current backup process of the data object that includes causing the respective backup worker to fetch the respective block from the source system.
20. One or more computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
determining, for a data object of a backup process for a source system, whether a size of the data object or an estimated backup time of the data object satisfies a criterion that, when satisfied, indicates that at least two blocks of the data object should be separately fetched from the source system by different workers;
determining one or more markers for end points of the at least two blocks using data from a prior backup of the data object; and
causing, at least partially concurrently for two or more blocks from the at least two blocks, a respective backup worker to fetch the respective block from the source system using at least one marker from the one or more markers that defines an end of the respective block.