Efficient backup of compaction based databases
Embodiments disclosed herein provide systems, methods, and computer readable media for sub-cluster recovery in a data storage environment having a plurality of storage nodes. In a particular embodiment, the method provides scanning data items in the plurality of nodes. While scanning, the method further provides indexing the data items into an index of a plurality of partition groups. Each partition group includes data items owned by a particular one of the plurality of storage nodes. The method then provides storing the index.
1. A method of incrementally backing up compaction-based database systems, the method comprising:
backing up two or more files from a first database system;
after backing up the two or more files, identifying a subsequent file from the first database system for backup;
determining that the subsequent file comprises a compaction of the two or more files; and
based on a determination by query to an ancestry map that the subsequent file corresponds to ancestor information of the two or more files in the ancestry map, refraining to back up the subsequent file, the ancestry map including a list of files backed up to the first database system.
2. The method of claim 1 , wherein backing up the two or more files comprises:
at a first time, backing up one or more first files;
after the first time, identifying a second file for backup; and
determining that the second file does not comprise a compaction of the one or more first files and, responsively, backing up the second file.
3. The method of claim 1 , wherein the subsequent file comprises a compaction of the two or more files when the subsequent file contains only information included in the two or more files.
4. The method of claim 1 , wherein determining by query to the ancestry map comprises:
accessing ancestor information for the subsequent file stored on the ancestry map; and
determining that the ancestor information indicates that the subsequent file was created by compacting the two or more files.
5. The method of claim 4 , further comprising:
when backing up the two or more files, adding ancestry information to the ancestry map; and
after determining that the subsequent file was created by compacting the two or more files, using the ancestry information to determine that the two or more files have already been backed up.
6. The method of claim 5 , further comprising:
adding subsequent ancestry information about the subsequent file to the ancestry map, wherein the subsequent ancestry information indicates that the two or more files are ancestors of the subsequent file.
7. The method of claim 1 , wherein determining by query to the ancestry map that the subsequent file corresponds to ancestor information of the two or more files in the ancestry map comprises:
determining a first maximum timestamp for data entries in the subsequent file; and
determining that the first maximum timestamp is not greater than a maximum timestamp for data entries in the two or more files.
8. The method of claim 1 , further comprising:
identifying the subsequent file for restoration to the first database system;
compacting the two or more files to recreate the subsequent file; and
restoring the subsequent file to the first database system.
9. The method of claim 1 , further comprising:
identifying the subsequent file for restoration to the first database system;
determining that the two or more files are ancestors of the subsequent file; and
restoring the two or more files to the first database system.
10. A system for incrementally backing up compaction-based database systems, the system comprising:
one or more computer readable storage media;
a processing system operatively coupled with the one or more computer readable storage media; and
program instructions stored on the one or more computer readable storage media that, when read and executed by the processing system, direct the processing system to:
back up two or more files from a first database system;
after backing up the two or more files, identify a subsequent file from the first database system for backup;
determine that the subsequent file comprises a compaction of the two or more files; and
based on a determination by query to an ancestry map that the subsequent file corresponds to ancestor information of the two or more files in the ancestry map, refrain to back up the subsequent file, the ancestry map including a list of files backed up to the first database system.
11. The system of claim 10 , wherein to back up the two or more files, the program instructions direct the processing system to:
at a first time, back up one or more first files;
after the first time, identify a second file for backup; and
determine that the second file does not comprise a compaction of the one or more first files and, responsively, backing up the second file.
12. The system of claim 10 , wherein the subsequent file comprises a compaction of the two or more files when the subsequent file contains only information included in the two or more files.
13. The system of claim 10 , wherein to determine by query to the ancestry map, the program instructions direct the processing system to:
access ancestor information for the subsequent file stored on the ancestry map; and
determine that the ancestor information indicates that the subsequent file was created by compacting the two or more files.
14. The system of claim 13 , wherein the program instructions further direct the processing system to:
when backing up the two or more files, add ancestry information to the ancestry map; and
after determining that the subsequent file was created by compacting the two or more files, use the ancestry information to determine that the two or more files have already been backed up.
15. The system of claim 14 , wherein the program instructions further direct the processing system to:
add subsequent ancestry information about the subsequent file to the ancestry map, wherein the subsequent ancestry information indicates that the two or more files are ancestors of the subsequent file.
16. The system of claim 10 , wherein to determine by query to the ancestry map that the subsequent file corresponds to ancestor information of the two or more files in the ancestry map, the program instructions direct the processing system to:
determine a first maximum timestamp for data entries in the subsequent file; and
determine that the maximum max timestamp is not greater than a maximum timestamp for data entries in the two or more files.
17. The system of claim 10 , wherein the program instructions further direct the processing system to:
identify the subsequent file for restoration to the first database system;
compact the two or more files to recreate the subsequent file; and
restore the subsequent file to the first database system.
18. The system of claim 10 , wherein the program instructions further direct the processing system to:
identify the subsequent file for restoration to the first database system;
determine that the two or more files are ancestors of the subsequent file; and
restore the two or more files to the first database system.
19. A computer readable storage medium having program instructions stored thereon for incrementally backing up compaction-based database systems, the program instructions, when executed by a processing system, direct the processing system to:
back up two or more files from a first database system;
after backing up the two or more files, identify a subsequent file from the first database system for backup;
determine that the subsequent file comprises a compaction of the two or more files; and
based on a determination by query to an ancestry map that the subsequent file corresponds to ancestor information of the two or more files in the ancestry map, refrain to back up the subsequent file, the ancestry map including a list of files backed up to the first database system.
20. The system of claim 19 , wherein to back up the two or more files, the program instructions direct the processing system to:
at a first time, back up one or more first files;
after the first time, identify a second file for backup; and
determine that the second file does not comprise a compaction of the one or more first files and, responsively, backing up the second file.