System and method for bulk removal of records in a database
A method of deleting a plurality of records in a log-structured merge (LSM) tree based storage system during a merge operation that writes data from a memory storage to an extent, includes receiving a command to delete a plurality of records, inserting a range tombstone into the memory storage component of the storage system, the range tombstone indicating the plurality of records to be deleted, selecting one or more delayed deletion processes based at least in part on whether the plurality of records is contiguous beyond a predetermined threshold amount, modifying a manifest of the storage system based on the selected delayed deletion process, and performing the merge operation, wherein the plurality of records is not written to the extent during the flush operation based at least in part on the delayed deletion process.
1. A computer-implemented method of deleting records in a log-structured merge (LSM) tree based storage system during a merge operation that writes data from a memory storage component to an extent, comprising:
receiving a command to delete a plurality of records comprising a table, where each of the plurality of records are identified based on a prefix identifier common to each record in the table;
inserting at least one range tombstone into the memory storage component of the storage system, the at least one range tombstone indicating the plurality of records of the table to be deleted;
performing a flush operation that annotates a manifest of the storage system with data indicating the table that is deleted in the storage system and a time the table was deleted from the storage system, wherein the annotation does not affect extent references in the storage system;
bypassing, when reading data from the storage system prior to the merge operation, scanned records that share the prefix identifier in the table; and
performing the merge operation, wherein the plurality of records to be deleted is prevented from being written to a new extent during the merge operation based at least in part on the annotated manifest.
2. A storage system, comprising:
at least one memory storing computer-executable instructions;
a database storing a plurality of records; and
at least one processor, wherein the at least one processor is configured to access the at least one memory and to execute the computer-executable instructions to:
receive a command to delete a plurality of records comprising a table, where each of the plurality of records are identified based on a prefix identifier common to each record in the table;
insert at least one range tombstone into a memory storage component of the storage system, the range tombstone indicating the plurality of records of the table to be deleted;
perform a flush operation that annotates a manifest of the storage system with data indicating the table that is deleted in the storage system and a time the table was deleted from the storage system, wherein the annotation does not affect extent references in the storage system;
bypass, when reading data from the storage system prior to any merge operation, scanned records that share the prefix identifier in the table; and
perform a merge operation, wherein the indicated plurality of records to be deleted is prevented from being written to a new extent during a flush the merge operation based at least in part on the annotated manifest.
3. A computer-implemented method of deleting records in a log-structured merge (LSM) tree based storage system during a merge operation that writes data from a memory storage component to an extent, comprising:
receiving a command to delete a plurality of records comprising a table, where each of the plurality of records are identified based on a prefix identifier common to each record in the table;
inserting at least one range tombstone into the memory storage component of the storage system, the at least one range tombstone indicating the plurality of records of the table to be deleted;
determining the indicated plurality of records to be deleted are within a large number of small, discontiguous ranges and contain a few number of records therein;
performing a flush operation that annotates a manifest of the storage system with data that indicates an object that is deleted from the storage system and a time that the object was deleted from the storage system, wherein the annotation does not affect extent references in the storage system;
bypassing, when reading data from the storage system prior to the merge operation, scanned records that share the prefix identifier in the table; and
performing the merge operation, wherein the plurality of records to be deleted is prevented from being written to a new extent during the merge operation based at least in part on the annotated manifest.