Targeted sweep method for key-value data storage

1. A computer-implemented method comprising:

writing an entry for each of one or more write commands of a write transaction to an end of a targeted sweep queue in a database having a key value store,

the entry for each of the one or more write commands comprising metadata including: data identifying a cell to which the one or more write commands relate, a start timestamp of the write transaction, and information identifying a type of the write transaction wherein writing the entry for each of the one or more write commands to the end of the targeted, sweep queue comprises:

partitioning a list of entries dependent on one or more parameters of a pre-defined targeted sweep strategy;

storing, as sweepable timestamp data of the targeted sweep queue, a fine partition timestamp derived from the start timestamp of the write transaction; and

storing, as sweepable cell data of the targeted sweep queue, for each entry within the partition, the metadata for each entry; and

performing the targeted sweep of the database by:

starting from a start of the targeted sweep queue,

selecting an entry from the targeted sweep queue based on a current sweep timestamp, and

using a single ranged tombstone to delete prior versions of a cell to which a write command of the entry relates.

2. The computer-implemented method of claim 1 , further comprising deleting the selected entry from the targeted sweep queue and selecting a next entry from the targeted sweep queue.

3. The computer-implemented method of claim 1 , further comprising committing the write transaction to the database and persisting each of the one or more write commands to the key value store.

4. The computer-implemented method of claim 3 , wherein performing the targeted sweep of the database comprises:

checking a commit timestamp of a write transaction associated with the write command of the entry; and

if the write transaction is not committed:

aborting the write transaction,

deleting the write command from the key value store,

popping the entry from the targeted sweep queue, and

selecting a next entry from the targeted sweep queue.

5. The computer-implemented method of claim 3 , wherein performing the targeted sweep of the database comprises:

checking a commit timestamp of a write transaction associated with the write command of the entry; and

if the commit timestamp is greater than or equal to the current sweep timestamp, pausing the targeted sweep.

6. The computer-implemented method of claim 3 , wherein performing the targeted sweep of the database comprises:

checking a commit timestamp of a write transaction associated with the write command of the entry; and

if the commit timestamp is less than the current sweep timestamp, using the single ranged tombstone to delete the prior versions of the cell.

7. The computer-implemented method of claim 1 , wherein the pre-defined targeted sweep strategy comprises a conservative strategy and performing the targeted sweep of the database having the conservative strategy comprises:

writing a deletion sentinel; and

using the single ranged tombstone to delete the prior versions of the cell without deleting the deletion sentinel or the write command.

8. The computer-implemented method of claim 1 , wherein the pre-defined targeted sweep strategy comprises a thorough strategy and performing the targeted sweep of the database having the thorough strategy comprises:

if the write command of the entry is a tombstone:

using the single ranged tombstone to delete the prior versions of the cell including any existing deletion sentinel and the write command.

9. The computer-implemented method of claim 1 , wherein the pre-defined targeted sweep strategy comprises a thorough strategy and performing the targeted sweep of the database having the thorough strategy comprises:

if the write command of the entry is not a tombstone:

using the single ranged tombstone to delete the prior versions of the cell including any existing deletion sentinel without deleting the write command.

10. The computer-implemented method of claim 1 , wherein performing the targeted sweep of the database further comprises:

when the current sweep timestamp is greater than a start timestamp of the entry, using the single ranged tombstone to delete the prior versions of the cell.

11. The computer-implemented method of claim 1 , wherein performing the targeted sweep of the database further comprises:

when the current sweep timestamp is less than or equal to a start timestamp of the entry, pausing the targeted sweep.

12. The computer-implemented method of claim 1 , wherein selecting the entry from the targeted sweep queue comprises:

selecting a stored sweep progress timestamp;

identifying a batch of entries from the targeted sweep queue having a first start timestamp greater than the stored sweep progress timestamp and a second start time less than the current sweep timestamp;

partitioning entries from the identified batch of entries based on a cell to which a write command of each entry relates; and

for each partition:

determining an entry having a greatest start timestamp of write transactions to the cell, and

determining the single ranged tombstone based on the greatest start timestamp.

13. The computer-implemented method of claim 12 , further comprising, after performing the targeted sweep of the database, updating the stored sweep progress timestamp.

14. The computer-implemented method of claim 1 , wherein the current sweep timestamp is lesser of: a start timestamp of a longest running write transaction and an earliest readable time.

15. The computer-implemented method of claim 1 , wherein the fine partition timestamp comprises an integer division of the start timestamp by a first pre-determined value.

16. The computer-implemented method of claim 1 , wherein the coarse partition timestamp comprises an integer division of the start timestamp by a second pre-determined value, the second pre-determined value greater than the first pre-determined value.

17. The computer-implemented method of claim 1 , comprising, for each write command, determining a shared number from a hash identifying the cell.

18. The computer-implemented method of claim 1 , wherein the fine partition timestamp is indexed by the one or more parameters and by a coarse partition timestamp derived from the start timestamp of the write transaction, and wherein the metadata is indexed by the one or more parameters and by the fine partition timestamp.

19. A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to perform a method comprising:

writing an entry for each of one or more write commands of a write transaction to an end of a targeted sweep queue in a database having a key value store,

the entry for each of the one or more write commands comprising metadata including: data identifying a cell to which the one or more write commands relate, a start timestamp of the write transaction, and information identifying a type of the write transaction, wherein writing the entry for each of the one or more write commands to the end of the targeted sweep queue comprises:

partitioning a list of entries dependent on one or more parameters of a pre-defined targeted sweep strategy;

storing, as sweepable timestamp data of the targeted sweep queue, a fine partition timestamp derived from the start timestamp of the write transaction; and

storing, as sweepable cell data of the targeted sweep queue, for each entry within the partition, the metadata for each entry; and

performing the targeted sweep of the database by:

starting from a start of the targeted sweep queue,

selecting an entry from the targeted sweep queue based on a current sweep timestamp, and

using a single ranged tombstone to delete prior versions of a cell to which a write command of the entry relates.

20. A system comprising:

one or more physical processors;

a memory storing instructions that, when executed by the one or more physical processors, cause the system to perform a method comprising:

writing an entry for each of one or more write commands of a write transaction to an end of a targeted sweep queue in a database having a key value store,

the entry for each of the one or more write commands comprising metadata including: data identifying a cell to which the one or more write commands relate, a start timestamp of the write transaction, and information identifying a type of the write transaction, wherein writing the entry for each of the one or more write commands to the end of the targeted sweep queue comprises:

partitioning a list of entries dependent on one or more parameters of a pre-defined targeted sweep strategy;

storing, as sweepable timestamp data of the targeted sweep queue, a fine partition timestamp derived from the start timestamp of the write transaction; and

storing, as sweepable cell data of the targeted sweep queue, for each entry within the partition, the metadata for each entry; and

performing the targeted sweep of the database by:

starting from a start of the targeted sweep queue,

selecting an entry from the targeted sweep queue based on a current sweep timestamp, and

using a single ranged tombstone to delete prior versions of a cell to which a write command of the entry relates.