Targeted sweep method for key-value data storage
A computer-implemented method for targeted sweep of a key-value data storage is provided. The method comprises before a write transaction to a database having a key value store commits, and before each of one or more write commands of the write transaction are persisted to the key value store, writing an entry for each of the one or more write commands to an end of a targeted sweep queue, the entry comprising metadata including: data identifying a cell to which the write command relates, a start timestamp of the write transaction, and information identifying a type of the write transaction.
1. A computer-implemented method comprising:
writing first entries corresponding to write commands of a write transaction to a targeted sweep queue in a database, wherein each entry from the first entries comprises metadata including:
data identifying a cell to which a write command of the write commands relate, and
a timestamp of the write transaction;
performing the targeted sweep of the database by deleting previous versions of cells associated with second entries, the second entries being selected from the first entries based on respective timestamps of the second entries;
determining whether an additional entry within a range of the timestamps has been written and persisted into the database; and
in response to determining that the additional entry has been written and persisted into the database, deleting any remaining entries prior to the additional entry.
2. The computer-implemented method of claim 1 , further comprising storing a progress of the targeted sweep to indicate a most recent timestamp associated with the second entries.
3. The computer-implemented method of claim 1 , wherein write commands associated with the second entries have been persisted into the database; and further comprising:
refraining from deleting previous versions associated with third entries, out of the first entries, wherein the third entries have not been persisted into the database.
4. The computer-implemented method of claim 1 , wherein the deletion of the previous versions includes writing sentinel values to the deleted previous versions and replacing the deleted previous versions of cells with a tombstone.
5. The computer-implemented method of claim 4 , wherein the deletion of the previous versions includes retaining most recent cells corresponding to the second entries.
6. The computer implemented method of claim 4 , wherein the deletion of the previous versions includes retaining the sentinel values and the write commands.
7. The computer implemented method of claim 6 , further comprising:
receiving a read request to obtain the second entries; and
indicating that the read request fails rather than displaying that no data is present.
8. The computer-implemented method of claim 1 , wherein the deletion of the second entries is dependent upon a determination of whether the write commands include a tombstone, as indicated by the metadata.
9. The computer-implemented method of claim 1 , wherein the deletion of the second entries includes deleting a cell corresponding to a most recent timestamp if the write commands include a tombstone and retaining a cell corresponding to a most recent timestamp otherwise.
10. The computer-implemented method of claim 1 , wherein the timestamps of the second entries indicate times at which previous and current values of associated cells have respective latest timestamps that indicate a same time or a time previous to a starting timestamp of a longest running write transaction of the database.
11. A system comprising:
one or more physical processors;
a memory storing instructions that, when executed by the one or more physical processors, cause the system to:
write first entries corresponding to write commands of a write transaction to a targeted sweep queue in a database, wherein each entry from the first entries comprises metadata including:
data identifying a cell to which a write command of the write commands relate, and
a timestamp of the write transaction;
perform the targeted sweep of the database by deleting previous versions of cells associated with second entries, the second entries being selected from the first entries based on respective timestamps of the second entries;
determine whether an additional entry within the range of the timestamps has been written and persisted into the database; and
in response to determining that the additional entry has been written and persisted into the database, delete any remaining entries prior to the additional entry.
12. The system of claim 11 , wherein the instructions further cause the system to:
storing a progress of the targeted sweep to indicate a most recent timestamp associated with the second entries.
13. The system of claim 11 , wherein write commands associated with the second entries have been persisted into the database; and the instructions further cause the system to:
refrain from deleting previous versions associated with third entries, out of the first entries, wherein the third entries have not been persisted into the database.
14. The system of claim 11 , wherein the deletion of the previous versions includes writing sentinel values to the deleted previous versions and replacing the deleted previous versions of cells with a tombstone.
15. The system of claim 14 , wherein the deletion of the previous versions includes retaining most recent cells corresponding to the second entries.
16. The system of claim 14 , wherein the deletion of the previous versions includes retaining the sentinel values and the write commands.
17. The system of claim 16 , wherein the instructions further cause the system to:
receive a read request to obtain the second entries; and
indicate that the read request fails rather than displaying that no data is present.
18. A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to perform a method comprising:
writing first entries corresponding to write commands of a write transaction to a targeted sweep queue in a database, wherein each entry from the first entries comprises metadata including:
data identifying a cell to which a write command of the write commands relate, and
a timestamp of the write transaction; and
performing the targeted sweep of the database by deleting previous versions of cells associated with second entries, the second entries being selected from the first entries based on respective timestamps of the second entries;
determining whether an additional entry within a range of the timestamps has been written and persisted into the database; and
in response to determining that the additional entry has been written and persisted into the database, deleting any remaining entries prior to the additional entry.