Targeted sweep method for key-value data storage
A computer-implemented method for targeted sweep of a key-value data storage is provided. The method comprises before a write transaction to a database having a key value store commits, and before each of one or more write commands of the write transaction are persisted to the key value store, writing an entry for each of the one or more write commands to an end of a targeted sweep queue, the entry comprising metadata including: data identifying a cell to which the write command relates, a start timestamp of the write transaction, and information identifying a type of the write transaction.
1. A computer-implemented method comprising:
writing entries corresponding to write commands of a write transaction to a queue in a database, wherein each entry from the entries comprises metadata including:
data identifying a cell to which a write command of the write commands relate, and
a timestamp of the write transaction;
selectively performing a sweep of the database by deleting any previous versions of respective particular cells based on whether most recent versions of the respective particular cells been persisted into the database, wherein the selectively performing of the sweep comprises delaying a deletion for any candidate cell of the particular cells in which a most recent version of the any candidate cell is unpersisted into the database until the most recent version of the any candidate cell has been persisted into the database.
2. The computer-implemented method of claim 1 , wherein the particular cells are within a first range of timestamps; and the sweep comprises a first sweep; and the computer-implemented method further comprising:
commencing a second sweep of cells within a second range of timestamps while pausing the deletion for any candidate cells within the first range of timestamps.
3. The computer-implemented method of claim 1 , wherein the writing of the entries corresponding to write commands is in response to receiving a database request; and the writing of the entries occurs prior to the entries being persisted to the database.
4. The computer-implemented method of claim 1 , wherein, if the most recent versions are persisted into the database, the most recent versions are persisted into a key-value store of the database.
5. The computer implemented method of claim 1 , wherein the deletion of the previous versions comprises retaining any read-only cells while deleting any writable cells.
6. The computer implemented method of claim 1 , further comprising:
partitioning the entries into different segments of the database.
7. The computer implemented method of claim 1 , wherein the performing of the sweep is based on any cells satisfying a sweep strategy and a range of timestamps indicative of a start time at which the cells commenced writing.
8. The computer-implemented method of claim 1 , wherein the performing of the sweep is based on a hash of the metadata.
9. The computer implemented method of claim 1 , wherein the performing of the sweep comprises grouping together any cells, to be swept in a single iteration, satisfying a sweep strategy and a range of timestamps indicative of a start time at which the cells commenced writing.
10. The computer implemented method of claim 1 , wherein the performing of the sweep comprises separately grouping any first rows permitting read-only operations and any second rows prohibiting read-only operations, and selectively sweeping over the any first rows separately from the any second rows.
11. A system comprising:
one or more physical processors; and
a memory storing instructions that, when executed by the one or more physical processors, cause the system to:
writing entries corresponding to write commands of a write transaction to a queue in a database, wherein each entry from the entries comprises metadata including:
data identifying a cell to which a write command of the write commands relate, and
a timestamp of the write transaction;
selectively performing a sweep of the database by deleting any previous versions of respective particular cells based on whether most recent versions of the respective particular cells have been persisted into the database, wherein the selectively performing of the sweep comprises delaying a deletion for any candidate cell of the particular cells in which a most recent version of the any candidate cell is unpersisted into the database until the most recent version of the any candidate cell has been persisted into the database.
12. The system of claim 11 , wherein the particular cells are within a first range of timestamps; and the sweep comprises a first sweep; and the instructions further cause the system to:
commence a second sweep of cells within a second range of timestamps while pausing the deletion for any candidate cells within the first range of timestamps.
13. The system of claim 11 , wherein the writing of the entries corresponding to write commands is in response to receiving a database request; and the writing of the entries occurs prior to the entries being persisted to the database.
14. The system of claim 11 , wherein, if the most recent versions are persisted into the database, the most recent versions are persisted into a key-value store of the database.
15. The system of claim 11 , wherein the deletion of the previous versions comprises retaining any read-only cells while deleting any writable cells.
16. The system of claim 11 , wherein the instructions further cause the system to:
partition the entries into different segments of the database.
17. The system of claim 11 , wherein the performing of the sweep is based on any cells satisfying a sweep strategy and a range of timestamps indicative of a start time at which the cells commenced writing.
18. The system of claim 11 , wherein the performing of the sweep is based on a hash of the metadata.
19. The system of claim 11 , wherein the selectively performing of the sweep is based on a comparison between a sweep timestamp and commit timestamps of the particular cells, wherein the sweep timestamp is based upon one or more access control attributes of the particular cells.