Scalable compaction in a concurrent transaction processing distributed database
A distributed database compaction system can perform compaction for transactional queries in an asynchronous matter without affecting completion of the queries. The compaction system can implement asynchronous transformation of key pairs in the database, and older keys can be periodically purged using a scheduled compactor. Subsequent queries use the compacted stored keys to perform efficient queries with direct reads of committed transactions and more efficient access to key values stores of the distributed database.
1. A method comprising:
receiving, by a distributed database, a plurality of transactional queries against a key-value database comprising data managed by key-value pairs;
performing, using asynchronous threads of execution nodes of the distributed database, asynchronous transformation and compaction of key-value pairs of data that corresponds to the plurality of transactional queries, the asynchronous threads transforming the key-value pairs from an initial format to a commit format that includes a commit time of data committed to the distributed database, the asynchronous threads compacting the key-value pairs by deleting the key-value pairs in the initial format;
processing, using transactional threads in the execution nodes, the plurality of transactional queries to generate results data, the asynchronous threads performing the asynchronous transformation and compaction while the transactional threads generate the results data; and
storing the results data.
2. The method of claim 1 , further comprising:
receiving a plurality of additional transactional queries against the distributed database.
3. The method of claim 2 , further comprising:
processing, by the transactional threads, the plurality of additional transactional queries using the key-value pairs in the commit format, the transactional threads using commit times in the key-value pairs to generate additional results data; and
storing the additional results data.
4. The method of claim 1 , wherein the distributed database comprises a transaction status table that indicates whether data of transactions has committed to the distributed database.
5. The method of claim 4 , wherein queries for data having key-value pairs in the initial format are executed by identifying transaction identifiers in the key- value pairs and access the transaction status table to determine commit statuses.
6. The method of claim 4 , further comprising:
performing, using a dedicated compactor thread in one of the execution nodes, transformation and compaction of the key-value pairs from the initial format to the commit format.
7. The method of claim 6 , further comprising:
performing, using the dedicated compactor thread, compaction of the transaction status table by deleting key-value pairs in the transaction status table that have corresponding key-value pairs in the commit format.
8. The method of claim 1 , wherein each key-value pair in the initial format includes a transaction identifier value.
9. The method of claim 8 , wherein the asynchronous threads perform transformation by rewriting each key-value pair such that the commit time replaces the transaction identifier value.
10. The method of claim 1 , wherein the key-value pairs having transaction identifiers are deleted by the asynchronous threads.
11. A system comprising:
one or more processors of a machine; and
a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:
receiving, by a distributed database, a plurality of transactional queries against a key-value database comprising data managed by key-value pairs;
performing, using asynchronous threads of execution nodes of the distributed database, asynchronous transformation and compaction of key-value pairs of data that corresponds to the plurality of transactional queries, the asynchronous threads transforming the key-value pairs from an initial format to a commit format that includes a commit time of data committed to the distributed database, the one or mom asynchronous threads compacting the key-value pairs by deleting the key-value pairs in the initial format;
processing, using transactional threads in the execution nodes, the plurality of transactional queries to generate results data, the asynchronous threads performing the asynchronous transformation and compaction while the transactional threads generate the results data; and
storing the results data.
12. The system of claim 11 , the operations further comprising:
receiving a plurality of additional transactional queries against the distributed database.
13. The system of claim 12 , the operations further comprising:
processing, by the transactional threads, the plurality of additional transactional queries using the key-value pairs in the commit format, the transactional threads using commit times in the key-value pairs to generate additional results data; and
storing the additional results data.
14. The system of claim 11 , wherein the distributed database comprises a transaction status table that indicates whether data of transactions has committed to the distributed database.
15. The system of claim 14 , wherein queries for data having key-value pairs in the initial format are executed by identifying transaction identifiers in the key-value pairs and access the transaction status table to determine commit statuses.
16. The system of claim 14 , the operations further comprising:
performing, using a dedicated compactor thread in one of the execution nodes, transformation and compaction of the key-value pairs from the initial format to the commit format.
17. The system of claim 16 , the operations further comprising:
performing, using the dedicated compactor thread, compaction of the transaction status table by deleting key-value pairs in the transaction status table that have corresponding key-value pairs in the commit format.
18. The system of claim 11 , wherein each key-value pair in the initial format includes a transaction identifier value.
19. The system of claim 18 , wherein the asynchronous threads perform transformation by rewriting each key-value pair such that the commit time replaces the transaction identifier value.
20. The system of claim 11 , wherein the key-value pairs having transaction identifiers are deleted by the asynchronous threads.
21. A non-transitory machine storage medium comprising instructions that, when executed by one or more processors of a machine, configure the machine to perform operations comprising:
receiving, by a distributed database, a plurality of transactional queries against a key-value database comprising data managed by key-value pairs;
performing, using asynchronous threads of execution nodes of the distributed database, asynchronous transformation and compaction of key-value pairs of data that corresponds to the plurality of transactional queries, the asynchronous threads transforming the key-value pairs from an initial format to a commit format that includes a commit time of data committed to the distributed database, the asynchronous threads compacting the key-value pairs by deleting the key-value pairs in the initial format;
processing, using transactional threads in the execution nodes, the plurality of transactional queries to generate results data, the asynchronous threads performing the asynchronous transformation and compaction while the transactional threads generate the results data; and
storing the results data.
22. The non-transitory machine storage medium of claim 21 , wherein the operations further comprise:
receiving a plurality of additional transactional queries against the distributed database.
23. The non-transitory machine storage medium of claim 22 , wherein the operations further comprise:
processing, by the transactional threads, the plurality of additional transactional queries using the key-value pairs in the commit format, the transactional threads using commit times in the key-value pairs to generate additional results data; and
storing the additional results data.
24. The non-transitory machine storage medium of claim 21 , wherein the distributed database comprises a transaction status table that indicates whether data of transactions has committed to the distributed database.
25. The non-transitory machine storage medium of claim 24 , wherein queries for data having key-value pairs in the initial format are executed by identifying transaction identifiers in the key-value pairs and access the transaction status table to determine commit statuses.
26. The non-transitory machine storage medium of claim 24 , wherein the operations further comprise:
performing, using a dedicated compactor thread in one of the execution nodes, transformation and compaction of the key-value pairs from the initial format to the commit format.
27. The non-transitory machine storage medium of claim 26 , wherein the operations further comprise:
performing, using the dedicated compactor thread, compaction of the transaction status table by deleting key-value pairs in the transaction status table that have corresponding key-value pairs in the commit format.
28. The non-transitory machine storage medium of claim 21 , wherein each key-value pair in the initial format includes a transaction identifier value.
29. The non-transitory machine storage medium of claim 28 , wherein the asynchronous threads perform transformation by rewriting each key-value pair such that the commit time replaces the transaction identifier value.
30. The non-transitory machine storage medium of claim 21 , wherein the key-value pairs having transaction identifiers are deleted by the asynchronous threads.