Scalable compaction for a distributed database
The subject technology performs, using asynchronous threads of execution nodes of a distributed database, asynchronous transformation and compaction of key-value pairs of data that correspond to a plurality of transactional queries, the asynchronous threads transforming the key-value pairs from an initial format to a commit format that includes a commit time of data committed to the distributed database, the asynchronous threads compacting the key-value pairs by deleting the key-value pairs in the initial format. The subject technology processes, using transactional threads in the execution nodes, the plurality of transactional queries to generate results data, the asynchronous threads performing the asynchronous transformation and compaction while the transactional threads generate the results data. The subject technology stores the results data.
1. A method comprising:
performing, using asynchronous threads of execution nodes of a distributed database, asynchronous transformation and compaction of key-value pairs of data that correspond to a plurality of transactional queries, the asynchronous threads transforming the key-value pairs from an initial format to a commit format that includes a commit time of data committed to the distributed database, the asynchronous threads compacting the key-value pairs by deleting the key-value pairs in the initial format;
processing, using transactional threads in the execution nodes, the plurality of transactional queries to generate results data, the asynchronous threads performing the asynchronous transformation and compaction while the transactional threads generate the results data; and
storing the results data.
2. The method of claim 1 , further comprising:
receiving a plurality of additional transactional queries against the distributed database;
processing, by the transactional threads, the plurality of additional transactional queries using the key-value pairs in the commit format, the transactional threads using commit times in the key-value pairs to generate additional results data; and storing the additional results data.
3. The method of claim 1 , wherein the distributed database comprises a transaction status table that indicates whether data of transactions has committed to the distributed database.
4. The method of claim 3 , wherein queries for data having key-value pairs in the initial format are executed by identifying transaction identifiers in the key-value pairs and access the transaction status table to determine commit statuses.
5. The method of claim 3 , further comprising: performing, using a dedicated compactor thread in one of the execution nodes, transformation and compaction of the key-value pairs from the initial format to the commit format.
6. The method of claim 5 , further comprising: performing, using the dedicated compactor thread, compaction of the transaction status table by deleting key-value pairs in the transaction status table that have corresponding key-value pairs in the commit format.
7. The method of claim 1 , wherein each key-value pair in the initial format includes a transaction identifier value.
8. The method of claim 7 , wherein the asynchronous threads perform transformation by rewriting each key-value pair such that the commit time replaces the transaction identifier value.
9. The method of claim 1 , wherein the key-value pairs having transaction identifiers are deleted by the asynchronous threads.
10. A system comprising:
one or more processors of a machine; and
a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:
performing, using asynchronous threads of execution nodes of a distributed database, asynchronous transformation and compaction of key-value pairs of data that correspond to a plurality of transactional queries, the asynchronous threads transforming the key-value pairs from an initial format to a commit format that includes a commit time of data committed to the distributed database, the asynchronous threads compacting the key-value pairs by deleting the key-value pairs in the initial format;
processing, using transactional threads in the execution nodes, the plurality of transactional queries to generate results data, the asynchronous threads performing the asynchronous transformation and compaction while the transactional threads generate the results data; and
storing the results data.
11. The system of claim 10 , wherein the operations further comprise:
receiving a plurality of additional transactional queries against the distributed database;
processing, by the transactional threads, the plurality of additional transactional queries using the key-value pairs in the commit format, the transactional threads using commit times in the key-value pairs to generate additional results data; and storing the additional results data.
12. The system of claim 10 , wherein the distributed database comprises a transaction status table that indicates whether data of transactions has committed to the distributed database.
13. The system of claim 12 , wherein queries for data having key-value pairs in the initial format are executed by identifying transaction identifiers in the key-value pairs and access the transaction status table to determine commit statuses.
14. The system of claim 12 , wherein the operations further comprise:
performing, using a dedicated compactor thread in one of the execution nodes, transformation and compaction of the key-value pairs from the initial format to the commit format.
15. The system of claim 14 , wherein the operations further comprise:
performing, using the dedicated compactor thread, compaction of the transaction status table by deleting key-value pairs in the transaction status table that have corresponding key-value pairs in the commit format.
16. The system of claim 10 , wherein each key-value pair in the initial format includes a transaction identifier value.
17. The system of claim 16 , wherein the asynchronous threads perform transformation by rewriting each key-value pair such that the commit time replaces the transaction identifier value.
18. The system of claim 10 , wherein the key-value pairs having transaction identifiers are deleted by the asynchronous threads.
19. A non-transitory machine storage medium comprising instructions that, when executed by one or more processors of a machine, configure the machine to perform operations comprising:
performing, using asynchronous threads of execution nodes of a distributed database, asynchronous transformation and compaction of key-value pairs of data that correspond to a plurality of transactional queries, the asynchronous threads transforming the key-value pairs from an initial format to a commit format that includes a commit time of data committed to the distributed database, the asynchronous threads compacting the key-value pairs by deleting the key-value pairs in the initial format;
processing, using transactional threads in the execution nodes, the plurality of transactional queries to generate results data, the asynchronous threads performing the asynchronous transformation and compaction while the transactional threads generate the results data; and
storing the results data.
20. The non-transitory machine storage medium of claim 19 , wherein the operations further comprise:
receiving a plurality of additional transactional queries against the distributed database;
processing, by the transactional threads, the plurality of additional transactional queries using the key-value pairs in the commit format, the transactional threads using commit times in the key-value pairs to generate additional results data; and
storing the additional results data.
21. The non-transitory machine storage medium of claim 19 , wherein the distributed database comprises a transaction status table that indicates whether data of transactions has committed to the distributed database.
22. The non-transitory machine storage medium of claim 21 , wherein queries for data having key-value pairs in the initial format are executed by identifying transaction identifiers in the key-value pairs and access the transaction status table to determine commit statuses.
23. The non-transitory machine storage medium of claim 21 , wherein the operations further comprise: performing, using a dedicated compactor thread in one of the execution nodes, transformation and compaction of the key-value pairs from the initial format to the commit format.
24. The non-transitory machine storage medium of claim 23 , wherein the operations further comprise: performing, using the dedicated compactor thread, compaction of the transaction status table by deleting key-value pairs in the transaction status table that have corresponding key-value pairs in the commit format.
25. The non-transitory machine storage medium of claim 19 , wherein each key-value pair in the initial format includes a transaction identifier value.
26. The non-transitory machine storage medium of claim 25 , wherein the asynchronous threads perform transformation by rewriting each key-value pair such that the commit time replaces the transaction identifier value.
27. The non-transitory machine storage medium of claim 19 , wherein the key-value pairs having transaction identifiers are deleted by the asynchronous threads.