Distributed database that uses hybrid table secondary indexes
The subject technology receives, by a distributed database, transactional queries against a key-value database comprising data managed by key-value pairs. The subject technology performs, using asynchronous threads of execution nodes of the distributed database, asynchronous transformation and compaction of key-value pairs of data that corresponds to the plurality of transactional queries. The subject technology performs, using asynchronous threads of execution nodes of the distributed database, asynchronous transformation and compaction of a set of secondary indexes that corresponds to the plurality of transactional queries. The subject technology processes, using transactional threads in the execution nodes, the plurality of transactional queries to generate results data, the asynchronous threads performing the asynchronous transformation and compaction while the transactional threads generate the results data.
1. A method comprising:
receiving, by a distributed database, a plurality of transactional queries against a key-value database comprising data managed by key-value pairs;
performing, using asynchronous threads of execution nodes of the distributed database, asynchronous transformation and compaction of key-value pairs of data that corresponds to the plurality of transactional queries, the asynchronous threads transforming the key-value pairs from an initial format to a commit format that includes a commit time of data committed to the distributed database, the asynchronous threads compacting the key-value pairs by deleting the key-value pairs in the initial format;
performing, using asynchronous threads of execution nodes of the distributed database, asynchronous transformation and compaction of a set of secondary indexes that corresponds to the plurality of transactional queries;
processing, using transactional threads in the execution nodes, the plurality of transactional queries to generate results data, the asynchronous threads performing the asynchronous transformation and compaction while the transactional threads generate the results data; and
storing the results data.
2. The method of claim 1 , further comprising:
receiving a plurality of additional transactional queries against the distributed database.
3. The method of claim 2 , further comprising:
processing, by the transactional threads, the plurality of additional transactional queries using the key-value pairs in the commit format, the transactional threads using commit times in the key-value pairs to generate additional results data; and
storing the additional results data.
4. The method of claim 1 , wherein the distributed database comprises a transaction status table that indicates whether data of transactions has committed to the distributed database.
5. The method of claim 4 , wherein queries for data having key-value pairs in the initial format are executed by identifying transaction identifiers in the key-value pairs and access the transaction status table to determine commit statuses.
6. The method of claim 4 , further comprising:
performing, using a dedicated compactor thread in one of the execution nodes, transformation and compaction of a second set of key-value pairs from the initial format to the commit format.
7. The method of claim 6 , further comprising:
performing, using the dedicated compactor thread, compaction of the transaction status table by deleting particular key-value pairs in the transaction status table that have corresponding key-value pairs in the commit format.
8. The method of claim 6 , further comprising:
performing, using the dedicated compactor thread, compaction of a second set of secondary indexes associated with a set of tables corresponding to the second set of key-value pairs.
9. The method of claim 1 , wherein each key-value pair in the initial format includes a transaction identifier value, wherein the asynchronous threads perform transformation by rewriting each key-value pair such that the commit time replaces the transaction identifier value.
10. The method of claim 1 , wherein the key-value pairs having transaction identifiers are deleted by the asynchronous threads.
11. A system comprising:
one or more processors of a machine; and
a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:
receiving, by a distributed database, a plurality of transactional queries against a key-value database comprising data managed by key-value pairs;
performing, using asynchronous threads of execution nodes of the distributed database, asynchronous transformation and compaction of key-value pairs of data that corresponds to the plurality of transactional queries, the asynchronous threads transforming the key-value pairs from an initial format to a commit format that includes a commit time of data committed to the distributed database, the asynchronous threads compacting the key-value pairs by deleting the key-value pairs in the initial format;
performing, using asynchronous threads of execution nodes of the distributed database, asynchronous transformation and compaction of a set of secondary indexes that corresponds to the plurality of transactional queries;
processing, using transactional threads in the execution nodes, the plurality of transactional queries to generate results data, the asynchronous threads performing the asynchronous transformation and compaction while the transactional threads generate the results data; and
storing the results data.
12. The system of claim 11 , wherein the operations further comprise:
receiving a plurality of additional transactional queries against the distributed database.
13. The system of claim 12 , wherein the operations further comprise:
processing, by the transactional threads, the plurality of additional transactional queries using the key-value pairs in the commit format, the transactional threads using commit times in the key-value pairs to generate additional results data; and
storing the additional results data.
14. The system of claim 11 , wherein the distributed database comprises a transaction status table that indicates whether data of transactions has committed to the distributed database.
15. The system of claim 14 , wherein queries for data having key-value pairs in the initial format are executed by identifying transaction identifiers in the key-value pairs and access the transaction status table to determine commit statuses.
16. The system of claim 14 , wherein the operations further comprise:
performing, using a dedicated compactor thread in one of the execution nodes, transformation and compaction of a second set of key-value pairs from the initial format to the commit format.
17. The system of claim 16 , wherein the operations further comprise:
performing, using the dedicated compactor thread, compaction of the transaction status table by deleting particular key-value pairs in the transaction status table that have corresponding key-value pairs in the commit format.
18. The system of claim 16 , wherein the operations further comprise:
performing, using the dedicated compactor thread, compaction of a second set of secondary indexes associated with a set of tables corresponding to the second set of key-value pairs.
19. The system of claim 11 , wherein each key-value pair in the initial format includes a transaction identifier value, wherein the asynchronous threads perform transformation by rewriting each key-value pair such that the commit time replaces the transaction identifier value.
20. The system of claim 11 , wherein the key-value pairs having transaction identifiers are deleted by the asynchronous threads.
21. A non-transitory computer-storage medium storing instructions that, when executed by one or more processors of a machine, configure the machine to perform operations comprising:
receiving, by a distributed database, a plurality of transactional queries against a key-value database comprising data managed by key-value pairs;
performing, using asynchronous threads of execution nodes of the distributed database, asynchronous transformation and compaction of key-value pairs of data that corresponds to the plurality of transactional queries, the asynchronous threads transforming the key-value pairs from an initial format to a commit format that includes a commit time of data committed to the distributed database, the asynchronous threads compacting the key-value pairs by deleting the key-value pairs in the initial format;
performing, using asynchronous threads of execution nodes of the distributed database, asynchronous transformation and compaction of a set of secondary indexes that corresponds to the plurality of transactional queries;
processing, using transactional threads in the execution nodes, the plurality of transactional queries to generate results data, the asynchronous threads performing the asynchronous transformation and compaction while the transactional threads generate the results data; and
storing the results data.
22. The non-transitory computer-storage medium of claim 21 , wherein the operations further comprise: receiving a plurality of additional transactional queries against the distributed database.
23. The non-transitory computer-storage medium of claim 22 , wherein the operations further comprise: processing, by the transactional threads, the plurality of additional transactional queries using the key-value pairs in the commit format, the transactional threads using commit times in the key-value pairs to generate additional results data; and storing the additional results data.
24. The non-transitory computer-storage medium of claim 21 , wherein the distributed database comprises a transaction status table that indicates whether data of transactions has committed to the distributed database.
25. The non-transitory computer-storage medium of claim 24 , wherein queries for data having key-value pairs in the initial format are executed by identifying transaction identifiers in the key-value pairs and access the transaction status table to determine commit statuses.
26. The non-transitory computer-storage medium of claim 24 , wherein the operations further comprise: performing, using a dedicated compactor thread in one of the execution nodes, transformation and compaction of a second set of key-value pairs from the initial format to the commit format.
27. The non-transitory computer-storage medium of claim 26 , wherein the operations further comprise: performing, using the dedicated compactor thread, compaction of the transaction status table by deleting particular key-value pairs in the transaction status table that have corresponding key-value pairs in the commit format.
28. The non-transitory computer-storage medium of claim 26 , wherein the operations further comprise: performing, using the dedicated compactor thread, compaction of a second set of secondary indexes associated with a set of tables corresponding to the second set of key-value pairs.
29. The non-transitory computer-storage medium of claim 21 , wherein each key-value pair in the initial format includes a transaction identifier value, wherein the asynchronous threads perform transformation by rewriting each key-value pair such that the commit time replaces the transaction identifier value.
30. The non-transitory computer-storage medium of claim 21 , wherein the key-value pairs having transaction identifiers are deleted by the asynchronous threads.