Mass insertion into single-threaded databases
Disclosed herein are system, method, and computer-readable device embodiments for mass insertion into single-threaded databases. An embodiment includes a processor and a memory, a storage layer to interface with a plurality of software applications and to receive data output from the plurality of software applications, and a listener that runs according to an update policy, to detect the presence of information newly stored within the storage layer. The processor and memory may be configured to maintain at least a part of a running database cluster including a plurality of nodes, with at least two nodes configured to run without multi-threading, and to execute an intermediate module to send at least part of the information to the database cluster, and to perform simultaneous access to multiple database nodes running without multi-threading.
1. A system, comprising:
at least one processor;
at least one storage layer configured to interface with a plurality of software applications and to receive data output from the plurality of software applications;
at least one listener configured to detect presence of information newly stored within the at least one storage layer, wherein the information comprises a key and a value in a key-value pair corresponding to the information; and
a memory operatively coupled to the at least one processor, the at least one processor configured to perform operations comprising:
calculating a hash of the key;
sending the key-value pair to an executor node associated with a corresponding database node, wherein the information corresponding to the key-value pair is unique to the corresponding database node, and wherein existing information in the at least one storage layer is distributed among a plurality of database nodes;
sending at least part of the information from the at least one storage layer to a database cluster; and
performing at least one simultaneous access to at least two of the plurality of database nodes configured to run as single-threaded database nodes.
2. The system of claim 1 , further comprising a plurality of executor nodes.
3. The system of claim 2 , the operations further comprising performing a mass-insertion across the plurality of database nodes, for a quantity of the plurality of database nodes equal to a corresponding quantity of the plurality of executor nodes.
4. The system of claim 3 , wherein the quantity of the plurality of database nodes and the corresponding quantity of the plurality of executor nodes excludes overprovisioned nodes corresponding to the plurality of database nodes or to the plurality of executor nodes.
5. The system of claim 1 , the operations further comprising generating or updating at least one content recommendation based at least in part on the key-value pair.
6. The system of claim 1 , wherein a function of the at least one listener comprises a wait state and a lambda function.
7. The system of claim 1 , wherein the at least one storage layer comprises object storage further comprising distributed objects in a unified namespace.
8. A computer-implemented method, comprising:
interfacing, via at least one processor, with a plurality of software applications to receive data output from the plurality of software applications;
detecting, via a listener, a presence of information newly stored within at least one storage layer, wherein the information comprises a key and a value in a key-value pair corresponding to the information;
calculating a hash of the key;
sending the key-value pair to an executor node associated with a corresponding database node, wherein the information corresponding to the key-value pair is unique to the corresponding database node, and wherein existing information in the at least one storage layer is distributed among a plurality of database nodes;
sending at least part of the information from the at least one storage layer to a database cluster; and
performing at least one simultaneous access to at least two of the plurality of database nodes configured to run as single-threaded database nodes.
9. The computer-implemented method of claim 8 , the operations further comprising performing a mass-insertion across the plurality of database nodes, for a quantity of the plurality of database nodes equal to a corresponding quantity of a plurality of executor nodes.
10. The computer-implemented method of claim 9 , wherein the quantity of the plurality of database nodes and the corresponding quantity of the plurality of executor nodes excludes overprovisioned nodes corresponding to the plurality of database nodes or to the plurality of executor nodes.
11. The computer-implemented method of claim 8 , further comprising generating or updating at least one content recommendation based at least in part on the key-value pair.
12. The computer-implemented method of claim 8 , wherein the listener comprises a wait state and a lambda function.
13. The computer-implemented method of claim 8 , wherein the at least one storage layer comprises object storage further comprising distributed objects in a unified namespace.
14. A non-transitory computer-readable medium having instructions stored thereon that, when executed by at least one computer processor, cause the at least one computer processor to perform operations comprising:
interfacing with a plurality of software applications to receive data output from the plurality of software applications;
detecting a presence of information newly stored within at least one storage layer, wherein the information comprises a key and a value in a key-value pair corresponding to the information;
calculating a hash of the key;
sending the key-value pair to an executor node associated with a corresponding database node, wherein the information corresponding to the key-value pair is unique to the corresponding database node, and wherein existing information in the at least one storage layer is distributed among a plurality of database nodes;
sending at least part of the information from the at least one storage layer to a database cluster; and
performing at least one simultaneous access to at least two of the plurality of database nodes configured to run as single-threaded database nodes.
15. The non-transitory computer-readable medium of claim 14 , the operations further comprising performing a mass-insertion across the plurality of database nodes, for a quantity of the plurality of database nodes equal to a corresponding quantity of a plurality of executor nodes.
16. The non-transitory computer-readable medium of claim 15 , wherein the quantity of the plurality of database nodes and the corresponding quantity of the plurality of executor nodes excludes overprovisioned nodes corresponding to the plurality of database nodes or to the plurality of executor nodes.
17. The non-transitory computer-readable medium of claim 14 , the operations further comprising generating or updating at least one content recommendation based at least in part on the key-value pair.
18. The non-transitory computer-readable medium of claim 14 , wherein the detecting comprises a wait state and a lambda function.
19. The non-transitory computer-readable medium of claim 14 , wherein the at least one storage layer comprises object storage further comprising distributed objects in a unified namespace.
20. The non-transitory computer-readable medium of claim 14 , the operations further comprising updating at least one state indicator for a shared resource among the plurality of database nodes.