Mass insertion into single-threaded databases
Disclosed herein are system, method, and computer-readable device embodiments for mass insertion into single-threaded databases. An embodiment includes a processor and a memory, a storage layer to interface with a plurality of software applications and to receive data output from the plurality of software applications, and a listener that runs according to an update policy, to detect the presence of information newly stored within the storage layer. The processor and memory may be configured to maintain at least a part of a running database cluster including a plurality of nodes, with at least two nodes configured to run without multi-threading, and to execute an intermediate module to send at least part of the information to the database cluster, and to perform simultaneous access to multiple database nodes running without multi-threading.
1 . A system, comprising:
at least one processor;
at least one storage layer configured to interface with a plurality of software applications and to receive data output from the plurality of software applications;
at least one listener configured to detect a presence of a data entry newly stored within the at least one storage layer, wherein the data entry comprises at least a key and a value in a key-value pair corresponding to the data entry; and
a memory operatively coupled to the at least one processor, the at least one processor configured to perform operations comprising:
sending the key-value pair to a computer cluster, wherein the computer cluster comprises a master node interfacing with a plurality of slave nodes, each slave node of the plurality of slave nodes assigned to a corresponding database node based on a hash value of the key of the key-value pair being unique to the corresponding database node, and wherein each of the corresponding database nodes is configured to run as a single-threaded database; and
performing at least one mass insertion of the data entry from the at least one storage layer by simultaneous access and storage of the data entry to at least two of a plurality of the corresponding database nodes.
2 . The system of claim 1 , wherein the hash value of the key is within a range of hashes assigned to the corresponding database node.
3 . The system of claim 1 , the operations further comprising performing the at least one mass-insertion across the plurality of corresponding database nodes, for a quantity of the plurality of corresponding database nodes equal to a corresponding quantity of the plurality of slave nodes.
4 . The system of claim 3 , wherein the quantity of the plurality of corresponding database nodes and the corresponding quantity of the plurality of slave nodes excludes overprovisioned nodes corresponding to the plurality of corresponding database nodes or to the plurality of slave nodes.
5 . The system of claim 1 , the operations further comprising generating or updating at least one content recommendation based at least in part on the key-value pair.
6 . The system of claim 1 , wherein a function of the at least one listener comprises a wait state and a lambda function.
7 . The system of claim 1 , wherein the at least one storage layer comprises object storage, and wherein the object storage comprises distributed objects in a unified namespace.
8 . A computer-implemented method, comprising:
interfacing, via at least one processor, with at least one storage layer to receive data output from a plurality of software applications;
detecting, via at least one listener, a presence of a data entry newly stored within the at least one storage layer, wherein the data entry comprises at least a key and a value in a key-value pair corresponding to the data entry;
sending the key-value pair to a computer cluster, wherein the computer cluster comprises a master node interfacing with a plurality of slave nodes, each slave node of the plurality of slave nodes assigned to a corresponding database node based on a hash value of the key of the key-value pair being unique to the corresponding database node, and wherein each of the corresponding database nodes is configured to run as a single-threaded database; and
performing at least one mass insertion of the data entry from the at least one storage layer by simultaneous access and storage of the data entry to at least two of a plurality of the corresponding database nodes.
9 . The computer-implemented method of claim 8 , further comprising performing a mass-insertion across the plurality of corresponding database nodes, for a quantity of the plurality of corresponding database nodes equal to a corresponding quantity of the plurality of slave nodes.
10 . The computer-implemented method of claim 9 , wherein the quantity of the plurality of corresponding database nodes and the corresponding quantity of the plurality of slave nodes excludes overprovisioned nodes corresponding to the plurality of corresponding database nodes or to the plurality of slave nodes.
11 . The computer-implemented method of claim 8 , further comprising generating or updating at least one content recommendation based at least in part on the key-value pair.
12 . The computer-implemented method of claim 8 , wherein the at least one listener comprises a wait state and a lambda function.
13 . The computer-implemented method of claim 8 , wherein the at least one storage layer comprises object storage, and wherein the object storage comprises distributed objects in a unified namespace.
14 . A non-transitory computer-readable medium having instructions stored thereon that, when executed by at least one computer processor, cause the at least one computer processor to perform operations comprising:
interfacing, via the at least one computer processor, with at least one storage layer to receive data output from a plurality of software applications;
detecting, via at least one listener, a presence of a data entry newly stored within the at least one storage layer, wherein the data entry comprises at least a key and a value in a key-value pair corresponding to the data entry;
sending the key-value pair to a computer cluster, wherein the computer cluster comprises a master node interfacing with a plurality of slave nodes, each slave node of the plurality of slave nodes assigned to a corresponding database node based on a hash value of the key of the key-value pair being unique to the corresponding database node, and wherein each of the corresponding database nodes is configured to run as a single-threaded database; and
performing at least one mass insertion of the data entry from the at least one storage layer by simultaneous access and storage of the data entry to at least two of a plurality of the corresponding database nodes.
15 . The non-transitory computer-readable medium of claim 14 , the operations further comprising performing a mass-insertion across the plurality of corresponding database nodes, for a quantity of the plurality of corresponding database nodes equal to a corresponding quantity of the plurality of slave nodes.
16 . The non-transitory computer-readable medium of claim 15 , wherein the quantity of the plurality of corresponding database nodes and the corresponding quantity of the plurality of slave nodes excludes overprovisioned nodes corresponding to the plurality of corresponding database nodes or to the plurality of slave nodes.
17 . The non-transitory computer-readable medium of claim 14 , the operations further comprising generating or updating at least one content recommendation based at least in part on the key-value pair.
18 . The non-transitory computer-readable medium of claim 14 , wherein the detecting comprises a wait state and a lambda function.
19 . The non-transitory computer-readable medium of claim 14 , wherein the at least one storage layer comprises object storage, and wherein the object storage comprises distributed objects in a unified namespace.
20 . The non-transitory computer-readable medium of claim 14 , the operations further comprising updating at least one state indicator for a shared resource among the plurality of corresponding database nodes.