Mass insertion into single-threaded databases
Disclosed herein are system, method, and computer-readable device embodiments for mass insertion into single-threaded databases. An embodiment includes a processor and a memory, a storage layer to interface with a plurality of software applications and to receive data output from the plurality of software applications, and a listener that runs according to an update policy, to detect the presence of information newly stored within the storage layer. The processor and memory may be configured to maintain at least a part of a running database cluster including a plurality of nodes, with at least two nodes configured to run without multi-threading, and to execute an intermediate module to send at least part of the information to the database cluster, and to perform simultaneous access to multiple database nodes running without multi-threading.
1. A system, comprising:
at least one processor;
at least one storage layer configured to interface with a plurality of software applications and to receive data output from the plurality of software applications;
at least one listener configured to detect presence of information newly stored within the at least one storage layer, wherein the information comprises a key and a value in a key-value pair corresponding to the information; and
a memory operatively coupled to the at least one processor, the at least one processor configured to:
run the at least one listener according to an update policy;
maintain at least a part of a running database cluster, wherein the database cluster comprises a plurality of nodes, wherein at least two of the plurality of nodes are configured to run without multi-threading;
calculate a hash of the key;
send the key-value pair to an executor module associated with a corresponding node, wherein the information corresponding to the key-value pair is unique to the corresponding node, and wherein existing information in the at least one storage layer is distributed among the plurality of nodes; and
execute at least one intermediate module, wherein the at least one intermediate module is configured to send at least part of the information from the at least one storage layer to the database cluster, wherein the at least one intermediate module is further configured to perform at least one simultaneous access to the at least two of the plurality of nodes configured to run without multi-threading.
2. The system of claim 1 , wherein the at least one simultaneous access to the at least two of the plurality of nodes configured to run without multi-threading comprises:
bypassing a database cluster interface to write directly into the plurality of nodes.
3. The system of claim 1 , wherein the update policy comprises at least one of:
an interval of polling of the at least one storage layer, a real-time response to new information detected in the at least one storage layer, a functional program, a monitoring of at least one event-driven trigger independently of the at least one storage layer, or any combination of the above.
4. The system of claim 1 , wherein the plurality of nodes are incapable of non-single-threaded operation, or wherein any multi-threaded operation is deactivated.
5. The system of claim 1 , wherein the at least one storage layer further comprises at least one of a data store or object storage configured to perform independent storage operations for multiple software applications of the plurality of software applications.
6. The system of claim 1 , wherein the at least one storage layer further comprises at least one of a filesystem common to multiple software applications of the plurality of software applications.
7. The system of claim 1 , wherein each node in the plurality of nodes corresponds to a range of hashes, wherein size of each range of hashes for each node is proportional to quantity of possible hashes for each node divided by quantity of nodes in the plurality of nodes, within a platform-specific tolerance.
8. The system of claim 1 , wherein the at least one intermediate module further comprises at least one driver controlling a plurality of executor modules comprising the executor module, wherein each executor module of the plurality of executor modules is configured to send a stream of information to at least one node in the plurality of nodes simultaneously with each other executor module, directed by the at least one driver.
9. A computer-implemented method, comprising:
interfacing with a plurality of software applications;
receiving data output from the plurality of software applications;
running a listener, according to an update policy, to detect presence of information newly stored within a storage layer, wherein the information comprises a key and a value in a key-value pair corresponding to the information;
maintaining at least a part of a running database cluster, wherein the database cluster comprises a plurality of nodes, wherein at least two of the plurality of nodes are configured to run without multi-threading;
calculating a hash of the key;
sending the key-value pair to an executor module associated with a corresponding node, wherein the information corresponding to the key-value pair is unique to the corresponding node, and wherein existing information in the storage layer is distributed among the plurality of nodes; and
executing at least one intermediate module, wherein the at least one intermediate module is configured to send at least part of the information, from the storage layer to the database cluster, wherein the at least one intermediate module is further configured to perform at least one simultaneous access to the at least two of the plurality of nodes configured to run without multi-threading.
10. The computer-implemented method of claim 9 , wherein the at least one simultaneous access to the at least two of the plurality of nodes configured to run without multi-threading comprises:
bypassing a database cluster interface to write directly into the plurality of nodes.
11. The computer-implemented method of claim 9 , wherein the update policy comprises at least one of:
an interval of polling of the storage layer, a real-time response to new information detected in the storage layer, a functional program, a monitoring of at least one event-driven trigger independently of the storage layer, or any combination of the above.
12. The computer-implemented method of claim 9 , wherein the plurality of nodes are incapable of non-single-threaded operation, or wherein any multi-threaded operation is deactivated.
13. The computer-implemented method of claim 9 , wherein the storage layer further comprises at least one of a data store or object storage configured to perform independent storage operations for multiple software applications of the plurality of software applications.
14. The computer-implemented method of claim 9 , wherein the storage layer further comprises at least one of a filesystem common to multiple software applications of the plurality of software applications.
15. The computer-implemented method of claim 9 , wherein each node in the plurality of nodes corresponds to a range of hashes, wherein size of each range of hashes for each node is proportional to quantity of possible hashes for each node divided by quantity of nodes in the plurality of nodes, within a platform-specific tolerance.
16. The computer-implemented method of claim 9 , wherein the at least one intermediate module further comprises at least one driver controlling a plurality of executor modules comprising the executor module, wherein each executor module of the plurality of executor modules is configured to send a stream of information to at least one node in the plurality of nodes simultaneously with each other executor module, directed by the at least one driver.
17. A non-transitory computer-readable device having instructions stored thereon that, when executed by at least one computing device, causes the at least one computing device to perform operations comprising:
interfacing with a plurality of software applications;
receiving data output from the plurality of software applications;
running a listener, according to an update policy, to detect presence of information newly stored within a storage layer, wherein the information comprises a key and a value in a key-value pair corresponding to the information;
maintaining at least a part of a running database cluster, wherein the database cluster comprises a plurality of nodes, wherein at least two of the plurality of nodes are configured to run without multi-threading;
calculating a hash of the key;
sending the key-value pair to an executor module associated with a corresponding node, wherein the information corresponding to the key-value pair is unique to the corresponding node, and wherein existing information in the storage layer is distributed among the plurality of nodes; and
executing at least one intermediate module, wherein the at least one intermediate module is configured to send at least part of the information, from the storage layer to the database cluster, wherein the at least one intermediate module is further configured to perform at least one simultaneous access to the at least two of the plurality of nodes configured to run without multi-threading.
18. The non-transitory computer-readable device of claim 17 , wherein the at least one simultaneous access to the plurality of nodes configured to run without multi-threading comprises:
bypassing a database cluster interface to write directly into the plurality of nodes.
19. The non-transitory computer-readable device of claim 17 , wherein the update policy comprises at least one of:
an interval of polling of the storage layer, a real-time response to new information detected in the storage layer, a functional program, a monitoring of at least one event-driven trigger independently of the storage layer, or any combination of the above.
20. The non-transitory computer-readable device of claim 17 , wherein the plurality of nodes are incapable of non-single-threaded operation, or wherein any multi-threaded operation is deactivated.
21. The non-transitory computer-readable device of claim 17 , wherein the storage layer further comprises at least one of a data store or object storage configured to perform independent storage operations for multiple software applications of the plurality of software applications.
22. The non-transitory computer-readable device of claim 17 , wherein the storage layer further comprises at least one of a filesystem common to multiple software applications of the plurality of software applications.
23. The non-transitory computer-readable device of claim 17 , wherein each node in the plurality of nodes corresponds to a range of hashes, wherein size of each range of hashes for each node is proportional to quantity of possible hashes for each node divided by quantity of nodes in the plurality of nodes, within a platform-specific tolerance.
24. The non-transitory computer-readable device of claim 17 , wherein the at least one intermediate module further comprises at least one driver controlling a plurality of executor modules comprising the executor module, wherein each executor module of the plurality of executor modules is configured to send a stream of information to at least one node in the plurality of nodes simultaneously with each other executor module, directed by the at least one driver.