High-throughput extract-transform-load (ETL) of program events for subsequent analysis
An event tap associated with a server, such as a Web server, at a machine can transform a server event into a tuple, select a database node for the tuple, and place the tuple in a queue for that database node, and then flush the queue periodically directly into database nodes. The use of an event tap can thus reduce the computational burden on the database while keeping the server event data in the database relatively fresh.
1. A system that includes one or more physical processors configured to:
load, by an event tap at one or more Web servers, event data representative of Web server events associated with one or more events captured as a result of use of one or more Web pages supported by the one or more Web servers, wherein the event data is in a raw form not readily suitable for directly storing and processing by a distributed database system that includes multiple nodes operable to each independently store and process data for the database system;
transform, by the event tap at the one or more Web servers, the event data from the raw form into a database representation format for storage and processing by the database system; and
distribute, by the event tap at the one or more Web servers, the database representation format between the multiple nodes of the distributed database system at least by:
selecting one of the multiple nodes of the database system as one selected node selected to directly store and process at least one portion of the database representation format of the event data;
selecting another of the multiple nodes of the database system as another selected node selected to directly store and process at least another portion of the database representation format of the event data;
arranging the at least one portion of the database representation format for delivery to the selected one node for processing by the one selected node directly; and
arranging the other portion of the database representation format for delivery to the other selected node for processing by the other selected node directly.
2. The system of claim 1 wherein the arranging of the database representation of the event data for delivery to the selected node comprises storing the database representation in a queue for delivery to the selected node, wherein for each one of the nodes of the database system, a queue is arranged to store database representation of the event data for delivery for that queue.
3. The system of claim 2 , wherein the queue is filled for a period of time before it is loaded to the selected node.
4. The system of claim 2 , wherein the queue overflows to disk if the selected database node is unavailable.
5. The system of claim 1 , wherein the transforming the event data into a database representation transforms the event data into a tuple.
6. The system of claim 1 , wherein the database representation of the event data is arranged for delivery to another node of the database system when the selected node is or becomes unable to accept the transformed event data.
7. The system of claim 1 , wherein an event interceptor on the server provides the event data.
8. The system of claim 1 , wherein the events are web server events associated with providing of content to a user across the Internet.
9. A computer-implemented method implemented at least partly by a device, the computer-implemented method comprising:
loading, by an event tap at one or more Web servers, event data representative of Web server events associated with one or more events captured as a result of use of one or more Web pages supported by the one or more Web servers, wherein the event data is in a raw form not readily suitable for directly storing and processing by a distributed database system that includes multiple nodes operable to each independently store and process data for the database system;
transforming, by the event tap at the one or more Web servers, the event data from the raw form into a database representation format for storage and processing by the database system; and
distributing, by the event tap at the one or more Web servers, the database representation format between the multiple nodes of the distributed database system at least by:
selecting one of the multiple nodes of the database system as one selected node selected to directly store and process at least one portion of the database representation format of the event data;
selecting another of the multiple nodes of the database system as another selected node selected to directly store and process at least another portion of the database representation format of the event data;
arranging the at least one portion of the database representation format for delivery to the selected one node for processing by the one selected node directly; and
arranging the other portion of the database representation format for delivery to the other selected node for processing by the other selected node directly.
10. The computer-implemented method of claim 9 , wherein the arranging of the database representation of the event data for delivery to the selected node comprises storing the database representation in a queue for delivery to the selected node, wherein for each one of the nodes of the database system, a queue is arranged to store database representation of the event data for delivery for that queue.
11. The computer-implemented method of claim 10 , wherein the queue is filled for a period of time before it is loaded to the selected node.
12. The computer-implemented method of claim 10 , wherein the queue overflows to disk if the selected database node is unavailable.
13. The computer-implemented method of claim 9 , wherein the transforming the event data into a database representation transforms the event data into a tuple.
14. The computer-implemented method of claim 9 , wherein the database representation of the event data is arranged for delivery to another node of the database system when the selected node is or becomes unable to accept the transformed event data.
15. The non-transitory computer readable storage medium of claim 9 , wherein an event interceptor on the Web server provides the event data.
16. A non-transitory computer readable storage medium storing at least executable computer code that when executed causes an event tap at one or more Web servers to:
load event data representative of Web server events associated with one or more events captured as a result of use of one or more Web pages supported by the one or more Web servers, wherein the event data is in a raw form not readily suitable for directly storing and processing by a distributed database system that includes multiple nodes operable to each independently store and process data for the database system;
transform, by the event tap at the one or more Web servers, the event data from the raw form into a database representation format for storage and processing by the database system; and
distribute, by the event tap at the one or more Web servers, the database representation format between the multiple nodes of the distributed database system at least by:
selecting one of the multiple nodes of the database system as one selected node selected to directly store and process at least one portion of the database representation format of the event data;
selecting another of the multiple nodes of the database system as another selected node selected to directly store and process at least another portion of the database representation format of the event data;
arranging the at least one portion of the database representation format for delivery to the selected one node for processing by the one selected node directly; and
arranging the other portion of the database representation format for delivery to the other selected node for processing by the other selected node directly.
17. The non-transitory computer readable storage medium of claim 16 , wherein the arranging of the database representation of the event data for delivery to the selected node comprises storing the database representation in a queue for delivery to the selected node, wherein for each one of the nodes of the database system, a queue is arranged to store database representation of the event data for delivery for that queue.
18. The non-transitory computer readable storage medium of claim 17 , wherein the queue is filled for a period of time before it is loaded to the selected node.
19. The non-transitory computer readable storage medium of claim 17 , wherein the queue overflows to disk if the selected database node is unavailable.
20. The non-transitory computer readable storage medium of claim 17 , wherein the database representation of the event data is arranged for delivery to another node of the database system when the selected node is or becomes unable to accept the transformed event data.
21. The non-transitory computer readable storage medium of claim 17 , wherein an event interceptor on the Web server provides the event data.
22. The non-transitory computer readable storage medium of claim 17 , wherein the events are web server events associated with providing of content to a user across the Internet.
23. The non-transitory computer readable storage medium of claim 16 , wherein the transforming the event data into a database representation transforms the event data into a tuple.