Network event capture and retention system
Methods and apparatus are provided to monitor and analyze activity occurring on a networked computer system. In some embodiments, a method is provided for capturing, in a data structure, at least a portion of a notification describing a network event provided by a node on a computer network, identifying a data element (e.g., an IP address of the node) within the notification, and updating an index and/or summary based on the data element. The data structure may be stored in a file system maintained on a site, and sites may exchange information related to the notification data stored on each. In some embodiments, a query which is issued to a site may be processed using data transferred from other sites, and/or may be split into one or more additional queries which may be transmitted for processing to other sites.
1 . A method for monitoring activity on a networked computer system, the networked computer system comprising a plurality of nodes, each of the plurality of nodes configured to transmit a notification for each event processed by the node, the networked computer system further comprising a plurality of sites, each of the plurality of sites being configured to capture the notifications transmitted by at least one node, the method comprising:
(A) each of the plurality of sites creating an indication of the notifications captured by the site;
(B) each of the plurality of sites transferring the indication to at least one other of the plurality of sites.
2 . The method of claim 1 , wherein the plurality of sites are organized into a hierarchy, wherein each site in the hierarchy is assigned at least one of a master site and a subordinate site.
3 . The method of claim 2 , wherein each site performs at least one transfer of an indication, wherein the at least one transfer is performed to at least one of a subordinate site and a master site.
4 . The method of claim 3 , wherein each site performs a transfer to its master site and receives a transfer from its master site, and wherein as a result of the transfers, each site receives the indication created by each other site.
5 . The method of claim 4 , further comprising acts of:
(C) receiving, by a site, a query requesting information on an event processed by a node from which the site does not capture notifications; and
(D) processing the query, by the site, by accessing at least one indication transferred to the site by another site.
6 . The method of claim 5 , further comprising acts of:
(E) determining, by the site, based on the at least one indication transferred to the site by another site, that the at least one indication does not contain the information requested by the query;
(F) identifying, by the site, which of the other sites may have captured the information requested by the query;
(G) creating, by the site, at least one additional query requesting the information from at least one other site identified in the act (F); and
(H) transmitting, by the site, the at least one additional query to the at least one other site.
7 . The method of claim 6 , further comprising acts of:
(I) receiving, at the site, a response for each of the at least one additional queries from the at least one other site; and
(J) aggregating, by the site, the responses received.
8 . The method of claim 1 , wherein the act (A) further comprises creating, by each of the plurality of sites, an indication which includes a representation of data stored in a file system on the site.
9 . The method of claim 1 , wherein each site captures notifications transmitted by nodes deployed in a geographic area.
10 . The method of claim 1 , wherein each site captures the notifications transmitted by a set of nodes, the set of nodes remaining unchanged.
11 . The method of claim 1 , wherein the plurality of sites comprises two portions, wherein a first portion includes sites configured to communicate with any other site in the first portion, and wherein a second portion includes at least one site configured to communicate with only one site in the first portion, the method comprising acts of:
(A) capturing, by a site in the second portion, an indication of the notifications captured by the site;
(B) transferring, by the site in the second portion, the indication to a site in the first portion.
12 . The method of claim 11 , further comprising an act of:
(C) transferring, by the site in the first portion, the indication to another site in the first portion.
13 . The method of claim 11 , wherein the act (B) is performed after a predetermined period of time elapses.
14 . The method of claim 11 , wherein the act (B) is performed after a notification captured by the site in the second portion satisfies a predetermined criterion.
15 . At least one computer-readable medium encoded with instructions which, when executed by a computer, perform a method for monitoring activity on a networked computer system, the networked computer system comprising a plurality of nodes, each of the plurality of nodes configured to transmit a notification for each event processed by the node, the networked computer system further comprising a plurality of sites, each of the plurality of sites being configured to capture the notifications transmitted by at least one node, the method comprising:
(A) each of the plurality of sites creating an indication of the notifications captured by the site;
(B) each of the plurality of sites transferring the indication to at least one other of the plurality of sites.
16 . The at least one computer-readable medium of claim 15 , wherein the plurality of sites are organized into a hierarchy, wherein each site in the hierarchy is assigned at least one of a master site and a subordinate site.
17 . The at least one computer-readable medium of claim 16 , further comprising instructions defining each site performing at least one transfer of an indication, wherein the at least one transfer is performed to at least one of a subordinate site and a master site.
18 . The at least one computer-readable medium of claim 17 , further comprising instructions defining each site performing a transfer to its master site and receiving a transfer from its master site, wherein, as a result of the transfers, each site receives the indication created by each other site.
19 . The at least one computer-readable medium of claim 18 , further comprising instructions defining:
(C) receiving, by a site, a query requesting information on an event processed by a node from which the site does not capture notifications; and
(D) processing the query, by the site, by accessing at least one indication transferred to the site by another site.
20 . The at least one computer-readable medium of claim 19 , further comprising instructions defining:
(E) determining, by the site, based on the at least one indication transferred to the site by another site, that the at least one indication does not contain the information requested by the query;
(F) identifying, by the site, which of the other sites may have captured the information requested by the query;
(G) creating, by the site, at least one additional query requesting the information from at least one other site identified in the act (F); and
(H) transmitting, by the site, the at least one additional query to the at least one other site.
21 . The at least one computer-readable medium of claim 20 , further comprising instructions defining:
(I) receiving, at the site, a response for each of the at least one additional queries from the at least one other site; and
(J) aggregating, by the site, the responses received.
22 . The at least one computer-readable medium of claim 15 , further comprising instructions defining creating, by each of the plurality of sites, an indication which includes a representation of data stored in a file system on the site.
23 . The at least one computer-readable medium of claim 15 , further comprising instructions defining each site capturing notifications transmitted by nodes deployed in a geographic area.
24 . The at least one computer-readable medium of claim 15 , further comprising instructions defining each site capturing the notifications transmitted by a set of nodes, the set of nodes remaining unchanged.
25 . The at least one computer-readable medium of claim 15 , wherein the plurality of sites comprises two portions, wherein a first portion includes sites configured to communicate with any other site in the first portion, and wherein a second portion includes at least one site configured to communicate with only one site in the first portion, further comprising instructions defining:
(K) capturing, by a site in the second portion, an indication of the notifications captured by the site;
(L) transferring, by the site in the second portion, the indication to a site in the first portion.
26 . The at least one computer-readable medium of claim 25 , further comprising instructions defining:
(M) transferring, by the site in the first portion, the indication to another site in the first portion.
27 . The at least one computer-readable medium of claim 25 , further comprising instructions defining performing the act (L) after a predetermined period of time elapses.
28 . The at least one computer-readable medium of claim 25 , further comprising instructions defining performing the act (L) after a notification captured by the site in the second portion satisfies a predetermined criterion.
29 . A system for monitoring activity on a networked computer system, the networked computer system comprising a plurality of nodes, each of the plurality of nodes configured to transmit a notification for each event processed by the node, the networked computer system further comprising a plurality of sites, each of the plurality of sites being configured to capture the notifications transmitted by at least one node, comprising:
a creation controller on each of the plurality of sites, said creation controller creating an indication of the notifications captured by the site;
a transfer controller on each of the plurality of sites, said transfer controller transferring the indication to at least one other of the plurality of sites.
30 . The system of claim 29 , wherein the plurality of sites are organized into a hierarchy, wherein each site in the hierarchy is assigned at least one of a master site and a subordinate site.
31 . The system of claim 30 , wherein each site performs at least one transfer of an indication, wherein the at least one transfer is performed to at least one of a subordinate site and a master site.
32 . The system of claim 31 , wherein each site performs a transfer to its master site and receives a transfer from its master site, and wherein as a result of the transfers, each site receives the indication created by each other site.
33 . The system of claim 31 , further comprising:
a receipt controller on a site, said receipt controller receiving a query requesting information on an event processed by a node from which the site does not capture notifications; and
a query controller, said query controller processing the query by accessing at least one indication transferred to the site by another site.
34 . The system of claim 33 , further comprising:
a determination controller, said determination controller determining, based on the at least one indication transferred to the site by another site, that the at least one indication does not contain the information requested by the query;
an identification controller, said identification controller identifying which of the other sites may have captured the information requested by the query;
a creation controller, said creation controller creating at least one additional query requesting the information from at least one other site identified by the identification controller; and
a transmission controller, said transmission controller transmitting the at least one additional query to the at least one other site.
35 . The system of claim 34 , further comprising:
a response controller, said response controller receiving a response for each of the at least one additional queries from the at least one other site; and
an aggregation controller, said aggregation controller aggregating the responses received.
36 . The system of claim 29 , wherein the creation controller on each of the plurality of sites further creates an indication which includes a representation of data stored in a file system on the respective site.
37 . The system of claim 29 , wherein each site captures notifications transmitted by nodes deployed in a geographic area.
38 . The system of claim 29 , wherein each site captures the notifications transmitted by a set of nodes, the set of nodes remaining unchanged.
39 . The system of claim 29 , wherein the plurality of sites comprises two portions, wherein a first portion includes sites configured to communicate with any other site in the first portion, wherein a second portion includes at least one site configured to communicate with only one site in the first portion, and wherein the system further comprises:
a remote collection controller on a site in the second portion, the remote collection controller capturing an indication of the notifications captured by the site;
a transfer controller, said transfer controller transferring the indication captured by the remote collection controller to a site in the first portion.
40 . The system of claim 39 , further comprising:
a local collection controller, said local collection controller transferring the indication received from the transfer controller to another site in the first portion.
41 . The system of claim 39 , wherein the transfer controller transfers the indication after a predetermined period of time elapses.
42 . The system of claim 39 , wherein the transfer controller transfers the indication upon a notification captured by the site in the second portion satisfying a predetermined criterion.
43 . A system for monitoring activity on a networked computer system, the networked computer system comprising a plurality of nodes, each of the plurality of nodes configured to transmit a notification for each event processed by the node, the networked computer system further comprising a plurality of sites, each of the plurality of sites being configured to capture the notifications transmitted by at least one node, comprising:
means for creating, on each of the plurality of sites, an indication of the notifications captured by the site;
means for transferring, on each of the plurality of sites, the indication to at least one other of the plurality of sites.
44 . The system of claim 43 , wherein the plurality of sites is organized into a hierarchy, and wherein each site in the hierarchy is assigned at least one of a master site and a subordinate site.
45 . The system of claim 44 , wherein each site performs at least one transfer of an indication, wherein the at least one transfer is performed to at least one of a subordinate site and a master site.
46 . The system of claim 45 , wherein each site performs a transfer to its master site and receives a transfer from its master site, and wherein as a result of the transfers, each site receives the indication created by each other site.
47 . The system of claim 46 , further comprising:
means for receiving a query requesting information on an event processed by a node from which the site does not capture notifications; and
means for processing the query by accessing at least one indication transferred to the site by another site.
48 . The system of claim 47 , further comprising:
means for determining, based on the at least one indication transferred to the site by another site, that the at least one indication does not contain the information requested by the query;
means for identifying which of the other sites may have captured the information requested by the query;
means for creating at least one additional query requesting the information from at least one other site identified by the identification controller; and
means for transmitting the at least one additional query to the at least one other site.
49 . The system of claim 48 , further comprising:
means for receiving a response for each of the at least one additional queries from the at least one other site; and
means for aggregating the responses received.
50 . The system of claim 43 , wherein the means for creating on each of the plurality of sites creates an indication which includes a representation 6 f data stored in a file system on the respective site.
51 . The system of claim 43 , wherein each site captures notifications transmitted by nodes deployed in a geographic area.
52 . The system of claim 43 , wherein each site captures the notifications transmitted by a set of nodes, the set of nodes remaining unchanged.
53 . The system of claim 43 , wherein the plurality of sites comprises two portions, wherein a first portion includes sites configured to communicate with any other site in the first portion, wherein a second portion includes at least one site configured to communicate with only one site in the first portion, and wherein the system further comprises:
means for capturing, on a site in the second portion, an indication of the notifications captured by the site;
means for transferring, to a site in the first portion, the indication captured by the means for capturing.
54 . The system of claim 53 , further comprising:
means for transferring the indication received at the site in the first portion to another site in the first portion.
55 . The system of claim 53 , wherein the means for transferring transfers the indication after a predetermined period of time elapses.
56 . The system of claim 53 , wherein the means for transferring transfers the indication upon a notification captured by the site in the second portion satisfying a predetermined criterion.