Network event capture and retention system
View Patent ↗Methods and apparatus are provided to monitor and analyze activity occurring on a networked computer system. In some embodiments, a method is provided for capturing, in a data structure, at least a portion of a notification describing a network event provided by a node on a computer network, identifying a data element (e.g., an IP address of the node) within the notification, and updating an index and/or summary based on the data element. The data structure may be stored in a file system maintained on a site, and sites may exchange information related to the notification data stored on each. In some embodiments, a query which is issued to a site may be processed using data transferred from other sites, and/or may be split into one or more additional queries which may be transmitted for processing to other sites.
1. A method of capturing and analyzing network events occurring on a computer network, the method comprising:
as notifications of the network events are transmitted by nodes within the computer network, identifying network characteristics of the notifications;
from the notifications of the network events and based on the identified network characteristics of the notifications, collecting network event data within separate observation record files; and
creating summaries of the network event data collected within the separate observation record files, each summary providing a measure of a particular identified network characteristic of the notifications;
wherein the computer network includes multiple distributed system sites, each distributed system site being configured to store a set of observation record files;
wherein creating the summaries of the network event data includes:
combining summaries of network event data for a first set of observation record files stored at a first distributed system site with a second set of observation record files stored at a second distributed system site to form an aggregate summary of network event data; and
wherein collecting the network event data within separate observation record files includes:
storing a first set of notifications in their entirety in the first set of observation record files, and
storing a second set of notifications in their entirety in the second set of observation record files.
2. A method as in claim 1 wherein creating the summaries of the network event data collected within the separate observation record files further includes:
providing a count of the number of times a particular Internet Protocol (IP) address occurs in observation records of a particular observation record file of the separate observation record files.
3. A method as in claim 2 wherein creating the summaries of the network event data collected within the separate observation record files further includes:
providing a count of the number of times the particular IP address occurs in observation records of another observation record file of the separate observation record files.
4. A method as in claim 3 wherein the particular observation record file contains observation records of network events from a first time period;
wherein the other observation record file contains observation records of network events from a second time period after the first time period; and
wherein combining the summaries of network event data includes:
aggregating the counts of the number of times the particular IP address occurs to form an overall count of the number of times the particular IP address occurs during the first and second time periods.
5. A method as in claim 1 wherein creating the summaries of the network event data collected within the separate observation record files further includes:
providing a measure of the amount of data transferred during particular Transmission Control Protocol (TCP) sessions represented in a particular observation record file of the separate observation record files.
6. A method as in claim 5 wherein creating the summaries of the network event data collected within the separate observation record files further includes:
providing a measure of the amount of data transferred during other TCP sessions represented in another observation record file of the separate observation record files.
7. A method as in claim 6 wherein the particular observation record file contains observation records of network events from a first time period;
wherein the other observation record file contains observation records of network events from a second time period after the first time period; and
wherein combining the summaries of network event data includes:
aggregating the measures to form an overall measure of the amount of data transferred during the particular TCP sessions and the other TCP sessions.
8. A method as in claim 1 wherein creating the summaries of the network event data collected within the separate observation record files further includes:
providing a duration of all Transmission Control Protocol (TCP) sessions represented in a particular observation record file of the separate observation record files.
9. A method as in claim 8 wherein creating the summaries of the network event data collected within the separate observation record files further includes:
providing a duration of all TCP sessions represented in another observation record file of the separate observation record files.
10. A method as in claim 9 wherein the particular observation record file contains observation records of network events from a first time period;
wherein the other observation record file contains observation records of network events from a second time period after the first time period; and
wherein combining the summaries of network event data includes:
aggregating the durations to form an aggregate duration measure of the durations of all TCP sessions represented in the particular observation record file and the other observation record file.
11. A method as in claim 1 wherein creating the summaries of the network event data collected within the separate observation record files includes:
providing a summary file having a header record section, an information record section, a type summary record section, and a file summary record section,
wherein the header record section identifies each section of the summary file,
wherein the information record section identifies the number of record types in the summary file,
wherein the type summary record section provides a value for the number of times that records a type appear in the summary file, and
wherein the file summary record section provides an indication of other files in which records of a particular type appear.
12. A method as in claim 11 wherein creating the summaries of the network event data collected within the separate observation record files further includes storing, in the summary file, at least one of:
(i) a count of the number of times a particular Internet Protocol (IP) address occurs in observation records of a particular observation record file of the separate observation record files,
(ii) a measure of the amount of data transferred during particular Transmission Control Protocol (TCP) sessions represented in a particular observation record file of the separate observation record files, and
(ii) a duration of all Transmission Control Protocol (TCP) sessions represented in a particular observation record file of the separate observation record files.
13. A method as in claim 1 wherein creating the summaries of the network event data further includes:
producing a series of summaries for a series of observation record files created at one-minute intervals, each summary of the series being produced for a particular one-minute interval; and
wherein combining the summaries of network event data includes:
aggregating summarized data within the series of summaries to produce a cumulative temporal summary of network event data collected for a period that is at least as long as an hour.
14. A method as in claim 1 wherein creating the summaries of the network event data includes:
producing a group of summaries for a group of observation record files created at a same time window during a group of days, each summary of the group of summaries being produced for the same time window during a different day of the group of days, and
wherein combining the summaries of network event data includes:
aggregating summarized data within the group of summaries to produce an aggregated summary of network event data collected for a period lasting the group of days.
15. A method as in claim 1 , further comprising:
while collecting network event data within separate observation record files, creating indexing files which are different from the separate observation record files and different from the summaries, each indexing file storing indices to network event data stored within a set of observation record files.
16. A method as in claim 1 , further comprising:
performing a set of electronic analysis operations on the created summaries to forensically ascertain aspects of a particular network characteristic of the computer network.
17. A computer program product having a non-transitory computer readable medium which stores a set of instructions to capture and analyze network events occurring on a computer network, the set of instructions, when carried out by computerized circuitry, causing the computerized circuitry to perform a method of:
as notifications of the network events are transmitted by nodes within the computer network, identifying network characteristics of the notifications;
from the notifications of the network events and based on the identified network characteristics of the notifications, collecting network event data within separate observation record files; and
creating summaries of the network event data collected within the separate observation record files, each summary providing a measure of a particular identified network characteristic of the notifications;
wherein the computer network includes multiple distributed system sites, each distributed system site being configured to store a set of observation record files;
wherein creating the summaries of the network event data includes:
combining summaries of network event data for a first set of observation record files stored at a first distributed system site with a second set of observation record files stored at a second distributed system site to form an aggregate summary of network event data; and
wherein collecting the network event data within separate observation record files includes:
storing a first set of notifications in their entirety in the first set of observation record files, and
storing a second set of notifications in their entirety in the second set of observation record files.
18. A computer program product as in claim 17 wherein the method further comprises:
while collecting network event data within separate observation record files, creating indexing files which are different from the separate observation record files and different from the summaries, each indexing file storing indices to network event data stored within a set of observation record files.
19. A computer program product as in claim 17 wherein the method further comprises:
performing a set of electronic analysis operations on the created summaries to forensically ascertain aspects of a particular network characteristic of the computer network.
20. Electronic apparatus, comprising:
memory; and
control circuitry coupled to the memory, the memory storing instructions which, when carried out by the control circuitry, cause the control circuitry to:
as notifications of the network events are transmitted by nodes within a computer network, identifying network characteristics of the notifications,
from the notifications of the network events and based on the identified network characteristics of the notifications, collecting network event data within separate observation record files, and
creating summaries of the network event data collected within the separate observation record files, each summary providing a measure of a particular identified network characteristic of the notifications;
wherein the computer network includes multiple distributed system sites, each distributed system site being configured to store a set of observation record files;
wherein the control circuitry, when creating the summaries of the network event data, is constructed and arranged to:
combine summaries of network event data for a first set of observation record files stored at a first distributed system site with a second set of observation record files stored at a second distributed system site to form an aggregate summary of network event data; and
wherein the control circuitry, when collecting the network event data within separate observation record files, is constructed and arranged to:
store a first set of notifications in their entirety in the first set of observation record files, and
store a second set of notifications in their entirety in the second set of observation record files.