System for organizing and fast searching of massive amounts of data
A system to collect and analyze performance metric data recorded in time-series measurements, converted into unicode, and arranged into a special data structure. The performance metric data is collected by one or more probes running on machines about which data is being collected. The performance metric data is also organized into a special data structure. The data structure at the server where analysis is done has a directory for every day of performance metric data collected with a subdirectory for every resource type. Each subdirectory contain text files of performance metric data values measured for attributes in a group of attributes to which said text file is dedicated. Each attribute has its own section and the performance metric data values are recorded in time series as unicode hex numbers as a comma delimited list. Analysis of the performance metric data is done using regular expressions.
1 . A process comprising:
receiving time-series performance metric data from probes; and
encoding said time-series performance metric data in a unicode format and storing the unicode encoded performance metric values.
2 . The process of claim 1 further comprising storing said unicoded performance metric values collected for each day in a file system which has one directory for each separate day, said directory having a separate sub directory for each type of resource for which performance metric data was collected.
3 . The process of claim 2 wherein each subdirectory stores files which store unicoded performance metric values as text files.
4 . The process of claim 3 wherein each text file stored therein unicoded performance metric values for a group of attributes related to the performance of the resource type corresponding to said subdirectory.
5 . The process of claim 4 wherein each text file has a separate section storing unicoded performance metric values for just one attribute.
6 . The process of claim 5 wherein each said section stores the unicoded performance metric values for the attribute corresponding to said section as a comma delimited list.
7 . A process comprising:
gathering performance metric data for one or more attributes of one or more resources using one or more probe programs;
organizing the performance metric data collected into a data structure having:
a directory dedicated to storage of performance metric data collected for all attributes of all resource types on one calendar day;
a separate subdirectory for every type resource for which performance metric data was collected;
a separate text file in each subdirectory storing all performance metric data for a group of attributes related to a particular resource within the resource group to which said subdirectory is devoted; and
within each said text file, a separate section storing performance metric data measurement values for each attribute in the group of attributes stored in said text file, said measurement values stored as a comma delimited list;
compressing the data structure created in the previous step; and
transmitting said compressed data structure over any data path to another computer for analysis.
8 . The process of claim 7 further comprising the step of creating a first file in said data structure which comprises a list of files in the data structure and including that list of files file as the first file in the collection of files which are compressed.
9 . The process of claim 8 further comprising the step of collecting configuration data and creating a configuration file which is included in the data structure which is compressed and transmitted.
10 . The process of claim 9 further comprising the step of creating a snapshot file as part of said configuration file, said snapshot file containing the entire configuration data about the resource on which said probe is executing and collecting data upon, said snapshot file containing a site name which is the location where said probe is being executed, a data source which identifies the data source or resource about which said probe is collecting data, and the date and time the snapshot was created including the beginning time and ending time for which the configuration data is valid, the resource type and ID for each resource reported upon, the attribute ID, the resource type and any event ID and, the version of the file.
11 . The process of claim 10 further comprising the step of creating a minisnapshot file each time a configuration change or configuration related event occurs, said minisnapshot file having the same format as said snapshot file.