Indexing and searching of large amounts of machine generated data collected from disparate sources
A computer-implemented method of searching large amounts of machine generated data collected from disparate sources, comprises steps of: receiving data from the disparate sources into a multi-level system of storage blocks; high-speed searching at a broad level for groups of the storage blocks containing records of interest; nested searching at an intermediate level to determine specific storage blocks containing the records of interest, within the groups of the storage blocks; distributed searching at a fine level to identify the records of interest, within the specific storage blocks; and displaying the records of interest to a user. In a computer data storage device, a data structure comprises: plural data files each holding a data group from a large data set, wherein the data groups are permitted to contain overlapping data; and plural index trees within each data file, each index tree representing a key into a data group; wherein an absence of a key in one data file results in an absence of a corresponding index tree in the one data file. A computer storage medium has stored thereon instructions for a computer to perform the method described above.
1. A computer-implemented method of searching large amounts of machine generated data collected from disparate sources, comprising steps of:
receiving data from the disparate sources into a multi-level system of storage blocks;
high-speed searching at a broad level for groups of the storage blocks containing records of interest;
nested searching at an intermediate level in which the multi-level system of storage blocks are iteratively searched to determine specific storage blocks containing the records of interest, within the groups of the storage blocks;
distributed searching in which searching resources are load balanced among the specific storage blocks at a fine level to identify the records of interest, within the specific storage blocks; and
indexing the data received using plural index databases corresponding to plural keys into the data received;
managing each index database as a separate index tree;
grouping the data received into sets defined by time periods; and
performing indexing and managing within one set separately from within another set, so the one set can be separately searched from the another set;
indexing on any field within the data received;
adding to one of the plural index databases a new field when found in the data received; and
removing from one of the plural index databases a filed no longer used in the data received;
displaying the records of interest to a user.
2. The method of claim 1 , wherein high-speed searching further comprises:
applying a first search criterion by which a distribution of the records of interest amongst the groups of storage blocks is identified.
3. The method of claim 2 , wherein nested searching further comprises:
creating and adjusting a time range filter to include in the specific storage blocks, only those storage blocks covering a time range when the records of interest are created.
4. The method of claim 1 , wherein displaying further comprises:
arranging the records of interest in time sequence.
5. A non-transitory computer storage medium on which are stored instructions for a computer to perform a method of searching large amounts of machine generated data collected from disparate sources, comprising steps of:
receiving data from the disparate sources into a multi-level system of storage blocks;
high-speed searching at a broad level for groups of the storage blocks containing records of interest;
nested searching at an intermediate level in which the multi-level system of storage blocks are iteratively searched to determine specific storage blocks containing the records of interest, within the groups of the storage blocks;
distributed searching in which searching resources are load balanced among the specific storage blocks at a fine level to identify the records of interest, within the specific storage blocks; and
indexing the data received using plural index databases corresponding to plural keys into the data received;
managing each index database as a separate index tree;
grouping the data received into sets defined by time periods; and
performing indexing and managing within one set separately from within another set, so the one set can be separately searched from the another set;
indexing on any field within the data received;
adding to one of the plural index databases a new field when found in the data received; and
removing from one of the plural index databases a filed no longer used in the data received;
displaying the records of interest to a user.
6. The storage medium of claim 5 , wherein high-speed searching further comprises:
applying a first search criterion by which a distribution of the records of interest amongst the groups of storage blocks is identified.
7. The storage medium of claim 6 , wherein nested searching further comprises:
creating and adjusting a time range filter to include in the specific storage blocks, only those storage blocks covering a time range when the records of interest are created.
8. The storage medium of claim 5 , wherein displaying further comprises:
arranging the records of interest in time sequence.