Cache aware searching
Embodiments are disclosed for performing cache aware searching. In response to a search query, a first bucket and a second bucket in remote storage for processing the search query. A determination is made that a first file in the first bucket is present in a cache when the search query is received. In response to the search query, a search is performed using the first file based on the determination that the first file is present in the cache when the search query is received, and the search is performed using a second file from the second bucket once the second file is stored in the cache.
1 . A method comprising:
obtaining, at a cache and from remote storage, a metadata file associated with a bucket of data, wherein the bucket of data stored at the remote storage includes the metadata file, an index file, and a journal file;
determining that the index file of the bucket of data is predicted to contain data relevant to a search query;
based on the determination that the index file of the bucket of data is predicted to contain data relevant to the search query, obtaining, at the cache and from the remote storage, the index file associated with the bucket of data;
determining that the journal file of the bucket of data is predicted to include data relevant to the search query; and
based on the determination that the journal file of the bucket of data is predicted to include data relevant to the search query, obtaining, at the cache and from the remote storage, the journal file associated with the bucket of data.
2 . The method of claim 1 , further comprising updating a location of the index file of the bucket of data in a cache map to indicate a cache address corresponding to a location of the metadata file in the cache.
3 . The method of claim 2 , further comprising communicating the cache address of the index file of the bucket of data to a search process.
4 . The method of claim 3 , further comprising assigning a usage status to the index file of the bucket of data.
5 . The method of claim 1 , further comprising generating a result of evaluating a query using the index file of the bucket of data.
6 . The method of claim 1 , wherein the metadata file is obtained based on a prediction that the metadata file will be used during execution of a query relative to the bucket of data.
7 . The method of claim 6 , wherein the prediction is based on a lack of any usage status in a file usage history for one or more metadata files in previously processed buckets of data.
8 . The method of claim 6 , wherein the prediction is based on a usage status in at least one of an immediately preceding bucket of data of a same file type.
9 . The method of claim 1 , further comprising updating a location of the metadata file in a cache map to indicate a cache address corresponding to a location of the metadata file in the cache.
10 . The method of claim 1 , wherein the determining that the journal file of the bucket of data is predicted to include data relevant to the search query is based on a usage status in a file usage history for index files in one or more buckets of data.
11 . The method of claim 1 , further comprising using at least one of the metadata file, the index file, and the journal file to execute the search query.
12 . The method of claim 1 , wherein the bucket of data comprises at least one event, each event in the at least one event comprises a portion of raw machine data associated with a timestamp.
13 . The method of claim 1 , wherein the bucket of data is associated with a first time frame.
14 . The method of claim 1 , wherein the remote storage is located in a cloud storage or a storage in an on-premises environment.
15 . A computer system, comprising:
a processor; and
a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations including:
obtaining, at a cache and from remote storage, a metadata file associated with a bucket of data, wherein the bucket of data stored at the remote storage includes the metadata file, an index file, and a journal file;
determining that the index file of the bucket of data is predicted to contain data relevant to a search query;
based on the determination that the index file of the bucket of data is predicted to contain data relevant to the search query, obtaining, at the cache and from the remote storage, the index file associated with the bucket of data;
determining that the journal file of the bucket of data is predicted to include data relevant to the search query; and
based on the determination that the journal file of the bucket of data is predicted to include data relevant to the search query, obtaining, at the cache and from the remote storage, the journal file associated with the bucket of data.
16 . The computer system of claim 15 , wherein the bucket of data comprises at least one event, each event in the at least one event comprises a portion of raw machine data associated with a timestamp.
17 . The computer system of claim 15 , wherein the determining that the journal file of the bucket of data is predicted to include data relevant to the search query is based on a usage status in a file usage history for index files in one or more buckets of data.
18 . The computer system of claim 15 , wherein the metadata file is obtained based on a prediction that the metadata file will be used during execution of a query relative to the bucket of data.
19 . The non-transitory computer-readable medium of claim 15 , wherein the determining that the journal file of the bucket of data is predicted to include data relevant to the search query is based on a usage status in a file usage history for index files in one or more buckets of data.
20 . The computer system of claim 15 , wherein the bucket of data is associated with a first time frame.
21 . A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processor to perform operations including:
obtaining, at a cache and from remote storage, a metadata file associated with a bucket of data, wherein the bucket of data stored at the remote storage includes the metadata file, an index file, and a journal file;
determining that the index file of the bucket of data is predicted to contain data relevant to a search query;
based on the determination that the index file of the bucket of data is predicted to contain data relevant to the search query, obtaining, at the cache and from the remote storage, the index file associated with the bucket of data;
determining that the journal file of the bucket of data is predicted to include data relevant to the search query; and
based on the determination that the journal file of the bucket of data is predicted to include data relevant to the search query, obtaining, at the cache and from the remote storage, the journal file associated with the bucket of data.