Efficient management of data storage and retrieval
Embodiments described herein are directed to facilitating efficient management of data storage and retrieval. In one embodiment, filter data associated with a bucket of data is obtained at a local data store from a remote data store. Based on analysis of the filter data, it is determined that the bucket of data is a candidate to contain data relevant to a search query. Based on such a determination, the index data associated with the bucket of data is obtained at the local data store from the remote data store. Thereafter, it may be determined that the bucket of data includes data relevant to the search query based on analysis of the index data. Based on the determination that the bucket of data includes data relevant to the search query, the journal data associated with the bucket of data is obtained at the local data store from the remote data store.
1 . A computer-implemented method, comprising:
obtaining, at a local data store and from a remote data store, filter data associated with a bucket of data, wherein the bucket of data stored at the remote data store includes the filter data, index data, and journal data;
determining that the bucket of data is a candidate to contain data relevant to a search query based on analysis of the filter data associated with the bucket of data;
based on the determination that the bucket of data is the candidate to contain data relevant to the search query, obtaining, at the local data store and from the remote data store, the index data associated with the bucket of data;
determining that the bucket of data includes data relevant to the search query based on analysis of the index data associated with the bucket of data;
based on the determination that the bucket of data includes data relevant to the search query, obtaining, at the local data store and from the remote data store, the journal data associated with the bucket of data; and
in accordance with completing the analysis of the index data associated with the bucket of data, analyzing a subsequent bucket of data to determine whether the subsequent bucket of data is a new candidate to contain data relevant to the search query.
2 . The computer-implemented method of claim 1 further comprising prefetching filter data associated with a set of buckets of data, including the bucket of data, in association with obtaining the search query.
3 . The computer-implemented method of claim 1 , wherein the analysis of the index data associated with the bucket of data is performed upon analyzing filter data associated with a predetermined number of buckets.
4 . The computer-implemented method of claim 1 , wherein the analysis of the index data associated with the bucket of data is performed upon determining that index data is downloaded to the local data store in association with a predetermined number of unread buckets.
5 . The computer-implemented method of claim 1 , wherein
analyzing the subsequent bucket of data determines that the subsequent bucket of data is eliminated as the new candidate to contain data relevant to the search query; and
based on the subsequent bucket of data being eliminated, analyzing a new subsequent bucket without downloading index data associated with the subsequent bucket of data to the local data store.
6 . The computer-implemented method of claim 1 further comprising analyzing the journal data associated with the bucket of data to extract data relevant to the search query.
7 . The computer-implemented method of claim 1 further comprising extracting data relevant to the search query from the journal data associated with the bucket of data and providing the extracted data relevant to the search query to a search head for providing a set of search results to a user device in response to the search query.
8 . The computer-implemented method of claim 1 , wherein upon the analysis of the index data associated with the bucket of data, initiating downloading of the journal data associated with the bucket of data to the local data store, initiating downloading of filter data associated with a first subsequent bucket of data, and analyzing filter data associated with a second subsequent bucket of data.
9 . The computer-implemented method of claim 1 , wherein the filter data comprises metadata and bloom filter data.
10 . The computer-implemented method of claim 1 , wherein the index data comprises time-series index file data.
11 . The computer-implemented method of claim 1 , wherein the journal data comprises raw event data.
12 . The computer-implemented method of claim 1 , wherein the remote data store comprises a third-party data store accessible over a network.
13 . A computing device, comprising:
a processor; and
a non-transitory computer-readable medium having stored thereon instructions when executed by the processor, cause the processor to perform operations including:
obtaining, at a local data store and from a remote data store, filter data associated with a bucket of data, wherein the bucket of data stored at the remote data store includes the filter data, index data, and journal data;
determining that the bucket of data is a candidate to contain data relevant to a search query based on analysis of the filter data associated with the bucket of data;
based on the determination that the bucket of data is the candidate to contain data relevant to the search query, obtaining, at the local data store and from the remote data store, the index data associated with the bucket of data;
determining that the bucket of data includes data relevant to the search query based on analysis of the index data associated with the bucket of data;
based on the determination that bucket of data includes data relevant to the search query, obtaining, at the local data store and from the remote data store, the journal data associated with the bucket of data; and
in accordance with completing the analysis of the index data associated with the bucket of data, analyzing a subsequent bucket of data to determine whether the subsequent bucket of data is a new candidate to contain data relevant to the search query.
14 . The computing device of claim 13 , wherein the analysis of the index data associated with the bucket of data is performed upon analyzing filter data associated with a predetermined number of buckets.
15 . A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processor to perform operations including:
obtaining, at a local data store and from a remote data store, filter data associated with a bucket of data, wherein the bucket of data stored at the remote data store includes the filter data, index data, and journal data;
determining that the bucket of data is a candidate to contain data relevant to a search query based on analysis of the filter data associated with the bucket of data;
based on the determination that the bucket of data is the candidate to contain data relevant to the search query, obtaining, at the local data store and from the remote data store, the index data associated with the bucket of data;
determining that the bucket of data includes data relevant to the search query based on analysis of the index data associated with the bucket of data; and
based on the determination that bucket of data includes data relevant to the search query, obtaining, at the local data store and from the remote data store, the journal data associated with the bucket of data; and
in accordance with completing the analysis of the index data associated with the bucket of data, analyzing a subsequent bucket of data to determine whether the subsequent bucket of data is a new candidate to contain data relevant to the search query.
16 . The medium of claim 15 , wherein the analysis of the index data associated with the bucket of data is performed upon determining that index data is downloaded to the local data store in association with a predetermined number of unread buckets.
17 . The medium of claim 15 , wherein
analyzing the subsequent bucket of data determines that the subsequent bucket of data is eliminated as the new candidate to contain data relevant to the search query; and
based on the subsequent bucket of data being eliminated, analyzing a new subsequent bucket without downloading index data associated with the subsequent bucket of data to the local data store.
18 . The medium of claim 15 , wherein upon the analysis of the index data associated with the bucket of data, initiating downloading of the journal data associated with the bucket of data to the local data store, initiating downloading of filter data associated with a first subsequent bucket of data, and analyzing filter data associated with a second subsequent bucket of data.