Cache aware searching based on one or more files in one or more buckets in remote storage
Embodiments are disclosed for performing cache aware searching. In response to a search query, a first bucket and a second bucket in remote storage for processing the search query. A determination is made that a first file in the first bucket is present in a cache when the search query is received. In response to the search query, a search is performed using the first file based on the determination that the first file is present in the cache when the search query is received, and the search is performed using a second file from the second bucket once the second file is stored in the cache.
1. A method, comprising:
in response to a search query, identifying a bucket to be searched;
determining that an event file associated with the bucket is stored in a cache, wherein the event file comprises a plurality of events;
determining that a gate file associated with the bucket is stored in a remote storage, wherein the gate file comprises summary information regarding the plurality of events of the event file;
determining a length of time to download the gate file to the cache; and
based on the determining the length of time, performing a search of at least a portion of the bucket in accordance with the search query,
wherein based on a determination that the length of time satisfies a timing threshold, performing the search of the at least a portion of the bucket includes searching the event file.
2. The method of claim 1 , wherein based on a determination that the length of time to download the gate file to the cache does not exceed the timing threshold, performing the search of the at least a portion of the bucket includes downloading the gate file and searching the gate file.
3. The method of claim 1 , wherein based on a determination that the length of time to download the gate file to the cache does not exceed the timing threshold, performing the search of the at least a portion of the bucket includes:
downloading the gate file to the cache;
searching the gate file; and
eliminating the bucket from further searching in response to searching the gate file, wherein eliminating the bucket comprises not searching a remaining file of the bucket.
4. The method of claim 1 , wherein based on a determination that the length of time to download the gate file to the cache does not exceed the timing threshold, performing the search of the at least a portion of the bucket includes:
downloading the gate file to the cache;
searching the gate file; and
searching the event file in response to searching the gate file.
5. The method of claim 1 , wherein the gate file comprises a host identifier, a source identifier, and a source type identifier for each of the plurality of events.
6. The method of claim 1 , wherein the gate file comprises a host identifier, a source identifier, and a source type identifier, and wherein based on a determination that the length of time to download the gate file to the cache does not exceed the timing threshold, performing the search of the at least a portion of the bucket includes:
downloading the gate file to the cache;
comparing at least one of the host identifier, the source identifier, or the source type identifier to the search query to obtain a comparison result indicating that the at least one of host identifier, source identifier, or the source type identifier does not match the search query; and
eliminating the bucket from further searching based on the comparison result, wherein eliminating the bucket comprises not searching a remaining file in the bucket.
7. The method of claim 1 , wherein the gate file comprises a host identifier, a source identifier, and a source type identifier, and wherein based on a determination that the length of time to download the gate file to the cache does not exceed the timing threshold, performing the search of the at least a portion of the bucket includes:
downloading the gate file to the cache;
comparing at least one of the host identifier, the source identifier, or the source type identifier to the search query to obtain a comparison result indicating that the at least one of host identifier, source identifier, or the source type identifier matches the search query; and
searching the event file based on the comparison result.
8. The method of claim 1 , further comprising:
identifying an order on a list of buckets for processing the search query, wherein the bucket is processed out of the order based on the determining the event file is stored in the cache.
9. The method of claim 1 , further comprising:
based on a relative size of the event file in the cache, selecting the event file for eviction after performing the search of the bucket.
10. The method of claim 1 , wherein each of the plurality of events comprises raw machine data associated with a timestamp.
11. The method of claim 1 , wherein the remote storage is located in a cloud storage or a storage in an on-premises environment.
12. A computer system, comprising:
a data store comprising a cache; and
one or more processors communicatively coupled to the data store and configured to:
in response to a search query, identifying a bucket to be searched;
determine that an event file associated with the bucket is stored in the cache, wherein the event file comprises a plurality of events;
determine that a gate file associated with the bucket is stored in a remote storage, wherein the gate file comprises summary information regarding the plurality of events of the event file;
determine a length of time to download the gate file to the cache; and
based on a determination of the length of time, perform a search of at least a portion of the bucket in accordance with the search query,
wherein to perform the search of at least a portion of the bucket, the one or more processors are configured to, based on a determination that the length of time satisfies a timing threshold, search the event file.
13. The computer system of claim 12 , wherein to perform the search of at least a portion of the bucket, the one or more processors are configured to, based on a determination that the length of time to download the gate file to the cache does not exceed the timing threshold, download the gate file, and search the gate file.
14. The computer system of claim 12 , wherein to perform the search of at least a portion of the bucket, the one or more processors are configured to, based on a determination that the length of time to download the gate file to the cache does not exceed the timing threshold:
download the gate file to the cache;
search the gate file; and
search the event file in response to the search of the gate file.
15. The computer system of claim 12 , wherein the gate file comprises a host identifier, a source identifier, and a source type identifier for each of the plurality of events.
16. The computer system of claim 12 ,
wherein the gate file comprises a host identifier, a source identifier, and a source type identifier, and
wherein to perform the search of at least a portion of the bucket, the one or more processors are configured to, based on a determination that the length of time to download the gate file to the cache does not exceed the timing threshold:
download the gate file to the cache;
compare at least one of the host identifier, the source identifier, or the source type identifier to the search query to obtain a comparison result indicating that the at least one of host identifier, source identifier, or the source type identifier matches the search query; and
search the event file based on the comparison result.
17. Non-transitory computer-readable media comprising computer-executable instructions, wherein execution of the computer-executable instructions causes one or more processors to:
in response to a search query, identify a bucket to be searched;
determine that an event file associated with the bucket is stored in a cache, wherein the event file comprises a plurality of events;
determine that a gate file associated with the bucket is stored in a remote storage, wherein the gate file comprises summary information regarding the plurality of events of the event file;
determine a length of time to download the gate file to the cache; and
based on a determination of the length of time, perform a search of at least a portion of the bucket in accordance with the search query,
wherein to perform the search of at least a portion of the bucket, the execution of the computer-executable instructions causes the one or more processors to, based on a determination that the length of time satisfies a timing threshold, search the event file.
18. The non-transitory computer-readable media of claim 17 , wherein to perform the search of at least a portion of the bucket, the execution of the computer-executable instructions causes the one or more processors to, based on a determination that the length of time to download the gate file to the cache does not exceed the timing threshold:
download the gate file to the cache;
search the gate file; and
eliminate the bucket from further searching in response to searching the gate file, wherein elimination of the bucket comprises not searching a remaining file of the bucket.
19. The non-transitory computer-readable media of claim 17 , wherein to perform the search of at least a portion of the bucket, the execution of the computer-executable instructions causes the one or more processors to, based on a determination that the length of time to download the gate file to the cache does not exceed the timing threshold:
download the gate file to the cache;
search the gate file; and
search the event file in response to the search of the gate file.
20. The non-transitory computer-readable media of claim 17 ,
wherein the gate file comprises a host identifier, a source identifier, and a source type identifier, and
wherein to perform the search of at least a portion of the bucket, the execution of the computer-executable instructions causes the one or more processors to, based on a determination that the length of time to download the gate file to the cache does not exceed the timing threshold:
download the gate file to the cache;
compare at least one of the host identifier, the source identifier, or the source type identifier to the search query to obtain a comparison result indicating that the at least one of host identifier, source identifier, or the source type identifier matches the search query; and
search the event file based on the comparison result.