Cache aware searching based on one or more files in remote storage
Embodiments are disclosed for performing cache aware searching. In response to a search query, a first bucket and a second bucket in remote storage for processing the search query. A determination is made that a first file in the first bucket is present in a cache when the search query is received. In response to the search query, a search is performed using the first file based on the determination that the first file is present in the cache when the search query is received, and the search is performed using a second file from the second bucket once the second file is stored in the cache.
1. A method, comprising:
in response to a search query, identifying a bucket to be searched;
determining that a first file of the bucket is stored in a remote storage;
determining a length of time to download the first file from the remote storage to a cache; and
based on a determination that the length of time satisfies a timing threshold, searching, in accordance with the search query, a second file of the bucket stored in the cache.
2. The method of claim 1 , wherein the first file comprises a gate file and the second file comprises an event file.
3. The method of claim 1 , wherein the second file comprises a plurality of events and the first file comprises summary information regarding the plurality of events.
4. The method of claim 1 , wherein the second file comprises a plurality of events and the first file comprises summary information regarding the plurality of events, wherein the first file comprises a host identifier, a source identifier, and a source type identifier for each of the plurality of events.
5. The method of claim 1 , further comprising determining the length of time satisfies the timing threshold.
6. The method of claim 1 , further comprising determining the second file is stored in the cache.
7. The method of claim 1 , wherein the search query is a first search query, the bucket is a first bucket, the remote storage is first remote storage, the length of time is a first length of time, the cache is a first cache, the timing threshold is a first timing threshold, the method further comprising:
in response to a second search query, identifying a second bucket to be searched;
determining that a third file of the second bucket is stored in a second remote storage;
determining a second length of time to download the third file from the second remote storage to a second cache; and
based on a determination that the second length of time does not exceed a second timing threshold, downloading the third file and searching the third file.
8. The method of claim 1 , wherein the search query is a first search query, the bucket is a first bucket, the remote storage is first remote storage, the length of time is a first length of time, the cache is a first cache, the timing threshold is a first timing threshold, the method further comprising:
in response to a second search query, identifying a second bucket to be searched;
determining that a third file of the second bucket is stored in a second remote storage;
determining a second length of time to download the third file from the second remote storage to a second cache; and
based on a determination that the second length of time does not exceed a second timing threshold, downloading the third file to the cache, searching the third file, and eliminating the second bucket from further searching in response to searching the third file, wherein eliminating the second bucket comprises not searching a remaining file of the second bucket.
9. The method of claim 1 , wherein the search query is a first search query, the bucket is a first bucket, the remote storage is first remote storage, the length of time is a first length of time, the cache is a first cache, the timing threshold is a first timing threshold, the method further comprising:
in response to a second search query, identifying a second bucket to be searched;
determining that a third file of the second bucket is stored in a second remote storage;
determining a second length of time to download the third file from the second remote storage to a second cache; and
based on a determination that the second length of time does not exceed a second timing threshold, downloading the third file to the cache, searching the third file, and searching a fourth file in response to searching the third file.
10. The method of claim 1 , wherein the search query is a first search query, the bucket is a first bucket, the remote storage is first remote storage, the length of time is a first length of time, the cache is a first cache, the timing threshold is a first timing threshold, the method further comprising:
in response to a second search query, identifying a second bucket to be searched;
determining that a third file of the second bucket is stored in a second remote storage, wherein the third file comprises summary information regarding a plurality of events, wherein the third file comprises a host identifier, a source identifier, and a source type identifier for each of the plurality of events;
determining a second length of time to download the third file from the second remote storage to a second cache;
based on a determination that the second length of time does not exceed a second timing threshold, downloading the third file to the cache;
comparing at least one of the host identifier, the source identifier, or the source type identifier to the second search query to obtain a comparison result indicating that the at least one of the host identifier, the source identifier, or the source type identifier does not match the second search query; and
eliminating the second bucket from further searching based on the comparison result, wherein eliminating the second bucket comprises not searching a remaining file in the second bucket.
11. The method of claim 1 , wherein the search query is a first search query, the bucket is a first bucket, the remote storage is first remote storage, the length of time is a first length of time, the cache is a first cache, the timing threshold is a first timing threshold, the method further comprising:
in response to a second search query, identifying a second bucket to be searched;
determining that a third file of the second bucket is stored in a second remote storage, wherein the third file comprises summary information regarding a plurality of events, wherein the third file comprises a host identifier, a source identifier, and a source type identifier for each of the plurality of events;
determining a second length of time to download the third file from the second remote storage to a second cache;
based on a determination that the second length of time does not exceed a second timing threshold, downloading the third file to the cache;
comparing at least one of the host identifier, the source identifier, or the source type identifier to the second search query to obtain a comparison result indicating that the at least one of the host identifier, the source identifier, or the source type identifier matches the second search query; and
searching a fourth file of the second bucket stored in the second cache based on the comparison result.
12. The method of claim 1 , further comprising:
identifying an order on a list of buckets for processing the search query, wherein the bucket is processed out of the order based on determining the second file is stored in the cache.
13. The method of claim 1 , further comprising:
based on a relative size of the second file in the cache, selecting the second file for eviction after searching the second file.
14. The method of claim 1 , wherein the second file comprises a plurality of events and the first file comprises summary information regarding the plurality of events, wherein each of the plurality of events comprises raw machine data associated with a timestamp.
15. The method of claim 1 , wherein the remote storage is located in a cloud storage or a storage in an on-premises environment.
16. The method of claim 1 , wherein the second file comprises the first file, wherein searching the second file stored in the cache comprises downloading the first file to the cache and searching the first file.
17. The method of claim 1 , wherein the first file and the second file comprise different files.
18. The method of claim 1 , wherein searching the second file comprises searching at least a portion of the bucket.
19. Non-transitory computer-readable media comprising computer-executable instructions, wherein execution of the computer-executable instructions causes one or more processors to:
in response to a search query, identify a bucket to be searched;
determine that a first file of the bucket is stored in a remote storage;
determine a length of time to download the first file from the remote storage to a cache; and
based on a determination that the length of time satisfies a timing threshold, search, in accordance with the search query, a second file of the bucket stored in the cache.
20. A computer system, comprising:
a data store comprising a cache; and
one or more processors communicatively coupled to the data store and configured to:
in response to a search query, identify a bucket to be searched;
determine that a first file of the bucket is stored in a remote storage;
determine a length of time to download the first file from the remote storage to the cache; and
based on a determination that the length of time satisfies a timing threshold, search, in accordance with the search query, a second file of the bucket stored in the cache.