Granted Patent B2

US 12,436,963 · App. 18/748,595 · Granted Oct 7, 2025

Inventors: Alexandros Batsakis (San Francisco, CA); Nitilaksha Satyaveera Halakatti (San Jose, CA); Ningxuan He (San Francisco, CA); Prem Kumar Jayaraj (Sunnyvale, CA); Manuel Gregorio Martinez (San Francisco, CA); Balaji Rao (San Jose, CA); Jianming Zhang (Campbell, CA); Steve Yu Zhang (San Francisco, CA)

G06F16/2471

Abstract

Claims (68)

1. A method, comprising:

receiving, at a query system, a query, wherein the query identifies a set of data and includes a plurality of computer-executable instructions that when executed by one or more processors causes the one or more processors to transform the set of data and generate a query result;

identifying an external data system in which at least a portion of the set of data is stored;

requesting, from the external data system, metadata corresponding to the at least a portion of set of data, wherein the metadata comprises location information for different portions of the at least a portion of the set of data within the external data system;

receiving the metadata from the external data system;

communicating the metadata to a buffer;

instructing a plurality of search nodes to retrieve the metadata from the buffer,

wherein the plurality of search nodes retrieve metadata from the buffer and use the location information from the metadata to retrieve the at least a portion of the set of data from the external data system,

wherein, based on a first set of the plurality of computer-executable instructions, the plurality of search nodes transform the set of data and generate a partial query result;

receiving the partial query result from the plurality of search nodes;

processing the partial query result based on a second set of the plurality of computer-executable instructions to generate the query result; and

communicating the query result to a client computing device.

2. The method of claim 1 , wherein the external data system is a shared storage system that stores data in a format that is different than data stored by a data intake and query system associated with the query system.

3. The method of claim 1 , wherein identifying the external data system comprises parsing the query to identify a reference to the external data system.

4. The method of claim 1 , further comprising parsing the query to identify one or more filter criteria, wherein the one or more filter criteria include at least one of a field identifier, a field value, or time.

5. The method of claim 4 , further comprising:

parsing the query to identify a plurality of filter criteria, wherein the plurality of filter criteria include the one or more filter criteria; and

communicating the plurality of filter criteria to the external data system, wherein receiving the metadata from the external data system comprises receiving the metadata from the external data system in response to the communicating the one or more filter criteria to the external data system.

6. The method of claim 1 , further comprising:

parsing the query to identify a plurality of filter criteria;

selecting one or more filter criteria from the plurality of filter criteria based on a determined functionality of the external data system; and

communicating the one or more filter criteria to the external data system, wherein receiving the metadata from the external data system comprises receiving the metadata from the external data system in response to the communicating the one or more filter criteria to the external data system.

7. The method of claim 1 , wherein the buffer is configured to retain metadata corresponding to a particular data object of the at least a portion of the set of data until the buffer receives a notification from a search node that the particular data object has been processed.

8. The method of claim 1 , wherein a particular search node is configured to iteratively retrieve metadata corresponding to different data objects of the at least a portion of the set of data from the buffer and processes the different data objects until all the metadata in the buffer has been processed.

9. The method of claim 1 , wherein each search node of the plurality of search nodes is configured to iteratively obtain metadata corresponding to different data objects from the buffer and process the different data objects corresponding to the metadata.

10. A system, comprising:

a data store; and

one or more computing devices, the one or more computing devices configured to:

receive a query, wherein the query identifies a set of data and includes a plurality of computer-executable instructions that when executed by one or more processors causes the one or more processors to transform the set of data and generate a query result;

identify an external data system in which at least a portion of the set of data is stored;

request, from the external data system, metadata corresponding to the at least a portion of set of data, wherein the metadata comprises location information for different portions of the at least a portion of the set of data within the external data system;

receive the metadata from the external data system;

communicate the metadata to a buffer;

instruct a plurality of search nodes to retrieve the metadata from the buffer,

wherein the plurality of search nodes retrieve metadata from the buffer and use the location information from the metadata to retrieve the at least a portion of the set of data from the external data system,

wherein, based on a first set of the plurality of computer-executable instructions, the plurality of search nodes transform the set of data and generate a partial query result;

receive the partial query result from the plurality of search nodes;

process the partial query result based on a second set of the plurality of computer-executable instructions to generate the query result; and

communicate the query result to a client computing device.

11. The system of claim 10 , wherein the external data system is a shared storage system that stores data in a format that is different than data stored by a data intake and query system associated with the query system.

12. The system of claim 10 , wherein to identify the external data system, the one or more processors are configured to identify a reference to the external data system.

13. The system of claim 10 , wherein the one or more processors are further configured to parse the query to identify one or more filter criteria, wherein the one or more filter criteria include at least one of a field identifier, a field value, or time.

14. The system of claim 13 , wherein the one or more processors are further configured to:

parse the query to identify a plurality of filter criteria, wherein the plurality of filter criteria include the one or more filter criteria; and

communicate the plurality of filter criteria to the external data system, wherein to receive the metadata from the external data system, the one or more processors are configured to receive the metadata from the external data system in response to communication of the one or more filter criteria to the external data system.

15. The system of claim 10 , wherein the one or more processors are further configured to:

parse the query to identify a plurality of filter criteria;

select one or more filter criteria from the plurality of filter criteria based on a determined functionality of the external data system; and

communicate the one or more filter criteria to the external data system wherein to receive the metadata from the external data system, the one or more processors are configured to receive the metadata from the external data system in response to communication of the one or more filter criteria to the external data system.

16. The system of claim 10 , wherein the buffer is configured to retain metadata corresponding to a particular data object of the at least a portion of the set of data until the buffer receives a notification from a search node that the particular data object has been processed.

17. The system of claim 10 , wherein a particular search node is configured to iteratively obtain metadata corresponding to different data objects of the at least a portion of the set of data from the buffer and processes the different data objects until all the metadata in the buffer has been processed.

18. The system of claim 10 , wherein each search node of the plurality of search nodes is configured to iteratively obtain metadata corresponding to different data objects from the buffer and process the different data objects corresponding to the metadata.

19. Non-transitory computer-readable media including first computer-executable instructions that, when executed by a computing device of a query system, cause the computing device to:

receive a query, wherein the query identifies a set of data and includes a plurality of computer-executable instructions that when executed by one or more processors causes the one or more processors to transform the set of data and generate a query result;

identify an external data system in which at least a portion of the set of data is stored;

request, from the external data system, metadata corresponding to the at least a portion of set of data, wherein the metadata comprises location information for different portions of the at least a portion of the set of data within the external data system;

receive the metadata from the external data system;

communicate the metadata to a buffer;

instruct a plurality of search nodes to retrieve the metadata from the buffer,

wherein the plurality of search nodes retrieve metadata from the buffer and use the location information from the metadata to retrieve the at least a portion of the set of data from the external data system,

wherein, based on a first set of the plurality of computer-executable instructions, the plurality of search nodes transform the set of data and generate a partial query result;

receive the partial query result from the plurality of search nodes;

process the partial query result based on a second set of the plurality of computer-executable instructions to generate the query result; and

communicate the query result to a client computing device.

20. The non-transitory computer-readable media of claim 19 , wherein the first computer-executable instructions further cause the one or more processors to:

parse the query to identify a plurality of filter criteria;

select one or more filter criteria from the plurality of filter criteria based on a determined functionality of the external data system; and

communicate the one or more filter criteria to the external data system, wherein to receive the metadata from the external data system, the first computer-executable instructions cause the computing device to receive the metadata from the external data system in response to communication of the one or more filter criteria to the external data system.

Assignments (4)

Continuity (2)

References Cited (400)