Data fabric service system deployment
Disclosed is a data fabric service system that can be implemented in a distributed computer network, such as a data intake and query system. The data index and query system can receive a search query and define a search scheme for applying the search query on distributed data storage systems including internal data storage and external data storage. The data index and query system may provide a portion of the search scheme to a search service of the data fabric service system, which can cause worker nodes of the data fabric service system to perform various functions—including applying the search query to the external data storage based on the portion of the search scheme in order to obtain search results.
1. A method, comprising:
defining, by a data intake and query system, a search scheme based at least in part on a query, wherein the data intake and query system comprises one or more worker nodes, a search service provider, and one or more indexers;
triggering, by the data intake and query system, the one or more indexers to search one or more internal data stores based at least in part on a first portion of the search scheme to obtain first partial search results; and
providing, by the data intake and query system, a second portion of the search scheme to the search service provider, wherein the search service provider:
defines, based at least in part on the second portion of the search scheme, a search process executable by the one or more worker nodes, and
directs the one or more worker nodes to communicate with one or more external data sources to obtain second partial search results based at least in part on the second portion of the search scheme.
2. The method of claim 1 , wherein the one or more worker nodes are configured to:
receive the search process from the search service provider, and
based on the search process, communicate with the one or more external data sources to obtain the second partial search results.
3. The method of claim 1 , wherein the one or more worker nodes are configured to:
receive the search process from the search service provider, and
based on the search process:
communicate with the one or more external data sources to obtain the second partial search results,
aggregate the second partial search results, and
provide aggregated partial search results to the search service provider.
4. The method of claim 1 , wherein the one or more worker nodes are communicatively coupled to the search service provider.
5. The method of claim 1 , wherein the data intake and query system further comprises a search head.
6. The method of claim 1 , wherein the data intake and query system further comprises a search head, wherein defining the search scheme comprises defining the search scheme by the search head.
7. The method of claim 1 , wherein the query is received from a user computing device.
8. The method of claim 1 , wherein one or more external data sources comprise one or more computing devices other than the one or more worker nodes, wherein the one or more worker nodes receive the second partial search results from the one or more computing devices using a parallel export technique.
9. The method of claim 1 , wherein the search process is a logical directed acyclic graph (DAG).
10. The method of claim 1 , wherein the one or more worker nodes are configured to process the second partial search results into time-ordered event chunks and stream the time-ordered event chunks to the search service provider.
11. The method of claim 2 , wherein the one or more external data sources store data in a structured format that is different from an events format of the one or more internal data stores.
12. The method of claim 1 , wherein the one or more internal data stores store data as a plurality of time-indexed events, each event of the plurality of time-indexed events including a portion of raw machine data associated with a timestamp.
13. The method of claim 1 , wherein the one or more indexers send the first partial search results to the one or more worker nodes.
14. The method of claim 1 , wherein the one or more indexers send the first partial search results to the one or more worker nodes, wherein the one or more worker nodes are configured to aggregate the first partial search results and the second partial search results.
15. The method of claim 1 , further comprising rendering an output of data indicative of at least one of the first partial search results or the second partial search results for display on a user interface of a display device associated with a user.
16. The method of claim 1 , wherein the one or more worker nodes are co-located with the one or more indexers.
17. The method of claim 1 , wherein the one or more indexers launch the one or more worker nodes.
18. The method of claim 1 , wherein the query is expressed in a pipelined search language.
19. A data intake and query system comprising:
a memory; and
a processing device coupled with the memory to:
define a search scheme based at least in part on a query, wherein the data intake and query system comprises one or more worker nodes, a search service provider, and one or more indexers;
trigger the one or more indexers to search one or more internal data stores based at least in part on a first portion of the search scheme to obtain first partial search results; and
provide a second portion of the search scheme to the search service provider, wherein the search service provider:
defines, based at least in part on the second portion of the search scheme, a search process executable by the one or more worker nodes, and
directs the one or more worker nodes to communicate with one or more external data sources to obtain second partial search results based at least in part on the second portion of the search scheme.
20. A non-transitory computer-readable medium encoding instructions thereon that, in response to execution by one or more processing devices of a data intake and query system, cause the one or more processing devices to:
define a search scheme based at least in part on a query, wherein the data intake and query system comprises one or more worker nodes, a search service provider, and one or more indexers;
trigger the one or more indexers to search one or more internal data stores based at least in part on a first portion of the search scheme to obtain first partial search results; and
provide a second portion of the search scheme to the search service provider, wherein the search service provider:
defines, based at least in part on the second portion of the search scheme, a search process executable by the one or more worker nodes, and
directs the one or more worker nodes to communicate with one or more external data sources to obtain second partial search results based at least in part on the second portion of the search scheme.