Search functionality of worker nodes in a data fabric service system
Disclosed is a technique that can be performed in a distributed computer network. The technique can include a worker node that receives search instructions defined by a search service based on at least a portion of a search scheme defined by a data intake and query system, to cause the worker node to obtain search results from distributed data storage systems communicatively coupled to the worker node over a network. The distributed data storage systems include an external data storage system and/or an internal data storage system of the data intake and query system. The worker node obtains the search results by searching the distributed data storage systems in accordance with the search instructions, and communicating, over the network to the search service, a combination of search results based on the search results to cause an output by the data intake and query system in accordance with the search scheme.
1. A method comprising:
receiving, by a worker node, search instructions defined by a search service, wherein the search instructions are based on at least a portion of a search scheme defined by a data intake and query system, wherein the search instructions instruct the worker node to obtain a plurality of search results from a plurality of distributed data storage systems communicatively coupled to the worker node over a network, wherein the plurality of distributed data storage systems include an external data storage system that is external to the data intake and query system and an internal data storage system of the data intake and query system;
obtaining, by the worker node, a first portion of search results of the plurality of search results from the external data storage system in accordance with the search instructions;
obtaining, by the worker node, a second portion of search results of the plurality of search results from the internal data storage system in accordance with the search instructions; and
communicating, by the worker node over the network to the search service, a combination of search results based on the first portion of search results and the second portion of search results to cause an output by the data intake and query system in accordance with the search scheme.
2. The method of claim 1 , wherein the search instructions are based on only a portion of the search scheme defined by the data intake and query system.
3. The method of claim 1 , wherein at least some of the second portion of search results are time-ordered events retrieved from the internal data storage system, each time-ordered event including a segment of raw machine data and an associated timestamp.
4. The method of claim 1 further comprising, prior to communicating the combination of search results to the search service:
producing, by the worker node, the combination of search results based on the first portion of search results and the second portion of search results, wherein the combination of search results includes a plurality of time-ordered events.
5. The method of claim 1 further comprising, prior to communicating the combination of search results to the search service:
producing, by the worker node, the combination of search results by aggregating the first portion of search results and the second portion of search results.
6. The method of claim 1 , wherein the combination of search results includes structured data retrieved from the external data storage system.
7. The method of claim 1 , wherein the combination of search results includes unstructured data retrieved from the internal data storage system.
8. The method of claim 1 , wherein the combination of search results is in a format specified by the search scheme.
9. The method of claim 1 further comprising, prior to communicating the combination of search results to the search service:
producing, by the worker node, the combination of search results based on structured data retrieved from the external data storage system and unstructured data retrieved from the internal data storage system.
10. The method of claim 1 , wherein the combination of search results includes data retrieved in parallel from peer indexers of the internal data storage system.
11. The method of claim 1 , wherein a device distinct from, and communicatively coupled to, the data intake and query system implements the search service over the network.
12. The method of claim 1 , wherein the data intake and query system implements the search service.
13. The method of claim 1 further comprising:
causing the combination of search results or data indicative of the combination of search results to be displayed on a display device.
14. The method of claim 1 , further comprising:
causing the output to be rendered on a display device in a timeline visualization.
15. The method of claim 1 , wherein the search instructions are based on user input expressed in a pipelined language.
16. The method of claim 1 , wherein the search instructions are based on user input expressed in a structured language.
17. The method of claim 1 further comprising:
producing, by the worker node, a plurality of aggregate search results from the first portion of search results and the second portion of search results.
18. The method of claim 1 , wherein at least some of the second portion of search results are formatted as a plurality of time-indexed events, each time-indexed event including a segment of raw machine data and an associated timestamp.
19. The method of claim 1 , wherein the plurality of search results includes the first portion of search results in a first format as time-indexed events including raw data and includes the second portion of search results in a second format different from the first format, and the combination of search results is in a format specified by the search scheme.
20. The method of claim 1 further comprising, prior to communicating the combination of search results to the search service: combining, by the worker node, the first portion of search results with the second portion of search results.
21. The method of claim 1 further comprising:
prior to communicating the combination of search results to the search service, combining, by the worker node, the first portion of search results with the second portion of search results in an arrangement of time-ordered events.
22. The method of claim 1 further comprising:
prior to communicating the combination of search results to the search service, deriving, by the worker node, the combination of search results by performing an operation on the first portion of search results and the second plurality of search results.
23. The method of claim 1 further comprising:
obtaining, by the worker node, a third portion of search results of the plurality of search results, the third portion of search results extracted from another external data storage system by another worker node, wherein the combination of search results is based on the first portion of search results, the second portion of search results, and the third portion of search results.
24. The method of claim 1 , wherein the worker node is a first worker node, the method further comprising:
obtaining, by the first worker node, a third portion of search results of the plurality of search results, the third portion of search results extracted from a second external data storage system by a second worker node, the second worker node being communicatively coupled over the network to the second external data storage system, wherein the combination of search results is based on the first portion of search results, the second portion of search results, and the third portion of search results.
25. The method of claim 1 , wherein the worker node is a first worker node, the method further comprising:
collecting, by the first worker node, a third portion of search results of the plurality of search results, the third portion of search results extracted from a second external data storage system by a second worker node, the second worker node being communicatively coupled over the network to the second external data storage system; and
sending, by the first worker node over the network, at least a portion of the first portion of search results to the second worker node, wherein the combination of search results is based on the first portion of search results, the second portion of search results, and the third portion of search results.
26. A worker node of data fabric system, the worker node comprising:
a processor; and
a memory having instructions executable by the processor to cause the worker node to:
receive search instructions defined by a search service, wherein the search instructions are based on at least a portion of a search scheme defined by a data intake and query system, wherein the search instructions instruct the worker node to obtain a plurality of search results from a plurality of distributed data storage systems communicatively coupled to the worker node over a network, wherein the plurality of distributed data storage systems include an external data storage system that is external to the data intake and query system and an internal data storage system of the data intake and query system;
obtain a first portion of search results of the plurality of search results from the external data storage system in accordance with the search instructions;
obtain a second portion of search results of the plurality of search results from the internal data storage system in accordance with the search instructions; and
communicate, over the network to the search service, a combination of search results based on the first portion of search results and the second portion of search results to cause an output by the data intake and query system in accordance with the search scheme.
27. A non-transitory storage medium storing program instructions, execution of which in a worker node of a data fabric system causes actions comprising:
receiving search instructions defined by a search service, wherein the search instructions are based on at least a portion of a search scheme defined by a data intake and query system, wherein the search instructions instruct the worker node to obtain a plurality of search results from a plurality of distributed data storage systems communicatively coupled to the worker node over a network, wherein the plurality of distributed data storage systems include an external data storage system that is external to the data intake and query system and an internal data storage system of the data intake and query system;
obtaining a first portion of search results of the plurality of search results from the external data storage system in accordance with the search instructions;
obtaining, by the worker node, a second portion of search results of the plurality of search results from the internal data storage system in accordance with the search instructions; and
communicating, over the network to the search service, a combination of search results based on the first portion of search results and the second portion of search results to cause an output by the data intake and query system in accordance with the search scheme.