Indexing data at a data intake and query system based on a node capacity threshold
As an indexer indexes and groups events, it can generate data slices that include events. Based on a slice rollover policy, the indexer can add a particular slice to an aggregate slice. Based on an aggregate slice backup policy, the indexer can store a copy of the aggregate slice to a shared storage system. The aggregate slice can be used for restore purposes in the event the indexer fails or becomes unresponsive.
1. A method, comprising:
based on a determination by computing device in communication with an indexing node of a distributed data processing system that the indexing node satisfies a node capacity threshold, receiving, at the indexing node, an instruction to request a message payload from a remote message bus;
based on receiving the instruction to request the message payload from the remote message bus requesting, by the indexing node, the message payload from the remote message bus;
receiving the message payload from the remote message bus;
extracting a plurality of events using the message payload, wherein each of the plurality of events comprises machine data generated by one or more computing devices in an information technology environment;
adding the plurality of events to a data bucket; and
storing at least one copy of the data bucket to a remote shared storage system.
2. The method of claim 1 , wherein the node capacity threshold comprises a CPU utilization threshold.
3. The method of claim 1 , wherein the node capacity threshold comprises a memory threshold.
4. The method of claim 1 , wherein the determination that the indexing node satisfies a node capacity threshold comprises a determination that the indexing node has sufficient resources to process at least one more message payload.
5. The method of claim 1 , wherein the message payload is a first message payload, the method further comprising: concurrently requesting a second message payload with the first message payload based on the instruction to request the first message payload from the remote message bus.
6. The method of claim 1 , wherein the plurality of events are generated by one or more ingestion nodes, the method further comprising, receiving, at the indexing node, an instruction to complete processing events that the indexing node is processing and shut down, wherein the indexing node shuts down independent of the one or more of ingestion nodes.
7. The method of claim 1 , wherein extracting the plurality of events using the message payload comprises extracting the plurality of events from the message payload.
8. The method of claim 1 , wherein extracting the plurality of events using the message payload comprises:
extracting, from the message payload, a location reference to the plurality of events, the location reference indicating a location of the plurality of events in a data store; and
retrieving, using the location reference, the plurality of events from the data store.
9. An indexing node of a data processing system, the indexing node comprising:
memory; and
one or more processing devices communicatively coupled to the memory and configured to:
based on a determination by a computing device in communication with the indexing node that the indexing node satisfies a node capacity threshold, receive an instruction to request a message payload from a remote message bus;
based on receipt of the instruction to request the message payload from the remote message bus request the message payload from the remote message bus;
receive the message payload from the remote message bus;
extract a plurality of events using the message payload, wherein each of the plurality of events comprises machine data generated by one or more computing devices in an information technology environment;
add the plurality of events to a data bucket; and
store at least one copy of the data bucket to a remote shared storage system.
10. The indexing node of claim 9 , wherein the node capacity threshold comprises a CPU utilization threshold.
11. The indexing node of claim 9 , wherein the node capacity threshold comprises a memory threshold.
12. The indexing node of claim 9 , wherein the determination that the indexing node satisfies a node capacity threshold comprises a determination that the indexing node has sufficient resources to process at least one more message payload.
13. The indexing node of claim 9 , wherein the message payload is a first message payload, wherein the one or more processing devices are further configured to concurrently request a second message payload with the first message payload based on the instruction to request the first message payload from the remote message bus.
14. The indexing node of claim 9 , wherein the plurality of events are generated by one or more ingestion nodes, wherein the one or more processing devices are further configured to receive an instruction to complete processing events that the indexing node is processing and shut down, wherein the indexing node shuts down independent of the one or more of ingestion nodes.
15. The indexing node of claim 9 , wherein to extract the plurality of events using the message payload, the one or more processing devices are configured to extract the plurality of events from the message payload.
16. The indexing node of claim 9 , wherein to extract the plurality of events using the message payload, the one or more processing devices are configured to:
extract, from the message payload, a location reference to the plurality of events, the location reference indicating a location of the plurality of events in a data store; and
retrieve, using the location reference, the plurality of events from the data store.
17. Non-transitory computer-readable media comprising computer-executable instructions that when executed by one or more processing devices of an indexing node of a data processing system, cause the one or more processing devices to:
based on a determination by a computing device in communication with the indexing node that the indexing node satisfies a node capacity threshold, receive an instruction to request a message payload from a remote message bus;
based on receipt of the instruction to request the message payload from the remote message bus request the message payload from the remote message bus;
receive the message payload from the remote message bus;
extract a plurality of events using the message payload, wherein each of the plurality of events comprises machine data generated by one or more computing devices in an information technology environment;
add the plurality of events to a data bucket; and
store at least one copy of the data bucket to a remote shared storage system.
18. The non-transitory computer-readable media of claim 17 , wherein the determination that the indexing node satisfies a node capacity threshold comprises a determination that the indexing node has sufficient resources to process at least one more message payload.
19. The non-transitory computer-readable media of claim 17 , wherein the message payload is a first message payload, wherein the computer-executable instructions further cause the one or more processing devices to concurrently request a second message payload with the first message payload based on the instruction to request the first message payload from the remote message bus.
20. The non-transitory computer-readable media of claim 17 , wherein to extract the plurality of events using the message payload, the computer-executable instructions cause the one or more processing devices to:
extract, from the message payload, a location reference to the plurality of events, the location reference indicating a location of the plurality of events in a data store; and
retrieve, using the location reference, the plurality of events from the data store.