Indexing data at a data intake and query system based on a node capacity threshold
As an indexer indexes and groups events, it can generate data slices that include events. Based on a slice rollover policy, the indexer can add a particular slice to an aggregate slice. Based on an aggregate slice backup policy, the indexer can store a copy of the aggregate slice to a shared storage system. The aggregate slice can be used for restore purposes in the event the indexer fails or becomes unresponsive.
1. A method, comprising:
receiving, at an indexing node of a data intake and query system, a message payload including a first plurality of events;
extracting the first plurality of events from the message payload;
adding two or more events of the first plurality of events to a first editable data slice, wherein the first editable data slice is associated with a hot bucket, the hot bucket including at least one index file, and a plurality of aggregate data slices;
based on a slice rollover policy, converting the first editable data slice to a first non-editable data slice and adding the first non-editable data slice to a first aggregate data slice of the plurality of aggregate data slices, wherein the first aggregate data slice comprises a plurality of non-editable data slices, each of the plurality of non-editable data slices comprising a second plurality of events;
concurrent to at least one of adding the two or more events to the first editable data slice or adding the first non-editable data slice to the first aggregate data slice, modifying one or more files of the hot bucket using the two or more events;
based on an aggregate slice backup policy and a determination that a warm bucket corresponding to the hot bucket has not been stored on a remote shared storage system, communicating a copy of the first aggregate data slice to the remote shared storage system for storage, wherein the copy of the first aggregate data slice is communicated independent of the hot bucket; and
based on a bucket rollover policy, converting the hot bucket to a first warm bucket and communicating a copy of the first warm bucket to the remote shared storage system for storage, wherein the copy of the first warm bucket includes a second copy of the first aggregate data slice.
2. The method of claim 1 , wherein each event of the first plurality of events includes raw machine data associated with a timestamp.
3. The method of claim 1 , wherein extracting the first plurality of events comprises decoding the first plurality of events.
4. The method of claim 1 , wherein the two or more events of the first plurality of events are added to the first editable data slice based on an association between the first plurality of events with a particular index, wherein the particular index is associated with the first editable data slice and the hot bucket.
5. The method of claim 1 , wherein adding the first non-editable data slice to the first aggregate data slice includes compressing the first non-editable data slice, and wherein the first non-editable data slice is added to the first aggregate data slice as a compressed first non-editable data slice.
6. The method of claim 1 , wherein the slice rollover policy includes a slice size threshold, and wherein the first non-editable data slice is added to the first aggregate data slice based on a determination that a size of the first non-editable data slice satisfies the slice size threshold.
7. The method of claim 1 , wherein the slice rollover policy includes a slice time threshold, and wherein the first non-editable data slice is added to the first aggregate data slice based on a determination that an amount of time since the first non-editable data slice was generated satisfies the slice time threshold.
8. The method of claim 1 , wherein the aggregate slice backup policy includes a slice size threshold, and wherein the copy of the first aggregate data slice is stored to the remote shared storage system based on a determination that a size of the first aggregate data slice satisfies the slice size threshold.
9. The method of claim 1 , wherein the aggregate slice backup policy includes a slice time threshold, and wherein the copy of the first aggregate data slice is stored to the remote shared storage system based on a determination that an amount of time since the first aggregate data slice was generated satisfies the slice time threshold.
10. The method of claim 1 , further comprising based on the bucket rollover policy, generating a second hot bucket.
11. The method of claim 1 , wherein the bucket rollover policy includes a bucket size threshold, wherein converting the hot bucket to a first warm bucket and communicating a copy of the first warm bucket to the remote shared storage system for storage is further based on a determination that a size of the hot bucket satisfies the bucket size threshold.
12. The method of claim 1 , wherein the bucket rollover policy includes a bucket timing threshold, wherein converting the hot bucket to a first warm bucket and communicating a copy of the first warm bucket to the remote shared storage system for storage is further based on a determination that an amount of time since the hot bucket was generated satisfies the bucket timing threshold.
13. The method of claim 1 , further comprising, based on a determination that the copy of the first warm bucket is stored on the remote shared storage system, removing the copy of the first aggregate data slice from the remote shared storage system.
14. The method of claim 1 , wherein the message payload is received from a remote message bus, and wherein the method further comprises:
identifying for each event of the first plurality of events, a particular bucket that includes the event; and
based on a determination that a copy of each bucket that includes at least one event of the first plurality of events was stored to the remote shared storage system, communicating an acknowledgement to the remote message bus, wherein the remote message bus deletes the message payload in response to the acknowledgement.
15. The method of claim 1 , further comprising, based at least in part on communicating the copy of the first aggregate data slice, generating an additional aggregate data slice, wherein the additional aggregate data slice is associated with the hot bucket.
16. A computing device, comprising:
memory; and
one or more processing devices communicatively coupled to the memory and configured to:
receive, at an indexing node of a data intake and query system, a message payload including a first plurality of events;
extract the first plurality of events from the message payload;
add two or more events of the first plurality of events to a first editable data slice, wherein the first editable data slice is associated with a hot bucket, the hot bucket including at least one index file, and a plurality of aggregate data slices;
based on a slice rollover policy, convert the first editable data slice to a first non-editable data slice and adding the first non-editable data slice to a first aggregate data slice of the plurality of aggregate data slices, wherein the first aggregate data slice comprises a plurality of non-editable data slices, each of the plurality of non-editable data slices comprising a second plurality of events;
concurrent to at least one of addition of the two or more events to the first editable data slice or addition of the first non-editable data slice to the first aggregate data slice, modify one or more files of the hot bucket using the two or more events;
based on an aggregate slice backup policy and a determination that a warm bucket corresponding to the hot bucket has not been stored on a remote shared storage system, communicate a copy of the first aggregate data slice to the remote shared storage system for storage, wherein the copy of the first aggregate data slice is communicated independent of the hot bucket; and
based on a bucket rollover policy, convert the hot bucket to a first warm bucket and communicate a copy of the first warm bucket to the remote shared storage system for storage, wherein the copy of the first warm bucket includes a second copy of the first aggregate data slice.
17. The computing device of claim 16 , wherein each event of the first plurality of events includes raw machine data associated with a timestamp.
18. The computing device of claim 16 , wherein the two or more events of the first plurality of events are added to the first editable data slice based on an association between the first plurality of events with a particular index, wherein the particular index is associated with the first editable data slice and the hot bucket.
19. The computing device of claim 16 , wherein the slice rollover policy includes a slice size threshold, and wherein the first non-editable data slice is added to the first aggregate data slice based on a determination that a size of the first non-editable data slice satisfies the slice size threshold.
20. Non-transitory computer-readable media comprising computer-executable instructions that when executed by one or more processing devices, causes the one or more processing devices to:
receive, at an indexing node of a data intake and query system, a message payload including a first plurality of events;
extract the first plurality of events from the message payload;
add two or more events of the first plurality of events to a first editable data slice, wherein the first editable data slice is associated with a hot bucket, the hot bucket including at least one index file, and a plurality of aggregate data slices;
based on a slice rollover policy, convert the first editable data slice to a first non-editable data slice and adding the first non-editable data slice to a first aggregate data slice of the plurality of aggregate data slices, wherein the first aggregate data slice comprises a plurality of non-editable data slices, each of the plurality of non-editable data slices comprising a second plurality of events;
concurrent to at least one of addition of the two or more events to the first editable data slice or addition of the first non-editable data slice to the first aggregate data slice, modify one or more files of the hot bucket using the two or more events;
based on an aggregate slice backup policy and a determination that a warm bucket corresponding to the hot bucket has not been stored on a remote shared storage system, communicate a copy of the first aggregate data slice to the remote shared storage system for storage, wherein the copy of the first aggregate data slice is communicated independent of the hot bucket; and
based on a bucket rollover policy, convert the hot bucket to a first warm bucket and communicate a copy of the first warm bucket to the remote shared storage system for storage, wherein the copy of the first warm bucket includes a second copy of the first aggregate data slice.