Generating and storing aggregate data slices in a remote shared storage system
A data intake and query system can ingest and index large amounts of data using one or more ingestors and indexers. The ingestors can ingest incoming data, use it to generate events, and communicate the generated events to a message bus. Indexers can monitor their capacity to process additional groups of events. Based on a determination that an indexer has capacity to process one or more groups of events, the indexer can request one or more messages from the message bus, and process the events associated with the one or more messages.
1. A method, comprising:
receiving, at an indexing node of a data intake and query system, a message payload from a remote message bus, wherein the message payload includes a first plurality of events, wherein each event of the first plurality of events includes raw machine data associated with a timestamp;
extracting the first plurality of events from the message payload;
adding a first set of events of the first plurality of events to a first editable data slice, wherein the first editable data slice is associated with a hot bucket, wherein the first set of events includes at least two events;
based on a slice rollover policy, converting the first editable data slice to a first non-editable data slice and adding the first non-editable data slice to a first aggregate data slice in volatile memory,
wherein the first aggregate data slice comprises a plurality of non-editable data slices associated with the hot bucket and each of the plurality of non-editable data slices comprises a second plurality of events, wherein the plurality of non-editable data slices includes the first non-editable data slice, wherein the first non-editable data slice includes the first set of events;
concurrent to at least one of adding the first set of events to the first editable data slice or adding the first non-editable data slice to the first aggregate data slice in volatile memory, modifying one or more files of the hot bucket using the first set of events, wherein the hot bucket includes:
at least one index file generated using the events of a plurality of aggregate data slices wherein the one or more files includes the at least one index file, and
the plurality of aggregate data slices, wherein the plurality of aggregate data slices comprises the first aggregate data slice;
based on an aggregate slice backup policy, determining whether a warm bucket corresponding to the hot bucket associated with the first aggregate data slice has been stored on a remote shared storage system, and based on a determination that a warm bucket corresponding to the hot bucket has not been stored on the remote shared storage system, communicating a first copy of the first aggregate data slice to the remote shared storage system for storage, wherein the first copy of the first aggregate data slice is communicated independent of the hot bucket; and
based on a bucket rollover policy, converting the hot bucket to a first warm bucket and communicating a copy of the first warm bucket to the remote shared storage system for storage, wherein the copy of the first warm bucket includes a second copy of the first aggregate data slice.
2. The method of claim 1 , wherein extracting the first plurality of events comprises decoding the first plurality of events.
3. The method of claim 1 , wherein the first set of events of the first plurality of events are added to the first editable data slice based on an association between the first plurality of events with a particular index, wherein the particular index is associated with the first editable data slice and the hot bucket.
4. The method of claim 1 , wherein the adding the first non-editable data slice to the first aggregate data slice includes compressing the first non-editable data slice, wherein the first non-editable data slice is added to the first aggregate data slice as a compressed first non-editable data slice.
5. The method of claim 1 , wherein the slice rollover policy includes a slice size threshold, and wherein the first non-editable data slice is added to the first aggregate data slice based on a determination that a size of the first non-editable data slice satisfies the slice size threshold.
6. The method of claim 1 , wherein the slice rollover policy includes a slice time threshold, and wherein the first non-editable data slice is added to the first aggregate data slice based on a determination that an amount of time since the first non-editable data slice was generated satisfies the slice time threshold.
7. The method of claim 1 , wherein the aggregate slice backup policy includes a slice size threshold, and wherein the first copy of the first aggregate data slice is stored to the remote shared storage system based on a determination that a size of the first aggregate data slice satisfies the slice size threshold.
8. The method of claim 1 , wherein the aggregate slice backup policy includes a slice time threshold, and wherein the first copy of the first aggregate data slice is stored to the remote shared storage system based on a determination that an amount of time since the first aggregate data slice was generated satisfies the slice time threshold.
9. The method of claim 1 , further comprising based on the bucket rollover policy, generating a second hot bucket.
10. The method of claim 1 , wherein the bucket rollover policy includes a bucket size threshold, wherein converting the hot bucket to a first warm bucket and communicating a copy of the first warm bucket to the remote shared storage system for storage is further based on a determination that a size of the hot bucket satisfies the bucket size threshold.
11. The method of claim 1 , wherein the bucket rollover policy includes a bucket timing threshold, wherein converting the hot bucket to a first warm bucket and communicating a copy of the first warm bucket to the remote shared storage system for storage is further based on a determination that an amount of time since the hot bucket was generated satisfies the bucket timing threshold.
12. The method of claim 1 , further comprising:
based on a determination that the copy of the first warm bucket is stored on the remote shared storage system, removing the first copy of the first aggregate data slice from the remote shared storage system.
13. The method of claim 1 , further comprising:
identifying for each event of the first plurality of events, a particular bucket that includes the event; and
based on a determination that a copy of each bucket that includes at least one event of the first plurality of events was stored to the remote shared storage system, communicating an acknowledgement to the remote message bus, wherein the remote message bus deletes the message payload in response to the acknowledgement.
14. The method of claim 1 , wherein the communicating the copy of the first aggregate data slice comprises flagging the first aggregate data slice for storage to the remote shared storage system.
15. The method of claim 1 , further comprising based at least in part on communicating the copy of the first aggregate data slice, generating an additional aggregate data slice, wherein the additional aggregate data slice is associated with the hot bucket.
16. A computing device, comprising:
memory; and
one or more processing devices communicatively coupled to the memory and configured to:
receive a message payload from a remote message bus, wherein the message payload includes a first plurality of events, wherein each event of the first plurality of events includes raw machine data associated with a timestamp;
extract the first plurality of events from the message payload;
add a first set of events of the first plurality of events to a first editable data slice, wherein the first editable data slice is associated with a hot bucket, wherein the first set of events includes at least two events;
based on a slice rollover policy, convert the first editable data slice to a first non-editable data slice and add the first non-editable data slice to a first aggregate data slice in volatile memory,
wherein the first aggregate data slice comprises a plurality of non-editable data slices associated with the hot bucket and each of the plurality of non-editable data slices comprises a second plurality of events, wherein the plurality of non-editable data slices includes the first non-editable data slice, wherein the first non-editable data slice includes the first set of events;
concurrent to at least one of addition of the first set of events to the first editable data slice or addition of the first non-editable data slice to the first aggregate data slice in volatile memory, modify one or more files of the hot bucket using the first set of events, wherein the hot bucket includes:
at least one index file generated using the events of a plurality of aggregate data slices, wherein the one or more files includes the at least one index file, and
the plurality of aggregate data slices, wherein the plurality of aggregate data slices comprises the first aggregate data slice;
based on an aggregate slice backup policy, determining whether a warm bucket corresponding to the hot bucket associated with the first aggregate data slice has been stored on a remote shared storage system, and based on a determination that a warm bucket corresponding to the hot bucket has not been stored on the remote shared storage system, communicating a first copy of the first aggregate data slice to the remote shared storage system for storage, wherein the first copy of the first aggregate data slice is communicated independent of the hot bucket; and
based on a bucket rollover policy, converting the hot bucket to a first warm bucket and communicating a copy of the first warm bucket to the remote shared storage system for storage, wherein the copy of the first warm bucket includes a second copy of the first aggregate data slice.
17. Non-transitory computer-readable media comprising computer-executable instructions that when executed by one or more processing devices, causes the one or more processing devices to:
receive a message payload from a remote message bus, wherein the message payload includes a first plurality of events, wherein each event of the first plurality of events includes raw machine data associated with a timestamp;
extract the first plurality of events from the message payload;
add a first set of events of the first plurality of events to a first editable data slice, wherein the first editable data slice is associated with a hot bucket, wherein the first set of events includes at least two events;
based on a slice rollover policy, convert the first editable data slice to a first non-editable data slice and add the first non-editable data slice to a first aggregate data slice in volatile memory,
wherein the first aggregate data slice comprises a plurality of non-editable data slices associated with the hot bucket and each of the plurality of non-editable data slices comprises a second plurality of events, wherein the plurality of non-editable data slices includes the first non-editable data slice, wherein the first non-editable data slice includes the first set of events;
concurrent to at least one of addition of the first set of events to the first editable data slice or addition of the first non-editable data slice to the first aggregate data slice in volatile memory, modify one or more files of the hot bucket using the first set of events, wherein the hot bucket includes:
at least one index file generated using the events of a plurality of aggregate data slices, wherein the one or more files includes the at least one index file; and
the plurality of aggregate data slices, wherein the plurality of aggregate data slices comprises the first aggregate data slice;
based on an aggregate slice backup policy, determining whether a warm bucket corresponding to the hot bucket associated with the first aggregate data slice has been stored on a remote shared storage system, and based on a determination that a warm bucket corresponding to the hot bucket has not been stored on the remote shared storage system, communicating a first copy of the first aggregate data slice to the remote shared storage system for storage, wherein the first copy of the first aggregate data slice is communicated independent of the hot bucket; and
based on a bucket rollover policy, converting the hot bucket to a first warm bucket and communicating a copy of the first warm bucket to the remote shared storage system for storage, wherein the copy of the first warm bucket includes a second copy of the first aggregate data slice.