Resegmenting chunks of data based on one or more criteria to facilitate load balancing
Resegmenting chunks of data for load balancing is disclosed. A plurality of first chunks of data is received. The plurality of first chunks of data includes one or more entries that include raw data produced by a component of an information technology environment and that reflects activity in the information technology environment. The plurality of first chunks of data is resegmented into a plurality of second chunks of data based on a source type of the plurality of first chunks. A first subset of the plurality of second chunks of data is distributed to a first indexer of a set of indexers. An occurrence of a trigger event is determined, and in response to the trigger event, a second subset of the plurality of second chunks of data is distributed to a second indexer of the set of indexers.
1. A method comprising:
receiving, at a forwarder device, a plurality of first chunks of data comprising one or more entries that include raw data that reflects activity in an information technology environment, wherein respective chunk boundaries of the plurality of first chunks are independent of entry boundaries of the one or more entries, wherein the entry boundaries occur in at least some of the plurality of first chunks, and wherein a first entry of the one or more entries bridges at least two successive chunks in the plurality of first chunks;
determining, at the forwarder device, based on one or more criteria for an entry boundary, a first entry boundary associated with the first entry included in a first chunk in the plurality of first chunks;
segmenting, at the forwarder device, the first chunk into an initial second chunk and a subsequent second chunk based on the first entry boundary;
marking, at the forwarder device, the initial second chunk as an end of entry for the first entry that bridges the at least two successive chunks;
generating a plurality of second chunks that includes the initial second chunk and the subsequent second chunk;
distributing a first subset of the plurality of second chunks to a first indexer of a set of indexers, wherein at least a portion of an incoming query is processed by the first indexer based on at least a portion of the first subset of the plurality of second chunks; and
switching, based at least in part on detecting that the first subset of the plurality of second chunks includes the initial second chunk that is marked as the end of entry, to distribute a second subset of the plurality of second chunks that begins at the subsequent second chunk to a second indexer of the set of indexers.
2. The method of claim 1 , wherein at least one of the one or more criteria comprises a source type associated with the plurality of first chunks.
3. The method of claim 1 , wherein at least one of the one or more criteria comprises a source of the raw data.
4. The method of claim 1 , wherein at least one of the one or more criteria comprises a textual pattern, and determining the first entry boundary comprises matching the textual pattern with data in the first chunk.
5. The method of claim 1 , further comprising:
determining a source type of the plurality of first chunks; and
accessing an entry boundary rule of a plurality of different entry boundary rules based on the source type, wherein the entry boundary rule includes at least one of the one or more criteria.
6. The method of claim 1 , wherein each second chunk of the plurality of second chunks begins or ends on a different entry boundary.
7. The method of claim 1 , wherein distributing the first subset of the plurality of second chunks to the first indexer comprises inserting successive second chunks included in the plurality of second chunks into an output queue associated with the first indexer.
8. The method of claim 1 , wherein at least some of the one or more entries comprise different lengths.
9. The method of claim 1 , wherein receiving the plurality of first chunks comprises receiving a file comprising the plurality of first chunks, wherein each first chunk comprises a block of data from the file.
10. One or more non-transitory computer readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
receiving, at a forwarder device, a plurality of first chunks of data comprising one or more entries that include raw data that reflects activity in an information technology environment, wherein respective chunk boundaries of the plurality of first chunks are independent of entry boundaries of the one or more entries, wherein the entry boundaries occur in at least some of the plurality of first chunks, and wherein a first entry of the one or more entries bridges at least two successive chunks in the plurality of first chunks;
determining, at the forwarder device, based on one or more criteria for an entry boundary, a first entry boundary associated with the first entry included in a first chunk in the plurality of first chunks;
segmenting, at the forwarder device, the first chunk into an initial second chunk and a subsequent second chunk based on the first entry boundary;
marking, at the forwarder device, the initial second chunk as an end of entry for the first entry that bridges the at least two successive chunks;
generating a plurality of second chunks that includes the initial second chunk and the subsequent second chunk;
distributing a first subset of the plurality of second chunks to a first indexer of a set of indexers, wherein at least a portion of an incoming query is processed by the first indexer based on at least a portion of the first subset of the plurality of second chunks; and
switching, based at least in part on detecting that the first subset of the plurality of second chunks includes the initial second chunk that is marked as the end of entry, to distribute a second subset of the plurality of second chunks that begins at the subsequent second chunk to a second indexer of the set of indexers.
11. The one or more non-transitory computer readable media of claim 10 , wherein at least one of the one or more criteria comprises a source type associated with the plurality of first chunks.
12. The one or more non-transitory computer readable media of claim 10 , wherein at least one of the one or more criteria comprises a source of the raw data.
13. The one or more non-transitory computer readable media of claim 10 , wherein at least one of the one or more criteria comprises a textual pattern, and determining the first entry boundary comprises matching the textual pattern with data in the first chunk.
14. The one or more non-transitory computer readable media of claim 10 , further comprising:
determining a source type of the plurality of first chunks; and
accessing an entry boundary rule of a plurality of different entry boundary rules based on the source type, wherein the entry boundary rule includes at least one of the one or more criteria.
15. The one or more non-transitory computer readable media of claim 10 ,
wherein each second chunk of the plurality of second chunks begins or ends on a different entry boundary.
16. The one or more non-transitory computer readable media of claim 10 , wherein distributing the first subset of the plurality of second chunks to the first indexer comprises inserting successive second chunks included in the plurality of second chunks into an output queue associated with the first indexer.
17. The one or more non-transitory computer readable media of claim 10 , wherein at least some of the one or more entries comprise different lengths.
18. The one or more non-transitory computer readable media of claim 10 , wherein receiving the plurality of first chunks comprises receiving a file comprising the plurality of first chunks, wherein each first chunk comprises a block of data from the file.
19. A computer-system, comprising:
one or more memories storing instructions; and
one or more processors for executing the instructions to:
receive, at a forwarder device, a plurality of first chunks of data comprising one or more entries that include raw data that reflects activity in an information technology environment, wherein respective chunk boundaries of the plurality of first chunks are independent of entry boundaries of the one or more entries, wherein the entry boundaries occur in at least some of the plurality of first chunks, and wherein a first entry of the one or more entries bridges at least two successive chunks in the plurality of first chunks;
determine, at the forwarder device, based on one or more criteria for an entry boundary, a first entry boundary associated with the first entry included in a first chunk in the plurality of first chunks;
segment, at the forwarder device, the first chunk into an initial second chunk and a subsequent second chunk based on the first entry boundary;
mark, at the forwarder device, the initial second chunk as an end of entry for the first entry that bridges the at least two successive chunks;
generate a plurality of second chunks that includes the initial second chunk and the subsequent second chunk;
distribute a first subset of the plurality of second chunks to a first indexer of a set of indexers, wherein at least a portion of an incoming query is processed by the first indexer based on at least a portion of the first subset of the plurality of second chunks; and
switch, based at least in part on detecting that the first subset of the plurality of second chunks includes the initial second chunk that is marked as the end of entry, to distribute a second subset of the plurality of second chunks that begins at the subsequent second chunk to a second indexer of the set of indexers.
20. The system of claim 19 , wherein the one or more processors execute the instructions to:
determine a source type of the plurality of first chunks; and
access an entry boundary rule of a plurality of different entry boundary rules based on the source type, wherein the entry boundary rule includes at least one of the one or more criteria.