Generating row-based and column-based chunks
In some examples, a set of events is received. A row-based chunk includes the set of events and metadata about the set of events is generated, and a column-based chunk that includes metadata about the set of events and, for each event in the set of events, a value of a first field of the multiple fields. The metadata about the set of events includes at least one of a minimum value or a maximum value of the first field over the events in the set of events.
1. A computer-implemented method of processing events, wherein an event includes multiple fields, the method comprising:
receiving a set of events;
generating a row-based chunk that includes the set of events and metadata about the set of events, wherein the row-based chunk is associated with a first retention policy;
generating a column-based chunk that includes metadata about the set of events and, for each event in the set of events, a value of a first field of the multiple fields, wherein the column-based chunk is associated with a second retention policy different from the first retention policy, and wherein the metadata about the set of events in the column-based chunk includes at least one of a minimum value or a maximum value of values of the first field over the events in the set of events;
reclaiming storage used by a file containing the row-based chunk according to the first retention policy; and
reclaiming storage used by a file containing the column-based chunk according to the second retention policy.
2. The method of claim 1 , wherein the set of events is organized in a data structure including rows and columns, the rows corresponding to the events, and the columns corresponding to the fields, wherein the generating of the row-based chunk comprises including a row of the data structure in the row-based chunk, and wherein the generating of the column-based chunk comprises including a column of the data structure in the column-based chunk.
3. The method of claim 2 , wherein the column-based chunk includes just one column of the columns of the data structure.
4. The method of claim 2 , further comprising generating a second column-based chunk that includes metadata about the set of events and another column of the data structure.
5. The method of claim 1 , wherein the metadata in the column-based chunk comprises both the minimum value and the maximum value of the values of the first field over the events in the set of events.
6. The method of claim 1 , wherein the metadata in the column-based chunk comprises a compressed version of content in the set of events.
7. The method of claim 1 , further comprising:
receiving a query containing a search term; and
identifying at least one of the row-based chunk and the column-based chunk as being responsive to the search term.
8. A non-transitory machine-readable storage medium storing instructions that upon execution cause a system to:
receive a first set of events, each event of the first set of events including multiple fields;
generate a row-based chunk that includes the first set of events and metadata about the first set of events, wherein the metadata about the first set of events includes at least one of a minimum value or maximum value of values of a first field over the events in the first set of events, and wherein the row-based chunk is associated with a first retention policy;
receive a second set of events, each event of the second set of events including the multiple fields;
generate a first column-based chunk that includes metadata about the second set of events and, for each event in the second set of events, a value of a second field of the multiple fields, wherein the metadata about the second set of events includes at least one of a minimum value or maximum value of values of the second field over the events in the second set of events, wherein the column-based chunk is associated with a second retention policy different from the first retention policy;
reclaim storage used by a file containing the row-based chunk according to the first retention policy; and
reclaim storage used by a file containing the first column-based chunk according to the second retention policy.
9. The non-transitory machine-readable storage medium of claim 8 , wherein the instructions upon execution cause the system to further generate a second column-based chunk that includes metadata about the second set of events and, for each event in the second set of events, a value of a third field of the multiple fields.
10. The non-transitory machine-readable storage medium of claim 9 , wherein the first column-based chunk includes values of the second field but not values of other fields of the multiple fields, and wherein the second column-based chunk includes values of the third field but not values of other fields of the multiple fields.
11. The non-transitory machine-readable storage medium of claim 8 , wherein the metadata in the row-based chunk comprises a compressed version of content in the first set of events, and the metadata in the first column-based chunk comprises a compressed version of content in the second set of events.
12. The non-transitory machine-readable storage medium of claim 8 , wherein the instructions upon execution cause the system to further:
receive a query containing a search term; and
identify at least one of the row-based chunk and the first column-based chunk as being responsive to the search term.
13. A system comprising:
at least one processor; and
a non-transitory storage medium storing instructions executable on the at least one processor to:
receive a query comprising a search term;
access a first column-based chunk that includes, for each event in a set of events, a value of a first field of multiple fields;
identify a value in the first column-based chunk that matches the search term;
identify an index location identifier associated with the identified value;
access a second column-based chunk that includes, for each event in the set of events, a table location identifier, wherein the first column-based chunk includes values of the first field but not values of other fields of the multiple fields, and wherein the second column-based chunk includes values of a second field but not values of other fields of the multiple fields;
identify the table location identifier in the second column-based chunk that is associated with the identified index location identifier;
access a row-based chunk that includes each event in the set of events; and
identify an event in the row-based chunk that is associated with the identified table location identifier.
14. The system of claim 13 , wherein the set of events are arranged in a data structure comprising rows and columns, wherein the row-based chunk includes the rows of the data structure, the first column-based chunk includes a first column of the data structure, and the second column-based chunk includes a second column of the data structure.
15. The system of claim 13 , wherein the row-based chunk is associated with a first retention policy, and the first column-based chunk is associated with a second, different retention policy, and the instructions are executable on the at least one processor to:
reclaim storage used by a file containing the row-based chunk according to the first retention policy; and
reclaim storage used by a file containing the first column-based chunk according to the second retention policy.