Filesystem destinations

1. A computer-implemented method, comprising:

obtaining a plurality of events for storage on one or more of a plurality of storage systems;

for each event of the plurality of events:

extracting at least one field value from the event,

comparing the at least one field value to a plurality of configurations of the plurality of storage systems to identify at least one storage system of the plurality of storage systems having a matching configuration,

transmitting the event to an ingest module queue for the at least one storage system,

selecting a partition for the event based on the at least one field value to obtain a selected partition,

mapping the selected partition to a file using a partition mapping, and

appending the event to the file on the at least one storage system;

for an event of the plurality of events, after appending the event, detecting that a file size of the file is greater than a threshold;

closing the file responsive to the detecting;

creating, on the at least one storage system, a new file comprising a new filename, wherein the new filename includes a same filename as the file except for an incremented sequence number;

obtaining a file descriptor for the new file; and

storing, in a partition map, the partition mapping to map the selected partition to the file descriptor.

2. The method of claim 1 , wherein the plurality of events is transmitted to heterogeneous subsets of the plurality of storage systems.

3. The method of claim 1 , further comprising:

adding a subset of the plurality of events to a vector;

detecting that the subset of the plurality of events satisfies a size threshold; and

process, based on the subset satisfying the size threshold, the subset to obtain a processed subset,

wherein the processed subset is appended to the file.

4. The method of claim 1 , further comprising:

creating a folder structure on the at least one storage system to match a plurality of partitions, the plurality of partitions comprising the partition, the folder structure grouping files temporally,

wherein a lowest level folder comprises a plurality of files defined based on a fixed time interval, wherein the plurality of files comprises the file.

5. The method of claim 1 , further comprising:

identifying a new time period corresponding to timestamps of the plurality of events;

partitioning the new time period into a plurality of partitions;

for each partition of at least a subset of the plurality of partitions:

creating a second new file on the at least one storage system, and

obtaining a second file descriptor for the second new file; and

storing, in the partition map, the partition mapping to map the partition to the second file descriptor.

6. The method of claim 1 , further comprising:

creating a second new file on the at least one storage system for a new partition,

obtaining a second file descriptor for the second new file;

storing, in the partition map, the partition mapping to map the new partition to the second file descriptor;

generating a link to the file based on a path to the second new file; and

outputting the link.

7. The method of claim 1 , further comprising:

processing the event based on a rule for the at least one storage system, wherein the rule specifies at least one selected from a group consisting of a threshold file size, a timeout, and a compression amount.

8. The method of claim 1 , further comprising:

processing the event based on a rule for the at least one storage system, wherein the rule specifies at least one selected from a group consisting of a partitioning scheme, a partitioning interval, a threshold file size, a timeout, and a compression amount,

wherein the plurality of storage systems has heterogeneous rules amongst the plurality of storage systems.

9. A computing device, comprising:

a processor; and

a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations including:

obtaining a plurality of events for storage on one or more of a plurality of storage systems,

for each event of the plurality of events:

extracting at least one field value from the event,

comparing the at least one field value to a plurality of configurations of the plurality of storage systems to identify at least one storage system of the plurality of storage systems having a matching configuration,

transmitting the event to an ingest module queue for the at least one storage system,

selecting a partition for the event based on the at least one field value to obtain a selected partition,

mapping the selected partition to a file using a partition mapping, and

appending the event to the file on the at least one storage system

for an event of the plurality of events, after appending the event, detecting that a file size of the file is greater than a threshold,

closing the file responsive to the detecting,

creating, on the at least one storage system, a new file comprising a new filename, wherein the new filename includes a same filename as the file except for an incremented sequence number,

obtaining a file descriptor for the new file, and

storing, in a partition map, the partition mapping to map the selected partition to the file descriptor.

10. The computing device of claim 9 , wherein the plurality of events is transmitted to heterogeneous subsets of the plurality of storage systems.

11. The computing device of claim 9 , wherein the operations further comprise:

adding a subset of the plurality of events to a vector;

detecting that the subset of the plurality of events satisfies a size threshold; and

process, based on the subset satisfying the size threshold, the subset to obtain a processed subset,

wherein the processed subset is appended to the file.

12. The computing device of claim 9 , wherein the operations further comprise:

creating a folder structure on the at least one storage system to match a plurality of partitions, the plurality of partitions comprising the partition, the folder structure grouping files temporally,

wherein a lowest level folder comprises a plurality of files defined based on a fixed time interval, wherein the plurality of files comprises the file.

13. The computing device of claim 9 , wherein the operations further comprise:

identifying a new time period corresponding to timestamps of the plurality of events;

partitioning the new time period into a plurality of partitions;

for each partition of at least a subset of the plurality of partitions:

creating a second new file on the at least one storage system, and

obtaining a second file descriptor for the second new file; and

storing, in the partition map, the partition mapping to map the partition to the second file descriptor.

14. The computing device of claim 9 , wherein the operations further comprise:

creating a second new file on the at least one storage system for a new partition,

obtaining a second file descriptor for the second new file;

storing, in the partition map, the partition mapping to map the new partition to the second file descriptor;

generating a link to the file based on a path to the second new file; and

outputting the link.

15. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform operations including:

obtaining a plurality of events for storage on one or more of a plurality of storage systems;

for each event of the plurality of events:

extracting at least one field value from the event,

comparing the at least one field value to a plurality of configurations of the plurality of storage systems to identify at least one storage system of the plurality of storage systems having a matching configuration,

transmitting the event to an ingest module queue for the at least one storage system,

selecting a partition for the event based on the at least one field value to obtain a selected partition,

mapping the selected partition to a file using a partition mapping, and

appending the event to the file on the at least one storage system;

for an event of the plurality of events, after appending the event, detecting that a file size of the file is greater than a threshold;

closing the file responsive to the detecting;

creating, on the at least one storage system, a new file comprising a new filename, wherein the new filename includes a same filename as the file except for an incremented sequence number;

obtaining a file descriptor for the new file; and

storing, in a partition map, the partition mapping to map the selected partition to the file descriptor.

16. The non-transitory computer-readable medium of claim 15 , further comprising:

adding a subset of the plurality of events to a vector;

detecting that the subset of the plurality of events satisfies a size threshold; and

process, based on the subset satisfying the size threshold, the subset to obtain a processed subset,

wherein the processed subset is appended to the file.