Storage system configuration based filesystem destinations

1 . A computer-implemented method, comprising:

obtaining a plurality of events for storage on one or more of a plurality of storage systems; and

for each event of the plurality of events:

extracting at least one field value from the event,

comparing the at least one field value to a plurality of configurations of the plurality of storage systems to identify at least one storage system of the plurality of storage systems having a matching configuration,

transmitting the event to an ingest module queue for the at least one storage system,

selecting a partition for the event based on the at least one field value and a partitioning rule for the at least one storage system to obtain a selected partition,

mapping the selected partition to a file using a partition mapping of a partition map, the partition map including a plurality of mappings between partition data and file pointers that uniquely identify corresponding open files, and

appending the event to the file on the at least one storage system based on a file pointer referencing the file identified via the mapping using the partition mapping.

2 . The computer-implemented method of claim 1 , further comprising:

identifying, from the partition map, the partition mapping,

wherein open files are open based on the file pointers, and wherein the open files comprises the file.

3 . The computer-implemented method of claim 1 , further comprising:

identifying, from the partition map, the partition mapping,

wherein mapping the selected partition to the file comprises obtaining a file descriptor of the file from the partition mapping, wherein the file descriptor keeps the file open on the at least one storage system.

4 . The computer-implemented method of claim 1 , wherein the plurality of events is appended concurrently and asynchronously to the plurality of storage systems.

5 . The computer-implemented method of claim 1 , wherein the plurality of events is transmitted to heterogeneous subsets of the plurality of storage systems.

6 . The computer-implemented method of claim 1 , further comprising:

evicting a particular partition mapping from the partition map when at least one condition exists, the at least one condition selected from a group including:

each file descriptor of the partition is not in use,

each file referenced by any file descriptor of the partition exceeds a file size threshold, and

an elapse time of adding an event to the file satisfies a threshold timeout period.

7 . The computer-implemented method of claim 1 , further comprising:

adding a subset of the plurality of events to a vector;

detecting that the subset of the plurality of events satisfies a size threshold; and

process, based on the subset satisfying the size threshold, the subset to obtain a processed subset,

wherein the processed subset is appended to the file.

8 . The computer-implemented method of claim 1 , further comprising:

creating a folder structure on the at least one storage system to match a plurality of partitions, the plurality of partitions comprising the partition, the folder structure grouping files temporally, wherein a lowest level folder comprises a plurality of files defined based on a fixed time interval, wherein the plurality of files comprises the file.

9 . The computer-implemented method of claim 1 , further comprising:

after appending the event, detecting that a file size of the file is greater than a threshold; and

closing the file responsive to the detecting.

10 . The computer-implemented method of claim 1 , further comprising:

identifying a new time period corresponding to timestamps of the plurality of events;

partitioning the new time period into a plurality of partitions;

for each partition of at least a subset of the plurality of partitions:

creating a new file on the at least one storage system, and

obtaining a file descriptor for the new file; and

storing, in the partition map, the partition mapping to map the partition to the file descriptor.

11 . The computer-implemented method of claim 1 , further comprising:

creating a new file on the at least one storage system for a new partition,

obtaining a file descriptor for the new file;

storing, in the partition map, the partition mapping to map the new partition to the file descriptor;

generating a link to the file based on a path to the new file; and

outputting the link.

12 . The computer-implemented method of claim 1 , further comprising:

processing the event based on a rule for the at least one storage system, wherein the rule specifies at least one selected from a group consisting of a threshold file size, a timeout, and a compression amount.

13 . The computer-implemented method of claim 1 , further comprising:

processing the event based on a rule for the at least one storage system, wherein the rule specifies at least one selected from a group consisting of a partitioning scheme, a partitioning interval, a threshold file size, a timeout, and a compression amount,

wherein the plurality of storage systems has heterogeneous rules amongst the plurality of storage systems.

14 . A computing device, comprising:

a processor; and

a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations including:

obtaining a plurality of events for storage on one or more of a plurality of storage systems; and

for each event of the plurality of events:

extracting at least one field value from the event,

comparing the at least one field value to a plurality of configurations of the plurality of storage systems to identify at least one storage system of the plurality of storage systems having a matching configuration,

transmitting the event to an ingest module queue for the at least one storage system,

selecting a partition for the event based on the at least one field value and a partitioning rule for the at least one storage system to obtain a selected partition,

mapping the selected partition to a file using a partition mapping of a partition map, the partition map including a plurality of mappings between partition data and file pointers that uniquely identify corresponding open files, and

appending the event to the file on the at least one storage system based on a file pointer referencing the file identified via the mapping using the partition mapping.

15 . The computing device of claim 14 , wherein the plurality of events is transmitted to heterogeneous subsets of the plurality of storage systems.

16 . The computing device of claim 14 , wherein the operations further comprise:

adding a subset of the plurality of events to a vector;

detecting that the subset of the plurality of events satisfies a size threshold; and

process, based on the subset satisfying the size threshold, the subset to obtain a processed subset, wherein the processed subset is appended to the file.

17 . The computing device of claim 14 , wherein the operations further comprise:

creating a folder structure on the at least one storage system to match a plurality of partitions, the plurality of partitions comprising the partition, the folder structure grouping files temporally,

wherein a lowest level folder comprises a plurality of files defined based on a fixed time interval, wherein the plurality of files comprises the file.

18 . The computing device of claim 14 , wherein the operations further comprise:

after appending the event, detecting that a file size of the file is greater than a threshold; and

closing the file responsive to the detecting.

19 . A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform operations including:

obtaining a plurality of events for storage on one or more of a plurality of storage systems; and

for each event of the plurality of events:

extracting at least one field value from the event,

comparing the at least one field value to a plurality of configurations of the plurality of storage systems to identify at least one storage system of the plurality of storage systems having a matching configuration,

transmitting the event to an ingest module queue for the at least one storage system,

selecting a partition for the event based on the at least one field value and a partitioning rule for the at least one storage system to obtain a selected partition,

mapping the selected partition to a file using a partition mapping of a partition map, the partition map including a plurality of mappings between partition data and file pointers that uniquely identify corresponding open files, and

appending the event to the file on the at least one storage system based on a file pointer referencing the file identified via the mapping using the partition mapping.

20 . The non-transitory computer-readable medium of claim 19 , wherein the plurality of events is transmitted to heterogeneous subsets of the plurality of storage systems.