Storage system event handling with capacity based eviction

1 . A computer-implemented method, comprising:

obtaining a plurality of events for storage on one or more storage systems;

for each event of the plurality of events:

comparing at least one field value of the event to a plurality of configurations of the one or more storage systems to identify a first storage system having a matching configuration,

selecting, from a plurality of partitions, a partition for the event based on the at least one field value and a partitioning scheme of the first storage system to obtain a selected partition, and

storing the event in a sub-queue, for the selected partition, of a partition queue;

detecting that the partition queue satisfies a queue size rule;

selecting, responsive to detecting the partition queue satisfying the queue size rule, a subset of partitions of the plurality of partitions to evict from the partition queue prior to each partition of the subset of partitions of the plurality of partitions individually satisfying a file size criterion or a timeout criterion for eviction, the subset of partitions of the plurality of partitions selected based on a plurality of attributes of the subset of partitions indicating that each partition of the subset of partitions is closer to satisfying the file size criterion or the timeout criterion for eviction than a set of remaining partitions of the plurality of partitions; and

transmitting events in the subset of partitions of the plurality of partitions in the partition queue to the first storage system.

2 . The method of claim 1 , further comprising:

calculating a total size of the partition queue;

comparing the total size to a maximum threshold; and

iteratively evicting partitions in the subset of partitions until the total size of the partition queue satisfies a reduction threshold,

wherein the queue size rule comprises the maximum threshold and the reduction threshold.

3 . The method of claim 1 , further comprising:

evicting a first portion of the plurality of partitions satisfying an individual eviction rule.

4 . The method of claim 1 , further comprising:

evicting a first portion of the plurality of partitions having a time attribute of the plurality of attributes satisfying the timeout criterion.

5 . The method of claim 1 , further comprising:

evicting a first portion of the plurality of partitions having a partition size attribute of the plurality of attributes satisfying the file size criterion.

6 . The method of claim 1 , further comprising:

iteratively evicting partitions in the subset of partitions until a total size of the partition queue satisfies a reduction threshold.

7 . The method of claim 1 , wherein transmitting the events comprises:

transmitting the events to an ingest module queue for the first storage system, and processing the events from the ingest module queue.

8 . The method of claim 1 , wherein transmitting the events comprises:

generating a file from the events in the selected subset of partitions,

setting a location of the file in a file system of the first storage system based on the selected subset of partitions, and

transmitting the file to the first storage system with the location.

9 . The method of claim 1 , wherein transmitting the events comprises:

generating a file from the events in the selected subset of partitions,

generating a path for the file in the first storage system based on the selected subset of partitions,

generating a link to the file based on the path,

transmitting the file to the first storage system, and

outputting the link.

10 . The method of claim 1 , further comprising:

partitioning a first subset of events of the plurality of events according to a first partitioning rule, wherein the first subset of events is assigned to the first storage system; and

partitioning a second subset of events assigned to a second storage system of the plurality of events according to a second partitioning rule, wherein the second subset of events is assigned to the second storage system of the one or more storage systems;

wherein the first partitioning rule is different than the second partitioning rule.

11 . The method of claim 1 , further comprising:

generating a file for the events in the subset of partitions based on a rule for the first storage system,

wherein the one or more storage systems is a plurality of storage systems, and wherein the plurality of storage systems has heterogeneous rules amongst the plurality of storage systems.

12 . A computing device, comprising:

a processor; and

a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations including:

obtaining a plurality of events for storage on one or more storage systems;

for each event of the plurality of events:

comparing at least one field value of the event to a plurality of configurations of the one or more storage systems to identify a first storage system having a matching configuration,

selecting, from a plurality of partitions, a partition for the event based on the at least one field value and a partitioning scheme of the first storage system to obtain a selected partition, and

storing the event in a sub-queue for the selected partition of a partition queue;

detecting that the partition queue satisfies a queue size rule;

selecting, responsive to detecting the partition queue satisfying the queue size rule, a subset of partitions of the plurality of partitions to evict from the partition queue prior to each partition of the subset of partitions of the plurality of partitions individually satisfying a file size criterion or a timeout criterion for eviction, the subset of partitions of the plurality of partitions selected based on a plurality of attributes of the subset of partitions indicating that each partition of the subset of partitions is closer to satisfying the file size criterion or the timeout criterion for eviction than a set of remaining partitions of the plurality of partitions; and

transmitting events in the subset of partitions of the plurality of partitions in the partition queue to the first storage system.

13 . The computing device of claim 12 , wherein the operations further comprise:

calculating a total size of the partition queue;

comparing the total size to a maximum threshold; and

iteratively evicting partitions in the subset of partitions until the total size of the partition queue satisfies a reduction threshold,

wherein the queue size rule comprises the maximum threshold and the reduction threshold.

14 . The computing device of claim 12 , wherein the operations further comprise:

evicting a first portion of the plurality of partitions satisfying an individual eviction rule.

15 . The computing device of claim 12 , wherein the operations further comprise:

evicting a first portion of the plurality of partitions having a time attribute of the plurality of attributes satisfying the timeout criterion.

16 . The computing device of claim 12 , wherein the operations further comprise:

evicting a first portion of the plurality of partitions having a partition size attribute of the plurality of attributes satisfying the file size criterion.

17 . The computing device of claim 12 , wherein the operations further comprise:

iteratively evicting partitions in the subset of partitions until a total size of the partition queue satisfies a reduction threshold.

18 . The computing device of claim 12 , wherein transmitting the events comprises:

transmitting the events to an ingest module queue for the first storage system, and

processing the events from the ingest module queue.

19 . A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processor to perform operations including:

obtaining a plurality of events for storage on one or more storage systems;

for each event of the plurality of events:

comparing at least one field value of the event to a plurality of configurations of the one or more storage systems to identify a first storage system having a matching configuration,

selecting, from a plurality of partitions, a partition for the event based on the at least one field value and a partitioning scheme of the first storage system to obtain a selected partition, and

storing the event in a sub-queue for the selected partition of a partition queue; and

detecting that the partition queue satisfies a queue size rule;

selecting, responsive to detecting the partition queue satisfying the queue size rule, a subset of partitions of the plurality of partitions to evict from the partition queue prior to each partition of the subset of partitions of the plurality of partitions individually satisfying a file size criterion or a timeout criterion for eviction, the subset of partitions of the plurality of partitions selected based on a plurality of attributes of the subset of partitions indicating that each partition of the subset of partitions is closer to satisfying the file size criterion or the timeout criterion for eviction than a set of remaining partitions of the plurality of partitions; and

transmitting events in the subset of partitions of the plurality of partitions in the partition queue to the first storage system.

20 . The non-transitory computer-readable medium of claim 19 , wherein the operations further comprise:

calculating a total size of the partition queue;

comparing the total size to a maximum threshold; and

iteratively evicting partitions in the subset of partitions until the total size of the partition queue satisfies a reduction threshold,

wherein the queue size rule comprises the maximum threshold and the reduction threshold.