Parallel processing of filtered transaction logs

1. A storage system, comprising:

at least one storage node configured to execute storage operations;

a transaction log configured to store a plurality of log entries corresponding to storage operations for the at least one storage node;

a first metadata streamer configured to sequentially retrieve a set of log entries from the transaction log; and

a plurality of transaction log filters configured to selectively provide filtered subsets of the set of log entries from the first metadata streamer to a plurality of metadata operations, wherein:

each transaction log filter of the plurality of transaction log filters corresponds to a different metadata operation of the plurality of metadata operations;

each different metadata operation of the plurality of metadata operations is configured to process a different filtered subset of the set of log entries;

the different metadata operations of the plurality of metadata operations are configured to operate in parallel on the different filtered subsets of the set of log entries;

a first transaction log filter of the plurality of transaction log filters is configured with a first metadata operation type selected from a group of metadata operation types, the group including:

replication;

notification;

metadata indexing;

garbage collection;

data scrub; and

parameter calculation;

a second transaction log filter of the plurality of transaction log filters is configured with a second metadata operation type selected from the group of metadata operation types;

the first metadata operation type is different than the second metadata operation type;

each metadata operation of the plurality of metadata operations is configured with an operation group;

a selected operation group includes a plurality of synchronized metadata operations; and

filtered subsets of the set of log entries are configured for release when each metadata operation for the selected operation group are all complete.

2. The storage system of claim 1 , further comprising:

a fanout socket configured to:

receive the set of log entries from the first metadata streamer;

replicate the set of log entries for each transaction log filter of the plurality of transaction log filters; and

push a replicated set of log entries to each transaction log filter of the plurality of transaction log filters.

3. The storage system of claim 2 , wherein:

the fanout socket further includes an expandable interface for interfacing with a variable number of transaction log filters; and

the plurality of transaction log filters includes at least three pluggable transaction log filters.

4. The storage system of claim 1 , further comprising:

a plurality of filtered transaction logs configured to temporarily store the filtered subsets of the set of log entries from the plurality of transaction log filters, wherein each transaction log filter of the plurality of transaction log filters corresponds to an assigned filtered transaction log from the plurality of filtered transaction logs.

5. The storage system of claim 4 , further comprising:

a plurality of additional metadata streamers configured to fetch log entries from the plurality of filtered transaction logs, wherein each assigned filtered transaction log from the plurality of filtered transaction logs corresponds to an assigned metadata streamer from the plurality of additional metadata streamers.

6. The storage system of claim 1 , wherein:

the first transaction log filter is configured with a first log entry selection criterion;

the second transaction log filter is configured with a second log entry selection criterion; and

the first log entry selection criterion and the second log entry selection criterion are different.

7. The storage system of claim 1 , further comprising:

a reference counter configured to:

track a processing number of metadata operations corresponding to the plurality of metadata operations; and

release, responsive to completing the plurality of metadata operations, the filtered subsets of the set of log entries from the first metadata streamer.

8. The storage system of claim 7 , wherein the reference counter is further configured to receive a completion notification from each metadata operation of the plurality of metadata operations.

9. The storage system of claim 1 , wherein each transaction log filter of the plurality of transaction log filters includes:

an operation type;

an expiration condition; and

at least one log entry selection criteria.

10. A computer-implemented method, comprising:

storing a transaction log comprised of a plurality of log entries corresponding to storage operations for at least one storage node;

sequentially retrieving a set of log entries from the transaction log;

selectively providing, using a first transaction log filter, a first filtered subset of the set of log entries to a first metadata operation, wherein the first metadata operation has a first metadata operation type selected from a group of metadata operation types, the group including:

replication;

notification;

metadata indexing;

garbage collection;

data scrub; and

parameter calculation;

selectively providing, using a second transaction log filter, a second filtered subset of the set of log entries to a second metadata operation, wherein the second metadata operation has a second metadata operation type selected from the group of metadata operation types;

and

executing in parallel:

the first metadata operation on the first filtered subset of the set of log entries; and

the second metadata operation on the second filtered subset of the set of log entries,

wherein:

the first filtered subset is a different filtered subset from the second filtered subset;

the first metadata operation type is a different metadata operation type from the second metadata operation type;

the first metadata operation and the second metadata operation are in a selected operation group; and

the first filtered subset and the second filtered subset are released when each metadata operation for the selected operation group are all complete.

11. The computer-implemented method of claim 10 , further comprising:

replicating the set of log entries for:

the first transaction log filter; and

the second transaction log filter;

pushing a first replicated set of log entries to the first transaction log filter; and

pushing a second replicated set of log entries to the second transaction log filter.

12. The computer-implemented method of claim 10 , further comprising:

adding a third transaction log filter for a third metadata operation;

selectively providing, using the third transaction log filter, a third filtered subset of the set of log entries to the third metadata operation; and

executing, in parallel with the first metadata operation and the second metadata operation, the third metadata operation on the third filtered subset of the set of log entries.

13. The computer-implemented method of claim 10 , further comprising:

storing the first filtered subset of the set of log entries in a first filtered transaction log;

storing the second filtered subset of the set of log entries in a second filtered transaction log;

removing, responsive to completing the first metadata operation on the first filtered subset, the first filtered subset from the first filtered transaction log; and

removing, responsive to completing the second metadata operation on the second filtered subset, the second filtered subset from the second filtered transaction log.

14. The computer-implemented method of claim 13 , further comprising:

fetching, using a first metadata streamer, log entries from the first filtered transaction log;

fetching, using a second metadata streamer, log entries from the second filtered transaction log;

requesting, by the first metadata operation, log entries from the first filtered transaction log; and

requesting, by the second metadata operation, log entries from the second filtered transaction log.

15. The computer-implemented method of claim 10 , further comprising:

determining a first log entry selection criterion for the first transaction log filter; and

determining a second log entry selection criterion for the second transaction log filter, wherein the first log entry selection criterion and the second log entry selection criterion are different.

16. The computer-implemented method of claim 10 , further comprising:

tracking a processing number of metadata operations for a plurality of metadata operations, wherein the plurality of metadata operations includes the first metadata operation and the second metadata operation; and

releasing, responsive to the processing number of metadata operations completing, the set of log entries retrieved from the transaction log.

17. The computer-implemented method of claim 16 , further comprising:

receiving a completion notification from each metadata operation of the plurality of metadata operations; and

determining, responsive to receiving the completion notification from each metadata operation of the plurality of metadata operations, the processing number of metadata operations are complete.

18. The computer-implemented method of claim 10 , further comprising:

determining a first expiration condition for the first transaction log filter;

determining a second expiration condition for the second transaction log filter; and

releasing, responsive to the first expiration condition and the second expiration condition being met, the set of log entries.

19. A system, comprising:

at least one storage node configured to execute storage operations;

a transaction log comprised of a plurality of log entries corresponding to storage operations for the at least one storage node;

means for sequentially retrieving a set of log entries from the transaction log;

means for selectively providing, using a first transaction log filter, a first filtered subset of the set of log entries to a first metadata operation, wherein the first metadata operation has a first metadata operation type selected from a group of metadata operation types, the group including:

replication;

notification;

metadata indexing;

garbage collection;

data scrub; and

parameter calculation;

means for selectively providing, using a second transaction log filter, a second filtered subset of the set of log entries to a second metadata operation, wherein the second metadata operation has a second metadata operation type selected from the group of metadata operation types;

and

means for executing in parallel:

the first metadata operation on the first filtered subset of the set of log entries; and

the second metadata operation on the second filtered subset of the set of log entries, wherein:

the first filtered subset is a different filtered subset from the second filtered subset;

the first metadata operation type is a different metadata operation type from the second metadata operation type;

the first metadata operation and the second metadata operation are in a selected operation group; and

the first filtered subset and the second filtered subset are released when each metadata operation for the selected operation group are all complete.