Using relative generation numbers to deduplicate file system events
Implementations are provided herein for using relative generation numbers for deduplicating kernel events modification events. The process can identify multiple modification events that take place on the same inode within a configurable relative amount of time and deduplicate the events against one another. A hash table can be used to store a global list of events associated with inodes, and thus only the hash table need be deduplicated. Filter buffer(s) setup when an Server Message Block (“SMB”) client requests a change notifications on a file and/or directory can then use the data from the hash table(s) to notify clients of change notify events.
1. A method comprising:
receiving a set of change notify requests received from a set of clients, wherein change notify requests are associated with at least one target file or target directory;
generating a set of user space filter buffers, wherein user space filter buffers are associated with change notify requests;
generating a hash table to store event entries wherein each event entry includes at least an event logical inode number (“LIN”), a parent LIN, an event type, and a generation number;
maintaining a hash table generation number associated with the hash table;
adding event entries to subsets of user space filters based on the event LIN associated with the event entry; and
notifying clients associated with the subset of user space filter buffers.
2. The method of claim 1 , further comprising:
receiving a new set of event entries from a kernel space buffer;
in response to receiving the new set of event entries:
incrementing the hash table generation number associated with the hash table; and
assigning the incremented hash table generation number as the generation number for each event entry in the new set of event entries; and
determining if the event LIN of an event entry in the new set of event entries matches the event LIN of an event entry in the hash table.
3. The method of claim 2 , further comprising:
in response to the event LIN of the event entry in the new set of event entries not matching the event LIN of an event entry in the hash table:
adding the event entry to the hash table; and
adding the event entry to the subset of user space filters associated with the event LIN.
4. The method of claim 2 , further comprising:
in response to the event LIN of the event entry in the new set of event entries matching the event LIN of an event entry in the hash table:
determining a value difference between the generation number of the event entry in the new set of event entries and the generation number of the event entry in the hash table;
in response to the value difference being less than a frequency threshold, deduplicating the event entry.
5. The method of claim 4 , further comprising:
in response to the value difference being more than the frequency threshold:
replacing the generation number of the event entry in the hash table with the generation number of the event entry in the new set of event entries.
6. The method of claim 1 , further comprising:
determining a value difference between the generation number of event entries of the hash table and the generation number associated with the hash table for each event entry in the hash table;
in response to the value difference being greater than a frequency threshold for an event entry in the hash table, deleting the event entry from the hash table.
7. A non-transitory computer readable medium with program instructions stored thereon to perform the following acts:
receiving a set of change notify requests received from a set of clients, wherein change notify requests are associated with at least one target file or target directory;
generating a set of user space filter buffers, wherein user space filter buffers are associated with change notify requests;
generating a hash table to store event entries wherein each event entry includes at least an event logical inode number (“LIN”), a parent LIN, an event type, and a generation number;
maintaining a hash table generation number associated with the hash table;
adding event entries to subsets of user space filters based on the event LIN associated with the event entry; and
notifying clients associated with the subset of user space filter buffers.
8. The non-transitory computer readable medium of claim 7 , with program instructions stored thereon to further perform the following acts:
receiving a new set of event entries from a kernel space buffer;
in response to receiving the new set of event entries:
incrementing the hash table generation number associated with the hash table; and
assigning the incremented hash table generation number as the generation number for each event entry in the new set of event entries; and
determining if the event LIN of an event entry in the new set of event entries matches the event LIN of an event entry in the hash table.
9. The non-transitory computer readable medium of claim 8 , with program instructions stored thereon to further perform the following acts:
in response to the event LIN of the event entry in the new set of event entries not matching the event LIN of an event entry in the hash table:
adding the event entry to the hash table; and
adding the event entry to the subset of user space filters associated with the event LIN.
10. The non-transitory computer readable medium of claim 8 , with program instructions stored thereon to further perform the following acts:
in response to the event LIN of the event entry in the new set of event entries matching the event LIN of an event entry in the hash table:
determining a value difference between the generation number of the event entry in the new set of event entries and the generation number of the event entry in the hash table;
in response to the value difference being less than a frequency threshold, deduplicating the event entry.
11. The non-transitory computer readable medium of claim 10 , with program instructions stored thereon to further perform the following acts:
in response to the value difference being more than the frequency threshold:
replacing the generation number of the event entry in the hash table with the generation number of the event entry in the new set of event entries.
12. The non-transitory computer readable medium of claim 7 , with program instructions stored thereon to further perform the following acts:
determining a value difference between the generation number of event entries of the hash table and the generation number associated with the hash table for each event entry in the hash table;
in response to the value difference being greater than a frequency threshold for an event entry in the hash table, deleting the event entry from the hash table.
13. A system comprising a cluster of nodes operating as a distributed file system wherein nodes among the cluster of nodes contain at least one storage device and at least one hardware processor configured to:
receive a set of change notify requests received from a set of clients, wherein change notify requests are associated with at least one target file or target directory;
generate a set of user space filter buffers, wherein user space filter buffers are associated with change notify requests;
generate a hash table to store event entries wherein each event entry includes at least an event logical inode number (“LIN”), a parent LIN, an event type, and a generation number;
maintain a hash table generation number associated with the hash table;
add event entries to subsets of user space filters based on the event LIN associated with the event entry; and
notify clients associated with the subset of user space filter buffers.
14. The system of claim 13 , further configured to:
receive a new set of event entries from a kernel space buffer;
in response to receiving the new set of event entries:
increment the hash table generation number associated with the hash table; and
assign the incremented hash table generation number as the generation number for each event entry in the new set of event entries; and
determine if the event LIN of an event entry in the new set of event entries matches the event LIN of an event entry in the hash table.
15. The system of claim 14 , further configured to:
in response to the event LIN of the event entry in the new set of event entries not matching the event LIN of an event entry in the hash table:
add the event entry to the hash table; and
add the event entry to the subset of user space filters associated with the event LIN.
16. The system of claim 14 , further configured to:
in response to the event LIN of the event entry in the new set of event entries matching the event LIN of an event entry in the hash table:
determine a value difference between the generation number of the event entry in the new set of event entries and the generation number of the event entry in the hash table;
in response to the value difference being less than a frequency threshold, deduplicate the event entry.
17. The system of claim 16 , further configured to:
in response to the value difference being more than the frequency threshold:
replace the generation number of the event entry in the hash table with the generation number of the event entry in the new set of event entries.
18. The system of claim 13 , further configured to:
determine a value difference between the generation number of event entries of the hash table and the generation number associated with the hash table for each event entry in the hash table;
in response to the value difference being greater than a frequency threshold for an event entry in the hash table, delete the event entry from the hash table.