File analytics systems and methods including retrieving metadata from file system snapshots
Examples of file analytics systems are described that may obtain metadata data from a virtualized file server using a snapshot of the file virtualized file. The analytics system may mount a snapshot to retrieve metadata used in determining an overall size, structure, owner, permissions, and/or storage locations of parts of a file system managed by the virtualized file server. The analytics tool may communicate directly with each of file server virtual machines of the virtualized file server during the metadata collection process to retrieve the respective portions of the metadata. During the metadata collection process, the analytics system may add a checkpoint or marker (e.g., index) after every completed metadata transaction to indicate where it left off accessing the metadata snapshots. The checkpoint may allow the analytics system to return to the checkpoint to resume the process should the process be interrupted for some reason.
1. At least one non-transitory computer-readable storage medium including instructions that, when executed by a computing node, cause the computing node to:
mount a snapshot of a file server virtual machine (FSVM) of a virtualized file server (VFS), the snapshot including a portion of a share hosted by at least the FSVM, the portion of the share comprising a plurality of files managed by the FSVM and metadata associated with the plurality of files;
retrieve first metadata from a first portion of the snapshot, wherein the first metadata is associated with at least one file of the plurality of files;
add a checkpoint to indicate that the first metadata is obtained;
retrieve second metadata from the snapshot, wherein the second metadata is associated with at least another file of the plurality of files;
update the checkpoint to indicate that the second metadata has been obtained from the snapshot; and
after an interruption, determine where to resume obtaining metadata from the snapshot based on the checkpoint.
2. The at least one non-transitory computer-readable storage medium of claim 1 , wherein the instructions, when executed, cause the computing node to obtain third metadata using a second snapshot of a second FSVM for a subset of files of a share managed by the second FSVM of the VFS in response to the first metadata indicating that that the second FSVM manages the subset of files of the share managed by the second FSVM.
3. The at least one non-transitory computer-readable storage medium of claim 2 , wherein the instructions, when executed, cause the computing node to analyze the first metadata to detect a hierarchy of the share managed by the second FSVM including the subset of files managed by the second FSVM.
4. The at least one non-transitory computer-readable storage medium of claim 1 , wherein the instructions, when executed, cause the computing node to generate an event based on detection of a metadata change for a file after comparing the first metadata to third metadata associated with a scan of a second snapshot.
5. The at least one non-transitory computer-readable storage medium of claim 4 , wherein the instructions, when executed, cause the computing node to obtain the third metadata from the second snapshot.
6. The at least one non-transitory computer-readable storage medium of claim 4 , wherein the instructions, when executed, cause the computing node to generate a move, add, or delete event for the file based on a comparison of the first metadata to the third metadata.
7. The at least one non-transitory computer-readable storage medium of claim 1 , wherein the instructions, when executed, cause the computing node to create an entry in an analytics data store for a metadata category of a first file based on the first metadata in response to lack of detection of an entry corresponding to the metadata category for the first file.
8. The at least one non-transitory computer-readable storage medium of claim 7 , wherein the instructions, when executed, cause the computing node to ignore the metadata category for the first file included in the first metadata in response to detection of that an entry exists for the metadata category of the first file exists.
9. The at least one non-transitory computer-readable storage medium of claim 1 , wherein the instructions, when executed, cause the computing node to create a first entry in an analytics data store based on the first metadata.
10. The at least one non-transitory computer-readable storage medium of claim 9 , wherein the instructions, when executed, cause the computing node to create an event entry for the file in the analytics data store based on an event data record received via an event pipeline.
11. The at least one non-transitory computer-readable storage medium of claim 10 , wherein the instructions, when executed, cause the computing node to create the event entry for a data item to present a change to the file relative to a state indicated by the first entry.
12. A system comprising:
a distributed file server comprising a cluster of file server virtual machines (FSVMs), wherein the distributed file server is configured to generate a first snapshot including a portion of a share hosted by at least a FSVM, the portion of the share comprising a plurality of files managed by the FSVM and metadata associated with the plurality of files; and
an analytics virtual machine configured to access the first snapshot to obtain first metadata associated with at least one file of the plurality of files, wherein the analytics virtual machine is further configured to analyze the first metadata to determine storage locations of files of the share,
wherein the first metadata is associated with at least one file of a virtual file server (VFS),
wherein, in response to a determination that a subset of the files of the share are managed by a second FSVM of the cluster of FSVMs, access a second snapshot of the second FSVM to retrieve second metadata of the share corresponding to the subset of files,
wherein the second metadata is associated with at least another file of the VFS.
13. The system of claim 12 , wherein the analytics virtual machine is further configured to add a checkpoint to indicate where to start a next access of the first snapshot.
14. The system of claim 12 , wherein the distributed file server is configured to generate the second snapshot corresponding to second files managed by the distributed file server managed by the second FSVM.
15. The system of claim 12 , wherein the analytics virtual machine is further configured to analyze the first metadata to detect a hierarchy of the share including the subset of files managed by the second FSVM.
16. The system of claim 12 , wherein the analytics virtual machine is further configured to generate, via an analytics tool, an event based on detection of a metadata change for one of the plurality of files after comparing the first metadata to third metadata associated with an access of a third snapshot.
17. The system of claim 16 , wherein the analytics virtual machine is further configured to access the third snapshot to retrieve the third metadata.
18. The system of claim 12 , wherein the analytics virtual machine is further configured to create an entry in an analytics data store for a metadata category of one of the plurality of files based on the first metadata in response to lack of detection of an entry corresponding to the metadata category of the one of the plurality of files.
19. The system of claim 18 , wherein the analytics virtual machine is further configured to ignore the metadata category for the one of the plurality of files included in the first metadata in response to detection of that an entry exists for the metadata category of the one of the plurality of files.
20. A method, comprising:
mounting, by an analytics virtual machine, a snapshot of a file server virtual machine (FSVM) of a virtualized file server (VFS), the snapshot including a portion of a share hosted by at least the FSVM, the portion of the share comprising a plurality of files managed by the FSVM and metadata associated with the plurality of files;
accessing the snapshot to retrieve first metadata, wherein the first metadata is associated with at least one file of the plurality of files;
adding a checkpoint to indicate that an access to the first metadata of the snapshot is complete;
accessing the snapshot to retrieve second metadata, wherein the second metadata is associated with at least another file of the plurality of files;
updating the checkpoint to indicate that the access to the second metadata of the snapshot is complete; and
after an interruption, determining where to resume scanning of the snapshot based on the checkpoint.
21. The method of claim 20 , further comprising accessing a second snapshot of a second FSVM to gather third metadata for a subset of files of a share managed by the second FSVM of the VFS in response to the first metadata indicating that that the second FSVM manages the subset of files of the share.
22. The method of claim 21 , further comprising analyzing the first metadata to detect a hierarchy of the share managed by the second FSVM including the subset of files managed by the second FSVM.
23. The method of claim 20 , further comprising generating an event based on detection of a metadata change for a file after comparing the first metadata to third metadata associated with an access of a second snapshot.
24. The method of claim 23 , further comprising accessing the second snapshot to retrieve the third metadata.
25. The method of claim 23 , further comprising generating a move, add, or delete event for the file based on a comparison of the first metadata to the third metadata.
26. The method of claim 20 , further comprising creating an entry in an analytics data store for a metadata category of a first file based on the first metadata in response to lack of detection of an entry corresponding to the metadata category for the first file.
27. The method of claim 26 , further comprising ignoring the metadata category for the file included in the first metadata in response to detection of that an entry exists for the metadata category of the first file exists.
28. The method of claim 20 , further comprising creating a first entry in an analytics data store based on the first metadata.
29. The at least one non-transitory computer-readable storage medium of claim 1 , wherein the metadata comprises size information, structure information, ownership information, permissions information, storage location information, or combinations thereof, associated with the FSVM of the virtualized file server.
30. The at least one non-transitory computer-readable storage medium of claim 1 , wherein the metadata comprises file size information, allocated storage quota information, creation information, modification information, ownership information, permissions information, or combinations thereof, associated with the plurality of files.
31. The system of claim 12 , wherein the metadata comprises size information, structure information, ownership information, permissions information, storage location information, or combinations thereof, associated with the FSVMs, including the FSVM.
32. The system of claim 12 , wherein the metadata comprises file size information, allocated storage quota information, creation information, modification information, ownership information, permissions information, or combinations thereof, associated with the plurality of files.
33. The method of claim 20 , wherein the metadata comprises size information, structure information, ownership information, permissions information, storage location information, or combinations thereof, associated with the FSVM.
34. The method of claim 20 , wherein the metadata comprises file size information, allocated storage quota information, creation information, modification information, ownership information, permissions information, or combinations thereof, associated with the plurality of files.