Analyzing and visualizing trends in the use and consumption of data in a shared storage system
A facility comprising methods and systems for analyzing and visualizing trends in the usage of data within a shared storage filesystem is disclosed. The facility analyzes the rate at which the filesystem or a portion thereof is used by periodically generating manifests of the usage of the filesystem and comparing one manifest to another manifest. Furthermore, the facility may store additional information relevant to the manifest, such as the time at which the manifest was taken; for each directory, the sum of all of the values determined for items in that directory (including any subdirectories); and so on. In this manner, the facility collects and stores information relevant to developing trend information for each item in the filesystem. The trends analysis and visualizations described herein provide quick insight into the changes deemed most interesting or significant between two times.
1. A method for analyzing data use in a computing system, wherein execution of the method by one or more processors performs actions, comprising:
employing a file system to periodically perform actions, including:
determining one or more paths in a plurality of directories at a first instance of time, wherein each determined path is associated with one or more items in a directory;
determining each first value of each item at the first instance of time in each directory;
employing a first value of each item that exceeds a first threshold to identify one or more items that are significant and a path in each directory that is associated with the one or more significant items;
employing a second instance of time to determine a second value for the one or more significant items; and
employing each difference that exceeds a second threshold between each first value and each second value for the one or more significant times to identify each change to each path that is significant and associated with the one or more significant items, wherein each significant path is displayed to a user.
2. The method of claim 1 , wherein each item further comprises one or more of a sub-directory or a file.
3. The method of claim 1 , further comprising:
storing each first value and each second value of the one or more significant items and each path associated with the one or more significant items in a manifest data structure; and
storing each difference between each first value and each second value for the one or more significant items in a change data structure.
4. The method of claim 1 , wherein the display of the significant path further comprises a visual representation of one or more of a graph, chart, or tabular data.
5. The method of claim 1 , further comprising:
employing each change to each path to determine a relevance of a significant path based on an amount of each change that is attributable to one or more of a directory or an item.
6. The method of claim 1 , further comprising storing
generating a node for a b-tree, wherein the node comprises the first value and the path associated with the one or more significant items; and
adding the generated node to the b-tree.
7. A computer-readable non-transitory storage medium storing instructions that, when executed by a computing system having a memory and a processor, cause the computing system to perform a method, the method comprising:
employing a file system to periodically perform actions, including:
determining one or more paths in a plurality of directories at a first instance of time, wherein each determined path is associated with one or more items in a directory;
determining each first value of each item at the first instance of time in each directory;
employing a first value of each item that exceeds a first threshold to identify one or more items that are significant and a path in each directory that is associated with the one or more significant items;
employing a second instance of time to determine a second value for the one or more significant items; and
employing each difference that exceeds a second threshold between each first value and each second value for the one or more significant times to identify each change to each path that is significant and associated with the one or more significant items, wherein each significant path is displayed to a user.
8. The computer-readable non-transitory storage medium of claim 7 , wherein each item further comprises one or more of a sub-directory or a file.
9. The computer-readable non-transitory storage medium of claim 7 , further comprises:
storing each first value and each second value of the one or more significant items and each path associated with the one or more significant items in a manifest data structure; and
storing each difference between each first value and each second value for the one or more significant items in a change data structure.
10. The computer-readable non-transitory storage medium of claim 7 , wherein the display of the significant path further comprises a visual representation of one or more of a graph, chart, or tabular data.
11. The computer-readable non-transitory storage medium of claim 7 , further comprising:
employing each change to each path to determine a relevance of a significant path based on an amount of each change that is attributable to one or more of a directory or an item.
12. The computer-readable non-transitory storage medium of claim 7 , further comprising:
generating a node for a b-tree, wherein the node comprises the first value and the path associated with the one or more significant items; and
adding the generated node to the b-tree.
13. A computer, comprising:
a memory to store instructions; and
one or more processors that execute the instructions to perform one or more actions, including:
employing a file system to periodically perform actions, including:
determining one or more paths in a plurality of directories at a first instance of time, wherein each determined path is associated with one or more items in a directory;
determining each first value of each item at the first instance of time in each directory;
employing a first value of each item that exceeds a first threshold to identify one or more items that are significant and a path in each directory that is associated with the one or more significant items;
employing a second instance of time to determine a second value for the one or more significant items; and
employing each difference that exceeds a second threshold between each first value and each second value for the one or more significant times to identify each change to each path that is significant and associated with the one or more significant items, wherein each significant path is displayed to a user.
14. The computer of claim 13 , wherein each item further comprises one or more of a sub-directory or a file.
15. The computer of claim 13 , further comprises:
storing each first value and each second value of the one or more significant items and each path associated with the one or more significant items in a manifest data structure; and
storing each difference between each first value and each second value for the one or more significant items in a change data structure.
16. The computer of claim 13 , wherein the display of the significant path further comprises a visual representation of one or more of a graph, chart, or tabular data.
17. The computer of claim 13 , further comprising:
employing each change to each path to determine a relevance of a significant path based on an amount of each change that is attributable to one or more of a directory or an item.
18. The computer of claim 13 , further comprising:
generating a node for a b-tree, wherein the node comprises the first value and the path associated with the one or more significant items; and
adding the generated node to the b-tree.