Incremental metadata aggregation for a file storage system
The described technology is generally directed towards incremental aggregation of metadata for a file storage system. According to an embodiment, a system can comprise a memory and a processor that can execute the components stored in the memory. The components can comprise a scanner component that can accessing a data structure storage component that can store a first data structure, and a branch of the first data structure can comprise a node that comprises at least one descendent link to a descendant node. The scanner component can further traverse from a first node to a second node by employing a first descendent link. Further, the method comprises a data collector that can collect node data from the first node and the second node. The system can further comprise a rollup data generator to aggregate, upon occurrence of a condition, the node data, resulting in aggregated node data.
1. A system, comprising:
a memory that stores a cache and computer executable components; and
a processor that executes the computer executable components stored in the memory, wherein the computer executable components comprise:
a scanner component to execute processes comprising:
accessing a data structure storage component that stores a first data structure, wherein a branch of the first data structure comprises a node that comprises at least one descendent link to a descendant node; and
traversing from a first node to a second node by employing a first descendent link comprised in the first node;
a data collector component to collect first node data from the first node and second node data from the second node and copy the first node data and the second node data into a second data structure, wherein the first and second node data respectively comprise information describing first and second objects referenced by the first node and the second node; and
a rollup data generator component to generate, upon occurrence of a condition, aggregated node data based on the second data structure, wherein the rollup data generator component stores the aggregated node data in the first data structure, and wherein the rollup data generator component further stores the aggregated node data in the first data structure based on a process comprising:
generating a first database write command for storage of a first portion of the aggregated node data for the second node;
generating a second database write command for storage of a second portion of the aggregated node data for the first node, wherein the rollup data generator component combines the first database write command and the second database write command into a database transaction; and
periodically committing the database transaction after an interval.
2. The system of claim 1 , wherein the data collector component copying the first node data and the second node data into the second data structure comprises:
generating a shadow first node and a shadow second node in the second data structure;
copying the first node data into the shadow first node; and
copying the second node data into the shadow second node, wherein the rollup data generator component generates the aggregated node data based on the shadow first node and the shadow second node.
3. The system of claim 2 , wherein the data structure storage component comprises a permanent storage device, and wherein the second data structure is cached in the memory.
4. The system of claim 3 , wherein the rollup data generator component further removes the shadow first node and the shadow second node from the cache.
5. The system of claim 1 , wherein the condition comprises an absence of a second descendent link from the second node.
6. The system of claim 1 , wherein the node comprised in the first data structure corresponds a directory in a file storage system.
7. The system of claim 6 , wherein objects referenced by the node correspond to files stored in the directory.
8. The system of claim 7 , wherein the first node data further comprises a size in the file storage system of the files to which the objects referenced by the first node correspond.
9. The system of claim 1 , wherein the processor is a multi-core processor that employs a first thread and a second thread to execute tasks from a queue, wherein a first instance of the scanner component traverses from the first node to the second node by employing the first thread to execute a first task from the queue, wherein the scanner component further stores a second task in the queue based on a second descendent link to a third node comprised in the first node, and wherein a second instance of the scanner component employs the second thread to execute the second task from the queue.
10. The system of claim 9 , wherein the aggregated node data further comprises additional node data aggregated by the second thread, and wherein the aggregated node data is only stored in the first data structure after the first thread and the second thread have completed tasks associated with the first node.
11. A method, comprising:
accessing, by a system comprising a processor, a data structure storage component that stores a first data structure, wherein a branch of the first data structure comprises a first node that comprises at least one descendent link to a descendant node;
collecting first node data from the first node and second node data from a second node by traversing from the first node to the second node by employing a first descendent link comprised in the first node;
copying the first node data and the second node data into a second data structure, wherein the first and second node data respectively comprise information describing first and second objects referenced by the first node and the second nodes;
generating, upon occurrence of a condition, aggregated node data based on the second data structure; and
storing the aggregated node data in the first data structure based on a process comprising:
generating a first database write command for storing a first portion of the aggregated node data for the second node,
generating a second database write command for storing a second portion of the aggregated node data for the first node,
combining the first datastore write command and the second datastore write command into a datastore transaction; and
periodically committing the datastore transaction after an interval.
12. The method of claim 11 , wherein the copying the first node data and the second node data into the second data structure comprises:
generating a shadow first node and a shadow second node in the second data structure;
copying the first node data into the shadow first node; and
copying the second node data into the shadow second node, wherein the generating the aggregated node data is based on the shadow first node and the shadow second node.
13. The method of claim 12 , wherein the first node data comprises a number of objects that are referenced by the first node, and wherein the objects referenced by the first node correspond to files stored in a directory of directories, to which the first node corresponds.
14. The method of claim 12 , wherein the first node data comprises a size of a file stored in a directory of the directories, to which the objects referenced by the first node correspond.
15. A non-transitory machine-readable storage medium comprising executable instructions that, when executed by a processor, facilitate performance of operations, the operations comprising:
accessing, by a system comprising a processor, a data structure storage component that stores a first data structure, wherein a branch of the first data structure comprises a first node that comprises at least one descendent link to a descendant node;
collecting first node data from the first node and second node data from a second node by employing a first descendent link comprised in the first node, wherein the first node data and the second node data respectively comprise first information and second information respectively describing a first object and a second object respectively referenced by the first node and the second node;
copying the first node data and the second node data into a second data structure; and
generating, upon occurrence of a condition, aggregated node data based on the second data structure,
storing the aggregated node data in the first data structure based on a process comprising:
generating a first database write command for storing a first portion of the aggregated node data for the second node,
generating a second database write command for storing a second portion of the aggregated node data for the first node,
combining the first database write command and the second database write command into a database transaction, and
committing the database transaction after a defined interval.
16. The non-transitory machine-readable storage medium of claim 15 , wherein the first node and the second node respectively correspond to directories in a file storage system.
17. The non-transitory machine-readable storage medium of claim 16 , wherein the aggregated node data comprises a number of objects referenced by a respective node, and wherein objects referenced by the respective node correspond to files stored in a directory of the directories in the file storage system, to which the respective node corresponds.
18. The non-transitory machine-readable storage medium of claim 17 , wherein the aggregated node data further comprises a size in the file storage system of the files to which the objects referenced by the respective node correspond.
19. The non-transitory machine-readable storage medium of claim 15 , wherein copying the first node data and the second node data into the second data structure comprises:
generating a shadow first node and a shadow second node in the second data structure;
copying the first node data into the shadow first node; and
copying the second node data into the shadow second node, wherein the generating the aggregated node data is based on the shadow first node and the shadow second node.
20. The non-transitory machine-readable storage medium of claim 19 , wherein the data structure storage component comprises a permanent storage device, and wherein the second data structure is cached in memory.