Aggregating metrics in file systems using structured journals
Embodiments are directed to managing file systems. Update information associated with a change of a metric associated with a target object may be provided. A journal that includes a base bin that includes base records that associate the metric with each object in the file system. Records that include the change of the metric associated with the ancestors of target object may be generated. Another record that includes the change of the metric associated with the target object may be generated. A level bin associated with the base bin of the journal may be provided based on the update information. The records may be stored in the level bin using a sort order based on the ordering of the base bin records. In response a query, the journal may be employed to reduce latency in generating query results.
1 . A method comprising:
providing update information associated with a target object in a file system, wherein the update information is associated with a change of a value of a metric associated with the target object from an initial value of the metric associated with the target object;
providing a journal based on the update information, wherein the journal includes a base bin that includes one or more base records, the one or more base records comprising a first column indicating a respective path-key for each object of a plurality of objects included in the file system and a second column indicating the initial value of the metric associated with the target object, the respective path-key corresponding to a respective file system path of the object;
determining one or more objects in the file system that are different from the target object and are hierarchical ancestors in the file system for the target object based on the update information and the file system;
generating one or more level bin records that correspond to the one or more objects, wherein each level bin record includes:
a third column comprising the change of the value of the metric associated with the target object and the change of the value of the metric associated with the one or more objects in the file system that are hierarchical ancestors in the file system for the target object, and
a fourth column comprising the respective path-key for each of the one or more objects;
providing a level bin that is associated with the base bin of the journal based on the update information; and
using the base bin and the level bin to reduce latency in generating, in response to one or more queries for the value of the metric associated with the target object or the value of the metric associated with the one or more objects, one or more query results comprising the changed value of the metric associated with the target object or the changed value of the metric associated with the one or more objects.
2 . The method of claim 1 , further comprising:
storing the one or more level bin records in the level bin, wherein a sort order of the one or more level bin records in the level bin is based on an ordering of the one or more base records in the base bin.
3 . The method of claim 1 , wherein generating the one or more query results further comprises:
determining a base value for each metric associated with the one or more query results based on the base bin;
determining one or more partial values for each metric associated with the one or more query results based on the one or more level bin records; and
generating the one or more query results based on the one or more partial values and the base value.
4 . The method of claim 1 , further comprising:
generating a new level bin based on one or more of the update information, an age of the one or more level bin records, or a capacity of the one or more level bin records.
5 . The method of claim 1 , further comprising:
providing a hierarchical index based on the file system; and
generating the one or more base records in the base bin based on the hierarchical index.
6 . The method of claim 1 , further comprising:
determining one or more partial values associated with one or more changes to one or more metrics comprising the metric based on one or more level bins in the journal;
updating the one or more base records in the base bin based on the one or more partial values; and
discarding the one or more level bins.
7 . The method of claim 1 , wherein the one or more base records are associated with a separate value of the metric with each object in the file system including the target object, and wherein the one or more base records are ordered based on the file system.
8 . The method of claim 1 , wherein each level bin comprises a hierarchical index.
9 . The method of claim 8 , wherein the hierarchical index comprises a fifth column indicating respective identifier attributes associated with each of the one or more objects.
10 . A system, comprising:
one or more network computers, comprising:
a transceiver;
a memory that is arranged to store at least instructions; and
one or more processor devices that are configured to execute the instructions, wherein the instructions are configured to cause the system to perform actions, comprising:
providing update information associated with a target object in a file system, wherein the update information is associated with a change of a value of a metric associated with the target object from an initial value of the metric associated with the target object;
providing a journal based on the update information, wherein the journal includes a base bin that includes one or more base records, the one or more base records comprising a first column indicating a respective path-key for each object of a plurality of objects included in the file system and a second column indicating the initial value of the metric associated with the target object, the respective path-key corresponding to a respective file system path of the object;
determining one or more objects in the file system that are different from the target object and are hierarchical ancestors in the file system for the target object based on the update information and the file system;
generating one or more level bin records that correspond to the one or more objects wherein each level bin record includes:
a third column comprising the change of the value of the metric associated with the target object and the change of the value of the metric associated with the one or more objects in the file system that are hierarchical ancestors in the file system for the target object, and
a fourth column comprising the respective path-key for each of the one or more objects;
providing a level bin that is associated with the base bin of the journal based on the update information; and
using the base bin and the level bin to reduce latency in generating, in response to one or more queries for the value of the metric associated with the target object or the value of the metric associated with the one or more objects, one or more query results comprising the changed value of the metric associated with the target object or the changed value of the metric associated with the one or more objects.
11 . The system of claim 10 , wherein the instructions are further configured to cause the system to perform further actions, comprising:
storing the one or more level bin records in the level bin, wherein a sort order of the one or more level bin records in the level bin is based on an ordering of the one or more base records in the base bin.
12 . The system of claim 10 , wherein the instructions are further configured to cause the system to perform further actions, comprising:
determining a base value for each metric associated with the one or more query results based on the base bin;
determining one or more partial values for each metric associated with the one or more query results based on the one or more level bin records; and
generating the one or more query results based on the one or more partial values and the base value.
13 . The system of claim 10 , wherein the instructions are further configured to cause the system to perform further actions, comprising:
generating a new level bin based on one or more of the update information, an age of the one or more level bin records, or a capacity of the one or more level bin records.
14 . The system of claim 10 , wherein the instructions are further configured to cause the system to perform further actions, comprising:
providing a hierarchical index based on the file system; and
generating the one or more base records in the base bin based on the hierarchical index.
15 . The system of claim 10 , wherein the instructions are further configured to cause the system to perform further actions, comprising:
determining one or more partial values associated with one or more changes to one or more metrics comprising the metric based on one or more level bins in the journal;
updating the one or more base records in the base bin based on the one or more partial values; and
discarding the one or more level bins.
16 . The system of claim 10 , wherein the one or more base records are associated with a separate value of the metric with each object in the file system including the target object, and wherein the one or more base records are ordered based on the file system.
17 . A non-transitory computer-readable medium storing code, the code comprising instructions executable by a processor to:
provide update information associated with a target object in a file system, wherein the update information is associated with a change of a value of a metric associated with the target object from an initial value of the metric associated with the target object;
provide a journal based on the update information, wherein the journal includes a base bin that includes one or more base records, the one or more base records comprising a first column indicating a respective path-key for each object of a plurality of objects included in the file system and a second column indicating the initial value of the metric associated with the target object, the respective path-key corresponding to a respective file system path of the object;
determine one or more objects in the file system that are different from the target object and are hierarchical ancestors in the file system for the target object based on the update information and the file system;
generate one or more level bin records that correspond to the one or more objects, wherein each level bin record includes:
a third column comprising the change of the value of the metric associated with the target object and the change of the value of the metric associated with the one or more objects in the file system that are hierarchical ancestors in the file system for the target object, and
a fourth column comprising the respective path-key for each of the one or more objects;
provide a level bin that is associated with the base bin of the journal based on the update information; and
use the base bin and the level bin to reduce latency in generating, in response to one or more queries for the value of the metric associated with the target object or the value of the metric associated with the one or more objects, one or more query results comprising the changed value of the metric associated with the target object or the changed value of the metric associated with the one or more objects.
18 . The non-transitory computer-readable medium of claim 17 , wherein the instructions are further executable by the processor to:
store the one or more level bin records in the level bin, wherein a sort order of the one or more level bin records in the level bin is based on an ordering of the one or more base records in the base bin.
19 . The non-transitory computer-readable medium of claim 17 , wherein the instructions are further executable by the processor to:
determine a base value for each metric associated with the one or more query results based on the base bin;
determine one or more partial values for each metric associated with the one or more query results based on the one or more level bin records; and
generate the one or more query results based on the one or more partial values and the base value.
20 . The non-transitory computer-readable medium of claim 17 , wherein the instructions are further executable by the processor to:
generate a new level bin based on one or more of the update information, an age of the one or more level bin records, or a capacity of the one or more level bin records.