Statistics management for scale-out storage
Systems and processes for statistics management in a distributed storage system using a flat cluster architecture. Statistics for managed objects are collected using virtual statistics groups across multiple storage nodes. The systems and processes are compatible with storage systems that utilize microservice architectures.
1. A method for use with a distributed storage system comprising a plurality of storage devices, the method comprising:
initializing a statistics group on a plurality of storage nodes, the statistics group associated with a managed object;
assigning ownership of the managed object to a first one of the storage nodes having a data chunk management service and storage devices;
collecting statistics values for the managed object on the first one of the storage nodes;
changing ownership of the managed object to a second one of the storage nodes, while continuing to store statistics values collected by the first one of the storage nodes on the first one of the storage nodes after the change of ownership, the second one of the storage nodes having a data chunk management service and storage devices;
collecting statistics values for the managed object on the second one of the storage nodes;
receiving, at an arbitrary one of the plurality of storage nodes, a request for a statistics value within the statistics group; and
responding to the request by: blindly querying, by the arbitrary storage node, each of the other storage nodes in the plurality for statistics values associated with the managed object, receiving, by the arbitrary storage node, the statistics values collected on at least one of the first storage nodes and the second one of the storage nodes, combining, by the arbitrary storage node, the statistics values collected on the first one of the storage nodes and the statistics values collected on the second one of the storage nodes, and returning the combined statistics values; and
using the combined statistics values to scale out the distributed storage system with cluster-level functions distributed evenly among the plurality of storage nodes.
2. The method of claim 1 wherein the managed object comprises a table configured to store metadata about storage chunks stored within the storage devices.
3. The method of claim 1 wherein combining the statistics values collected on the first one of the storage nodes and the statistics values collected on the second one of the storage nodes comprises computing a statistics function over the statistics values collected on the first one of the storage nodes and the statistics values collected on the second one of the storage nodes.
4. The method of claim 3 wherein computing a statistics function over the statistics values comprises computing a sum of the statistics values, determining a most recent value from the statistics values, determining a maximum value from the statistics values, or determining a minimum value from the statistics values.
5. The method of claim 1 wherein combining the statistics values collected on the first one of the storage nodes and the statistics values collected on the second one of the storage nodes comprises appending time series data collected on the second one of the storage nodes to time series data collected on the first one of the storage nodes.
6. The method of claim 1 wherein the distributed storage system employs a microservice architecture.
7. A distributed storage system, comprising:
a plurality of storage nodes each having a plurality of storage devices and configured to:
initialize a statistics group on a plurality of storage nodes, the statistics group associated with a managed object;
assign ownership of the managed object to a first one of the storage nodes having a data chunk management service and storage devices;
collect statistics values for the managed object on the first one of the storage nodes;
change ownership of the managed object to a second one of the storage nodes, while continuing to store statistics values collected by the first one of the storage nodes on the first one of the storage nodes after the change of ownership, the second one of the storage nodes having a data chunk management service and storage devices;
collect statistics values for the managed object on the second one of the storage nodes;
receive, at an arbitrary one of the plurality of storage nodes, a request for a statistics value within the statistics group;
respond to the request by: blindly querying, by a statistics manager included in the arbitrary storage node, a statistics client included in each of the other storage nodes in the plurality for statistics values associated with the managed object, receiving, by the arbitrary storage node, the statistics values collected on at least one of the first storage nodes and the second one of the storage nodes, combining, by the arbitrary storage node, the statistics values collected on the first one of the storage nodes and the statistics values collected on the second one of the storage nodes, and returning the combined statistics values; and
use the combined statistics values to scale out the distributed storage system with cluster-level functions distributed evenly among the plurality of storage nodes.
8. The distributed storage system of claim 7 wherein the managed object is a table configured to store metadata about storage chunks stored within the storage devices.
9. The distributed storage system of claim 7 wherein ones of the plurality of storage nodes are configured to computing a statistics function over the statistics values collected on the first one of the storage nodes and the statistics values collected on the second one of the storage nodes.
10. The distributed storage system of claim 9 wherein ones of the plurality of storage nodes are configured to compute a sum of the statistics values, to determine a most recent value from the statistics values, to determine a maximum value from the statistics values, or to determine a minimum value from the statistics values.
11. The distributed storage system of claim 7 wherein ones of the plurality of storage nodes are configured to append time series data collected on the second one of the storage nodes to time series data collected on the first one of the storage nodes.
12. The distributed storage system of claim 7 wherein the statistics manager is a microservice.
13. The distributed storage system of claim 7 wherein the statistics manager includes a REST (Representational State Transfer) API configured to process statistics requests from user applications.
14. The distributed storage system of claim 7 wherein the statistics manager includes a database to store statistics received from the plurality of statistics clients.